Job scheduling on the Cray T3E at NSC

NQS and Scheduling

1. jobs requesting N PEs will not be started if there are not N PEs available.
2. queues have a run limit associated with them, which is the maximum number of jobs from that queue that can be running at any one time (see qstat -m).

Once a batch job starts running, it is just a shell script run by the user. Commands are executed by the script as if the user had typed them interactively. All of these commands are under control of the GRM (see below), according to the GRM rules, and are not under direct control of NQS.

Once your job is released to NQS, you can use "qstat -a" to see how your batch jobs are progressing through the system. For example:

t3e 120% qstat -a
----------------------------------
NQS 3.3.0.10 BATCH REQUEST SUMMARY
----------------------------------
IDENTIFIER    NAME    USER     LOCATION/QUEUE        JID  PRTY REQMEM REQTIM ST
------------- ------- -------- --------------------- ---- ---- ------ ------ ---
15354.sn6326  HIROMB  sm_towil smhi.sh_la@sn6326           327 262144     ** Qce
15366.sn6326  HL_h4420 smp_lmeu smhi_prod@sn6326      2357   20    622     ** R06
15206.sn6326  sounding x_mardi  long@sn6326            325   20    450     ** R04
15365.sn6326  top_0_oc b685850  test@sn6326           2340   39    321     ** R04
15150.sn6326  script1_ x_bodur  p1@sn6326             2304   20   7906     ** R05
15363.sn6326  komega  a91033   saab.sh_256mb@sn6326        932 262144     ** Qce
15151.sn6326  script2_ x_bodur  spec_req@sn6326             912 262144     ** Qqs
15205.sn6326  sounding x_mardi  univ.lo_la@sn6326       30   20    450     ** R04
15300.sn6326  sounding x_mardi  univ.lo_la@sn6326           547 262144     ** Qce
15295.sn6326  case3   a96141   saab.lo_256mb@sn6326  1286   20   1334     ** R05
15290.sn6326  case3   a96090   saab.lo_256mb@sn6326  1310   20    354     ** R04
15292.sn6326  case1   b685849  saab.lo_256mb@sn6326        968 262144     ** Qce
15293.sn6326  case1   b685849  saab.lo_256mb@sn6326        968 262144     ** Qce
15296.sn6326  case3   a96141   saab.lo_256mb@sn6326        714 262144     ** Qce
15368.sn6326  FB_NS_WT t97175   saab.lo_256mb@sn6326        266 262144     ** Qce
15356.sn6326  erect_1 x_antst  univ.lo_256mb@sn6326        394 262144     ** Qce
15364.sn6326  case4   a96141   test.256mb@sn6326     2355   20    876     ** R05
15124.sn6326  gbchh   x_nscssn bonus@sn6326                957 262144     ** Qqe
15173.sn6326  ODELIUS x_micod  bonus@sn6326                952 262144     ** Qqe
15258.sn6326  falk19  x_nscms  bonus@sn6326                720 262144     ** Qqe
15299.sn6326  ODELIUS x_micod  bonus@sn6326                650 262144     ** Qqe
15367.sn6326  Run_3.21 sm_ngust bonus@sn6326                301 262144     ** Qqe
no pipe queue entries

An "R" in the ST column means a job is running and "Q" means the job is queued.

Global Resource Manager

At a low level, parallel processes are managed by the Global Resource Manager (GRM). Batch jobs released by NQS as well as jobs run interactively comes under control of the GRM, which runs jobs on a first-fit basis.

Normally, a batch job submitted to NQS will pass through GRM since NQS checks that enough PEs are available. There are however a few instances where this might not be true and the user might think his jobs is executing by looking at the NQS queue while it in fact is stuck in GRM:

1. If the migration of a job ( see below) does not work , NQS might start the job believing there are enough PEs but GRM will hold the job since there are not enough contiguous PEs available.
2. An interactive parallel application starts running before the batch script (especially if the script does some work, such as copying files) starts the parallel application.

An excellent tool to help understand and monitor the system is a program called "xlotto" (/usr/local/bin/xlotto), developed at NSC. Just type "xlotto" and you will get a map over processor usage as well as running processes.

Job migration