Systems  
Status displays
System status
Retired systems «
  Software  
Biology
Chemistry »
Mathematics »
Physics »
Visualization »
 
 
 
 
 
 

1999-07-01
faxen@nsc.liu.se

Batch queues on NSC's T3E

Below are five sections with information about the system:

 
Please note that the maximum number of jobs in the system (executing and/or queued) is three per user.

General goals

The queues are based on the following primary goals:
  • Priority is given according to the following schedule:
  • - Daytime (weekdays 08:00-18:00): test and short (small and large) jobs are prioritized for increased throughput.

    - Non daytime: long and short jobs have equal priority.

    Priority means both having part of the machine reserved AND be the first choice when there is competition from other classes of jobs.

  • The batch system will try to maintain a usage of the system that corresponds as closely as possible to the financing of NSC.
  • The batch system assumes that the total available time is divided up between users at allocation time and that the batch system is only responsible for spreading the usage out over time. Users who attempt to use more than their allocated time can still use the system but only through the bonus system.

Implementation details

  • Job definitions
  • short: <= 6 hours
    long (128MB PE): > 6 hours <= 48 hours
    long (256MB PE): > 6 hours <= 24 hours
    small (128MB PE): <= 48 PE
    small (256MB PE): <= 32 PE
    large: > 48 PE
    test: <= 65 PE, <=60 min
    test ( 256MB PE): <= 48 PE, <=60 min
    interactive: <= 16 PE, <= 15 minutes.

  • 11 job queues, one for each job type
  • Job type          Time limit (hours)  PE-limit   
    ----------------------------------------------------
    short+small              6               48 
    long+small              48               48
    short+large              6              130
    long+large              48              130
    short_256mb              6               32
    long_256mb              24               32
    spec_req                 -                -
    p1                       -                1
    test                     1               65
    test.256mb               1               48
    bonus                    -                -
    ----------------------------------------------------
    
  • Remember that the system will choose the correct queue based on the number and type of PEs and the time limit that you are requesting. Use the information above when making your decision on the resources that you are requesting.
  • 20 PEs are reserved for interactive and test jobs during daytime (08:00-17.00).
  • Jobs that does not fit in the queue model (usually very long and sometimes also large jobs) will be put in a special queue. You have to submit a mail to support@nsc.liu.se to request that this job is started! We will then start the job at an appropriate time.
  • In situations where the system is poorly utilized or over utilized, manual intervention will occur. This might involve check pointing a running job and putting it back in the queue system to be continued when it is time to run again.
  • Within each queue, priority will be given for length of time in the queue, fair share and smallest requested MPP CPU time.
  • Priorities between queues (intra-queue) will change between daytime and non daytime.
  • The maximum number of jobs in the system (executing and/or queued) is 3 per user.
  • Special queues are defined for the use of 256MB PEs. See How to run on nodes with 256 MB of local memory.
  • We also have different queues for the University, SAAB, and SMHI. See further information below!
  • Jobs that are queued for a long time (short jobs: 24 hours, long jobs: 48 hours) are automatically moved to high priority queues:long,short,long_256mb and short_256mb.
  • The bonus queue is for users who run out of their allocated CPU-time for the current accounting period. Look here for a description of the bonus system

  • Disclaimer

  • The number of PEs that will be reserved as well as what constitutes a short, long, small and large job can and will be modified over time as part of the tuning of the queue system!

Some important things to remember.

  • We have restricted the limits for interactive use to 16 PEs and 15 minutes. If you need to do larger short runs, use the test queue with a limit of 65 PEs and 30 min.
  • We will continue to do manual start of jobs to optimize the use of the T3E. Sometimes this will require us to stop, checkpoint and requeue a job that was manually started.
  • Within each queue, priority is given to length of time in the queue AND smallest run time. Make sure you do not specify more time than you actually need!
  • There is a new tool "/usr/bin/xlotto" that graphically shows the usage of the system as well as the status of all queues. See http://www.nsc.liu.se/xlotto/xlotto.nsc.html for more information.
  • You need to send us an email whenever you submit a job that goes to the queue for special jobs ( > 48 hours and/or > 130 PEs).


Different queues for SAAB, SMHI and University users.

We have implemented separate batch queues for SAAB, SMHI and University users. The six "main" batch queues:
long_small  ( abbreviated "lo_sm" )    : <= 48h, <= 48PE
long_large  ( abbreviated "lo_la" )    : <= 48h, <=130PE
short_small ( abbreviated "sh_sm" )    : <=  6h, <= 48PE
short_large ( abbreviated "sh_la" )    : <=  6h, <=130PE
short_256mb ( abbreviated "sh_256mb" ) : <=  6h, <= 32PE
long_256mb  ( abbreviated "lo_256mb" ) : <= 24h, <= 32PE
are defined for SAAB, SMHI and University users.

This means that we have a total of 18 "main" queues plus the p1, test, test256mb, spec_req, bonus and smhi_production queues.

The queues will be prioritized as earlier described, higher priority to short jobs during day time and to long and large jobs during night time.

A convenient command to view a summary of jobs in each user group is to use the following command:
qstat -mC saab_jobs   (or smhi_jobs, univ_jobs):


------------------------------------
NQS 3.3.0.5 QUEUE COMPLEX: saab_jobs
------------------------------------
QUEUE NAME                RUN     QUEUE-PE'S    R-PE'S  R-TIME  P-TIME
                        LIM/CNT     LIM/CNT     LIMIT   LIMIT   LIMIT
----------------------- --- ---  ------ ------  ------  ------  ------
saab.lo_sm               20/1        **/16          48  172800      **
saab.lo_la               20/0        **/0          130  172800      **
saab.sh_la               20/0        **/0          130   21600      **
saab.sh_sm               20/1        **/11          48   21600      **
saab.sh_256mb            20/0        **/0           32   21600      **
saab.lo_256mb            20/1        **/19          32   86400      **
----------------------- --- ---  ------ ------  ------  ------  ------
saab_jobs                --/3       144/46    
----------------------- --- ---  ------ ------  ------  ------  ------


The bonus system.

The bonus system continously surveys all active accounts on the system. When a project has used up the allocated CPU hours for the current month, this is detected by the bonus system and the following actions will take place:

1) All members of the project will be informed (through email) that the allocated CPU hours for the current month are consumed and that jobs will only be executed when there is "free" time on the system.

2) Jobs from the project (with no time left) is only executed when the usage of the system is below a certain threshold. This is done by queuing all batch requests from the project in the bonus queue. The bonus queue is always stopped and thus does not allow any jobs to be executed, it is merely a place holder. When the load is light however, one batch job will be moved from the bonus queue to the regular queue system and thus be eligible for execution. It is always the oldest job that will be picked.

This has the dual benefit of making sure everyone get's his/her share of the resources as well as making it possible for users with no time left to use the system at off peak periods.

A user can always monitor his/hers own usage statistics with the command "/usr/local/bin/project"

Test accounts generally will not be allowed to run under the bonus system.When the allocated hours are consumed for a particular month, no runs are therefore possible until the following month.






Page last modified: 2002-12-11 13:56
For more information contact us at info@nsc.liu.se.