Systems  
Status displays
System status
Retired systems
 
 
 
 
 
 
 
 
 
 
 
 

Krypton User Guide

1 About this document

This is a temporary Krypton User Guide. We assume that you already have used an SMHI cluster at NSC, such as Gimle. If not, please read the Gimle User Guide.

Document history
DateVersionChanges
2012-07-031.0Initial version based on the Triolith document
2012-07-091.1More SLURM information
2012-07-121.2Devel reservation, implicit -C for thin nodes
2012-11-011.3SLURM reconfiguration etc

2 About Krypton

Krypton is a Linux-based cluster with 240 compute nodes and associated system and login servers.

The compute nodes are all of the HP Proliant SL230s type with two Intel E5-2660 (2.2 GHz Sandybridge) processors with 8 cores each (16 cores per node). Two of the compute nodes have 256 GiB memory each, ten of the compute nodes have 128 GiB memory each and the remaining compute nodes have 32 GiB each. The fast interconnect is Infiniband from Mellanox (FBR IB, 56 Gb/s) in a 2:1 blocking configuration. Each node has a 500GB local hard disk, of which ~440GiB is available as scratch space for user jobs (double that on the nodes with more RAM).

The operating system is CentOS 6 x86_64.

Initially, all software present on older systems will not be present on Krypton. If you would like to request a certain software package to be installed, please email smhi-support@nsc.liu.se.

The /home file system is shared with Gimle. Be careful if you change shell configuration files etc. on Krypton, as they will also be used on Gimle, and vice versa. You may use the $NSC_RESOURCE_NAME variable to differentiate between the clusters.

The Accumulus file systems (/nobackup/smhid*, /nobackup/fouo*, /nobackup/rossby*, /nobackup/fm*) are available.

The contents of the /software directory from Gimle is not available. Since most software will benefit significantly from being recompiled to use AVX we felt that it was best to start from scratch with an empty software directory and compile applications as they are needed.

Recompile your own applications! If you have previously compiled your own software we definitely recommend recompiling it on Krypton. See instructions on this page for how to build your applications on Krypton.

Submitting jobs on Krypton is done as on Gimle, with minor changes for risk jobs and fat nodes (see below). We are still using SLURM (sbatch, squeue etc), so only minor changes will required in your job scripts. There is no Moab scheduler in the background on Krypton, only SLURM.

3 Modifying your job scripts

You should check your existing job scripts before trying to use them on Krypton. Here are some of the changes you might have to do:

3.1 16 cores per node

If you have hard-coded the number of cores per node, please note that Krypton has 16 cores per compute node, not 8 as on Gimle/Bore/Byvind.

NOTE: It is not recommended to hard-code the number of cores in this way. Please use the relevant SLURM environment variables instead, e.g SLURM_JOB_CPUS_ON_NODE. For more information, read the sbatch man page.

3.2 Change /scratch/local to $SNIC_TMP

We are preparing for a future were at least some nodes (for example the nodes with more memory) are shared between multiple independent jobs. To be ready for that, we cannot let the top-level /scratch/local directory be writable.

There is still a local scratch directory available on each node (currently implemented as a subdirectory to /scratch/local). Instead of /scratch/local, use the environment variable $SNIC_TMP, which will be set to that directory (which is created for each job and deleted when the job ends).

E.g: if your job script looks like this

#!/bin/bash
#SBATCH -t 00:10:00
#
./myapp --tempdir=/scratch/local

then change it to

#!/bin/bash
#SBATCH -t 00:10:00
#
./myapp --tempdir=$SNIC_TMP

The name $SNIC_TMP comes from the NSC academic systems operated for SNIC. We decided to reuse that variable name instead of setting up another name just for the sake of it.

3.3 Jobs will require less wall time

If you move existing jobs from Gimle, the actual run time will almost certainly be shorter if you run them on the same number of nodes.

You can either reduce the number of nodes or request a shorter wall time to accommodate this.

3.4 Applications not installed yet or in another location

Many applications that are installed on Gimle are not yet available on Krypton. Before submitting a job, check that the application is available.

Even if the application is available, the version on Krypton might be different, so you might need to use a different path in your job script. Most module names will also have changed.

3.5 Account names

You might have to add the -A flag to specify account name. Read more under "Batch queue system" below.

4 Differences in /software and module files

On Krypton, we are doing several changes in how we manage the software installed on the system.

  1. There will eventually be a web page with updated, automatically generated documentation of all applications installed in /software. This is not yet present.
  2. There will be no default versions of software modules. Background: On earlier systems, for most applications NSC provided a default version. If you ran e.g "module add intel" you got the version of the Intel compilers that we considered the best. However, that version would be changed from time to time, and the behaviour of the application would then change without the user being aware of it. On Krypton, we will only recommend a certain version, it will be up to the user to decide if and when to change the version used. E.g "module add intel" will now display a message stating what the recommended version is, and the user can then load it using e.g "module load intel/12.1.4". In order to notify the user that we have changed our recommendation, wenever you try to load a non-recommended version a warning message might be displayed (but the module will still work as expected).
  3. No compilers are loaded when you login to the system. If you do "module load build-environment/nsc-recommended", you will get the the recommended versions of the Intel compilers (icc, ifort etc), Intel MPI and Intel MKL loaded. If you want to access the GCC compiler that is bundled with CentOS, you will have to load the gcc/4-centos6 module. The thinking behind the "build-environment/nsc-recommended" is that it will contain a core set of development tools that we have tested together.

5 Accessing Krypton

You access Krypton the same way you would access Gimle.

The external name of the login node is krypton.nsc.liu.se.

E.g "ssh -X sm_abcde@krypton.nsc.liu.se".

See the Gimle User Guide for more details on logging into NSC systems, security etc.

6 Batch queue system

6.1 Scheduling policy

Krypton uses simple first-in-first-out scheduling (with backfill so that shorter jobs that fit can run as long as they do not delay the start of jobs with higher priority).

The maximum wall time for a job is 7 days. The default time limit (if you do use a "-t" flag) is 2 hours. Please use the "-t" flag to set a time limit that is appropriate for each job!

6.2 Limits between groups

The following groups have a fixed number of nodes available for normal jobs (limits up-to-date as of 2012-11-01):

Group = SLURM AccountNodes
rossby73
sm_fouo73
fm15

These nodes are always allocated to their corresponding group.

At the moment, this is implemented using SLURM partitions, but that may change in the future.

The following groups all share a partition ("other") containing 67 nodes:

Group = SLURM AccountMax nodes per group
sm_foup59
sm_foua14
sm_foul30
sm_ml12
sm_mo12

As you see, the sum of the limits is greater than 67. This means that your jobs may be blocked due to an insufficient amount of nodes in total, or because of your group hitting the group-based limit.

6.3 Specifying your SLURM Account

If you are a member in more than one group, you should always use an option like "-A rossby", "-A sm_fouo" etc. to sbatch/interactive to tell SLURM what account to run under.

When you login, we set the SBATCH_ACCOUNT environment variable to your default account. If you are only part of one group, that should be enough and you do not need to use the -A option for normal job submission. You might have to use it under special circumstances, such as cron jobs.

6.4 Requesting nodes with more memory

The 12 nodes with more memory are kept in a separate partition. To use them, add "_fat" to your account name. For example, if you are part of the sm_foup group, use "-A sm_foup_fat" when you submit jobs.

The 10 fat nodes (128 GiB) will be used before the 2 huge nodes (256 GiB). If you need the huge nodes, use "-C huge" in addition to the "-A" option discussed above.

6.5 Running risk jobs

All groups on Krypton can submit risk jobs that are able to run on all available nodes (even fat and huge ones). The drawback is that risk jobs will be killed as soon as the nodes they run on are needed to be able to run a non-risk job.

To submit risk jobs, add "_risk" to your account name. For example, if you are part of the sm_fouo group, use "-A sm_fouo_risk".

If the risk job is a batch job (not interactive), you will probably also like to add "--requeue", so that the risk job is requeued automatically if it is preempted. Without this flag, the job will be canceled as soon as it is preempted.

If you want to control the set of nodes the risk jobs can be placed on (instead of all available), you can use the -C flag together with one of the names "fm", "rossby", "fouo", "other", "fat", "huge". You can also combine them using the "|" character (like " -C 'rossby|fouo' ").

6.6 Getting status

As stated above, there is no Moab scheduler running on Krypton. The resource manager SLURM handles the scheduling as well. This means that Moab commands like showq, checkjob etc. are not available. Instead, use SLURM commands like:

# Show the queue:
squeue

# Show node status summary:
sinfo

# Show the reason messages for nodes that are not available:
sinfo -R

# Show all details about job 123:
scontrol show job 123

You can also see an overview of running jobs and node status on the Scheduling status page for Krypton

Please note that queued jobs that are not running will not show up in the graphical overview (SLURM does not reserve nodes for them in advance like Moab did).

7 Building software on Krypton

Optimize for AVX. At the moment, we recommend that you add

-xAVX

to your compile flags.

The normal NSC compiler wrappers and mpprun is available, so to build and run an MPI application you only need to do:

module add build-environment/nsc-recommended
icc -Nmpi -o myapp myapp.c

And to run it in a batch your or "interactive" shell:

mpprun ./myapp

The compiler wrapper and mpprun will handle the compiler options to build against the loaded MPI version (Intel MPI in this case; it's part of build-environment/nsc-recommended) and how to launch an MPI application built against that MPI.

8 How to get help

You can contact the Krypton support team using the email address smhi-support@nsc.liu.se. You can use this address for anything related to Krypton, e.g

  • Asking a question
  • Telling us that something is wrong
  • Start a discussion regarding some long-term issue or future needs
  • Requesting the installation of a software package

When reporting a problem, please include all relevant information in your initial email, such as:

  • A relevant subject line (e.g "I cannot start Matlab on Krypton").
  • Your Krypton username.
  • The fact that your mail is about the Krypton system.
  • Which software you are using, including compilers (for example "ifort 12.1.4") and switches (for example "-xAVX").
  • A short description of the problem, specifying what actions you have performed, which results you got, and which results you expected to get.
  • If you have more than one separate problem, please send one email for each problem.
  • If possible include specific information to nail down when and where the problem arised, e.g. job ID, nodes, and/or point in time. It makes it easier for us to dig out logged information and possibly correlate your problem with other activities on the system.

You may use English or Swedish. We will try to reply in the same language. Please note that as we have some staff that are not fluent in Swedish, you may sometimes get an answer in English regardless of the language of your original question.

We read email to the support address during normal office hours (approximately 8-17 CET/CEST). We try to always give you some kind of answer (not counting the automated one) within two working days (but you will usually hear from us sooner than that).

You will get an automated reply from our support ticket system within a few minutes. If you want to add more information about your problem, reply to that email, that way the additional information will automatically be added to our database.

When you have a new question, please send a new email, do not reply to an old conversation. A reply to an old email might only reach the person who handled that problem, and that person could be busy, on leave etc. Sending a new email ensures that your request is seen by all support staff as soon as possible.






Page last modified: 2014-03-14 14:34
For more information contact us at info@nsc.liu.se.