Tetralith DCS large-scale storage and analysis nodes

Please note that these nodes are only available to the user groups listed below. Also, these projects can only be used to run jobs on the DCS analysis nodes, not on the other Tetralith nodes.

If you feel that your own user group have similar needs, contact support@nsc.liu.se to discuss your options.

Getting access to the analysis nodes and project storage

To access the analysis nodes and project storage, follow the normal NSC process for getting a login account on Tetralith. When applying, you should apply for membership in the DCS/analysis project for your user group:

If you are already a Tetralith user but not a member of one of the projects above, login to SUPR and request membership in the appropriate project.

Using project storage

The groups above have been allocated space on NSC's Centre Storage system.

Read the Centre Storage User Guide for more information.

For guidelines on where to store what type of data inside your project directory, please talk to your group's Technical contact (see the table above).

Use the snicquota command to see how much space you and your project colleagues are using.

Using the analysis nodes

Four Tetralith "fat" compute nodes (384 GB RAM each) are dedicated for the DCS projects for analysis use.

Since the number of nodes is limited, please make sure to only request as much resources as you need. I.e don't request a full node ("-N1") if you can make do with just a few cores (e.g "-n2").

If you need to run more or larger analysis jobs than fits on the DCS nodes, you need to also be a member of a "normal" (non-DCS) project, and then use that project (you can still access your DCS project directory from such jobs).

To use the analysis nodes, you need to use our batch scheduler/resource manager. There are two main reasons for this:

• Make it possible to automate data processing on a large scale using batch jobs.
• Make it possible for many users to share the analysis nodes without interfering with each other (e.g by using up all memory on a node).

Tetralith uses Slurm for scheduling (deciding who and what gets to use the system at any given time) and resource management (keeping track of nodes and allocated resources, starting and stopping jobs etc).

The analysis nodes are in a "reservation" on Tetralith. This means that they are separate from the other compute nodes - you must specify that you want to use them, and you must be authorized to use them.

To use analysis nodes, you must:

1. Use one of the DCS/analysis projects (and be a member of it): -A <project name>
2. State that you want to use the dcs reservation: --reservation=dcs

Example (interactive session using the Rossby project, four cores, 16GB RAM for 2 hours):

$ interactive -A snic2014-10-1 --reservation=dcs -t 2:00:00 -n 4 --mem=16000

Example (batch job using the MISU project, two cores and 32GB RAM for 24h)

$ sbatch -A snic2014-10-3 -n 2 --mem=32000 --reservation=dcs -t 24:00:00 myjob.sh

If you get the error "sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified", you are not a member of the project you're trying to use. Use the projinfo command to show which projects you are a member of.

Running batch jobs without annoying other users

The analysis nodes are primarily intended for interactive use. It is permitted to run batch jobs on them as long as you make sure that you don't use all available resources.

Note: one user can easily block all analysis nodes for other users by submitting a large volume of jobs at once. Doing this is not acceptable! Analysis nodes must be available for interactive use, i.e it is not acceptable for most users to have to wait for several hours to get an interactive shell. Please use common sense!

Common sense can be assisted by checking the queue status before submitting a large volume of jobs. To get an overview of running and queued jobs, use the squeue command with suitable options, e.g squeue --reservation=dcs -o "%.10i %.9P %.8u %.8T %.11L %.11l %.8N %.10m".