Tornado User Guide

Description

Hardware

Software

Policy

Getting an account on Tornado

Getting a grid certificate

Security

The batch queueing system

Storage and file systems

Compiling

Math libraries

Quick getting started example

Recommendations

Description

Tornado is a climate research cluster owned by Rossby Centre (80%) and MISU (20%). It is located in NSCs computer facility in Linköping and is maintained by NSC.

Hardware

Tornado consists of several different types of nodes. It has 129 compute nodes, 2 analysis nodes, 1 login node and system/disk servers. Every node in Tornado is a Dell PowerEdge 1850 Server (except for the new login node which is a Dell PowerEdge 1950), but they are not strictly identical, the analysis nodes have for example a total of 12GB RAM and the new login node has a Quad Core (4x) 3.0GHz Xeon CPU w/ 4GB RAM. These nodes are connected in a InfiniBand network.

• 64-bit Dual Intel Xeon EM64T CPU @ 3.40GHz w/ 2MB cache/CPU.

• New login node: Quad Core Intel Xeon EM64T CPU @ 3.0GHz w/ 2MB cache/CPU.

• Every node except for the analysis nodes and the login node have 2GB ECC DDR-2 RAM. The analysis nodes have a total of 12GB RAM, and the login node has 4GB or RAM.

• InfiniBand HCA: Mellanox Technologies MT25208 InfiniHost III Ex.

• Mellanox 144 port Infiniband Switch.

• 3Com Gigabit Ethernet switch.

Node Configuration:

Policy

A short guideline on how to use the different types of nodes on tornado:

Policy for the login node:

• Suitable stuff: Editing, Compiling, File transfers.

• You may run tiny test runs (< 60 CPU-seconds _and_ < 2 CPU:s _and_ < 1 GiB memory, _else_ use a compute node).

Policy for the analysis nodes:

• Do _not_ use the analysis nodes if your application is capable of running elsewhere.

• Things suitable for the analysis nodes: Anything that requires > 2GiB of memory.

Policy for the compute nodes:

• n1-n129 must be allocated throught the batch queueing system.

• Things suitable for compute nodes:

  • Jobs (>= 60 CPU:seconds _or_ >= 2 CPU:s _or_ >= 1 GiB memory).

  • Parallell applications.

  • Compiling (using interactive jobs).

  • Test runs (interactive/regular jobs utilizing the development reservation).

Tip:

• Use the command top to get realtime information about running processes on the node (default sorted by cpu-usage, press shift-m to sort by memory usage).

• Use nice to run applications with a lower priority (e.g. nice -n 19 myapplication executes myapplication with the lowest priority). Use renice to alter priority of already running processes.

Software:

*Matlab has the following toolboxes:

Local toolboxes (see /usr/local/matlab/toolboxes)

• mexnc-2.0.14-a64-R14

• netcdf_toolbox-1.0.8

• snctools-2.0.16

Commercial toolboxes

• MATLAB Compiler

• Chemical engineering

• Communication Toolbox

• Communication blockset

• Control Systems Toolbox

• Data aquisition Toolbox

• DSP blockset

• Fuzzy Logic Toolbox

• Image Processing Toolbox

• Image Aquisition Toolbox

• Instrument control Toolbox

• Mapping toolbox

• Neural Networks Toolbox

• Non-linear Control Toolbox

• Optimization Toolbox

• PDE toolbox

• Real-Time workshop

• Real-Time Windows target

• Robust Control Toolbox (inkl My-analysis och LMI control)

• Signal Processing Toolbox

• Simmechanics Toolbox

• SimPowersystems

• Simulink

• Spline Toolbox

• Stateflow Toolbox

• Statistics Toolbox

• Symbolic Math toolbox

• System Identification Toolbox

• Virtual reality Toolbox

• Wavelet toolbox

• xPC Target

If you wish to have some specific software installed, contact support@nsc.liu.se or smhi-support@nsc.liu.se (only for SMHI users) specifying Tornado + relevant information in the subject header.

Environment

We use module to handle the environment when there exists several versions of the same software installed, e.g. different compilers. This application sets up the correct paths to the binaries, man-pages, libraries, etc. for the currently selected module. Here is a list of the most useful arguments to module:

module

module list

module avail

module load example

module unload example

A default environment is automatically declared when you log in. Check using module list

Example: Switch from the default 64-bit icc compiler to a 32-bit icc compiler:

Example:
[perl@tornado ~]$ module list
Currently loaded modules:
  1) icc
  2) ifort
  3) mkl
  4) pbs
  5) scali
  6) matlab
  7) dotmodules
  8) default

[perl@tornado ~]$ module avail

In directory /etc/cmod/modulefiles:

  +default                     -ifort/8.1.032_EM64T
  +dotmodules                  -ifort/8.1.033_EM64T
  -icc/8.1.027_EM64T           -ifort/8.1.035_IA32
  -icc/8.1.029_EM64T           -ifort/9.0.024_EM64T
  -icc/8.1.030_IA32            -ifort/9.0.024_IA32
  -icc/8.1.032_EM64T           -ifort/9.0.027_EM64T (def)
  -icc/8.1.033_EM64T           -ifort/default
  -icc/8.1.033_IA32            -matlab/default
  -icc/8.1.035_IA32            -mkl/7.2.1p3 (def)
  -icc/9.0.023_EM64T           -mkl/7.2.1p3_IA32
  -icc/9.0.023_IA32            -mkl/default
  -icc/9.0.026_EM64T (def)     -pbs/default
  -icc/default                 -root
  -ifort/8.1.026_IA32          -scali/default
  -ifort/8.1.027_EM64T         -totalview/7.1 (def)
  -ifort/8.1.029_EM64T         -totalview/default
  -ifort/8.1.029_IA32

[perl@tornado ~]$ module unload icc
[perl@tornado ~]$ module load icc/8.1.030_IA32
[perl@tornado ~]$ module list
Currently loaded modules:
  1) ifort
  2) mkl
  3) pbs
  4) scali
  5) matlab
  6) dotmodules
  7) default
  8) icc/8.1.030_IA32

Tip: The environment is specified in the files located under /etc/cmod/modulefiles.

Resource Name Environment Variable

If you are using several NSC resources and copying scripts between them, it can be useful for a script to have a way of knowing what resource it is running on. You can use the NSC_RESOURCE_NAME variable for that:

[username@tornado ~]$ echo "Running on $NSC_RESOURCE_NAME"
Running on tornado

Getting an account on Tornado

1. Contact support@nsc.liu.se or smhi-support@nsc.liu.se (for SMHI users only) with topic “New account on Tornado” or similar. We already have a list of people who are authorized for an account (given by our contact persons from Rossby and MISU - Anders Ullerstig and Michael Burger).

2. Supply the following information: Name, address, e-mail and telephone number.

3. We will send an e-mail back containing your account information + password. Please change this password as soon as possible

4. Read the security section below.

Getting a grid certificate

In order to get access to NSC's file storage you must have a valid grid certificate. The correct procedure to get a certificate is described at (please read both "User Guide - Introduction" and "Getting a Grid Certificate"):

http://www.nsc.liu.se/systems/storage.html#User Guide - Introduction

The layout of the storage system is also described and there is also a detailed tutorial of how to use the storage system including a lot of examples.

Accessing Tornado

Log in to Tornado using ssh:

ssh username@tornado.nsc.liu.se

Use scp and sftp for transferring of files between Tornado and other computers:

Example: copy the file named example to Tornado:

scp ./example user@tornado.nsc.liu.se:~/documents

Example: connect to Tornado using sftp:

sftp user@tornado.nsc.liu.se
Connecting to tornado.nsc.liu.se...
sftp>

Security

SSH public-key authentication

There is an alternative to traditional passwords. This method of authentication is known as key-pair or public-key authentication. While a password is simple to understand (the secret is in your head until you give it to the ssh server which grants or denies access), a key-pair is somewhat more complicated.

A key-pair is as the name suggests a pair of cryptographic keys. One of the keys is called the private key (this one should be kept secure and protected with a pass phrase) and a public key (this one can be passed around freely as the name suggests).

After you have created the pair, you have to copy the public key to all systems that you want to ssh to. The private key is kept as secure as possible and protected with a good pass phrase. On your laptop/workstation you use a key-agent to hold the private key while you work.

• Can be much more secure than regular password authentication

• Can be less secure if used incorrectly (understand before use)

• Allows multiple logins without reentering password/pass phrase

• Allows safer use of ssh chains

Short description of the necessary steps involved using SSH public-key authentication (read Damien Miller's guide above for more details):

• Generate a key-pair on your desktop/laptop, choose a good pass phrase and make sure private key is secure (once).

• Put your public key into ~/.ssh/authorized_keys on desired systems.

• Load your private key into your key-agent (ssh-add with OpenSSH).

• Run ssh all you want without reentering your pass phrase and without the risk of anyone stealing your password.

The Batch Queueing System

In order to utilize the cluster as efficient as possible a batch queueing system is used (Torque). Instead of running your applications interactively you submit “jobs” to the batch queuing system which automatically starts your job when there are resources available. A job can have several attributes but among the most important ones are:

• The number of compute nodes that the applications should use.

• The maximum allowed time for the application to run.

This makes it easier to ensure that the cluster is not idling at any time and we can easily specify that no more than one job at a time may use a node, preventing inefficient task switching and swapping. By also using an job scheduler (Moab) we can also easily implement policies, that allows us to prioritize between different types of jobs.

Tip: A graphical overview of the scheduling status is available at:

http://www.nsc.liu.se/cgi-bin/tornadostatus

Submitting Jobs

For a quick introduction how to compile and run jobs see Quick getting started example

There are two ways to submit jobs to the batch queue system, either as an interactive job or as a batch job. Interactive jobs are most useful for debugging as you get interactive access to the input and the output of the job when it is running. But the normal way to run the applications is by submitting them as batch jobs since otherwise you would have to manually start your application.

qsub is the name of the command for submitting jobs, either if it is a batch job or an interactive job.

Example of useful arguments to qsub (more important arguments are marked bold), read the man page for additional arguments and details:

-I

-j oe

-l nodes=n:ppn=2

-l walltime=hh:mm:ss

-m abe

-M email1@smhi.se,email2@misu.se

-N myjobname

Batch Jobs

Two sample PBS-scripts are available for download: pbssample.simple and pbssample.advanced.

1. Create a PBS-script. This is a shell-script with additional declarations for the arguments to qsub, the arguments to qsub are declared as #PBS pbsargument, e.g. -N myjobname is specified as #PBS -N myjobname.

   Example of a PBS-script named pbssample.simple using 24 nodes and is submitted to riskjobb. The wall clock time is 10 minutes and an e-mail will be sent when the job exits normally or exits with an error:

   #!/bin/sh # Request 24 nodes using both processors for the job and request 10 minutes of wall-clock time. #PBS -l nodes=24:ppn=2,walltime=00:10:00 # Request regular output (stdout) and error output (stderr) to the same # file. #PBS -j oe # Send mail when the job exits normally (e) or aborts with an error (a). #PBS -m ae # Goto the directory from which you submitted the job. cd $PBS_O_WORKDIR # Start the job with mpprun on the nodes that the batch queue system have # allocated for your job. mpprun helloworld

2. Submit the job, by specifying the PBS-script as the only argument to qsub:

   [perl@tornado example]$ qsub pbssample.simple 5723.tornado

3. Check the status of the job with qstat. by specifying qstat -n you get information about allocated nodes. For even more details add -f as an argument. You should also try using showq.

   [perl@tornado example]# qstat Job id Name User Time Use S Queue ---------------- ---------------- ---------------- -------- - ----- 5723.tornado pbssample.simpl perl 00:00:00 R riskjobb [perl@tornado example]# qstat -n tornado: Req'd Req'd Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time --------------- -------- -------- ---------- ------ --- --- ------ ----- - 5724.tornado perl riskjobb pbssample. -- 24 -- -- 01:00 R 00:00 n72/0+n71/0+n70/0+n69/0+n68/0+n67/0+n66/0+n65/0+n64/0+n63/0+n62/0+n61/0 +n60/0+n59/0+n58/0+n57/0+n56/0+n55/0+n54/0+n53/0+n52/0+n51/0+n50/0+n49/0

4. Since we specified -m ae, an e-mail will be sent to the local user at tornado when/if the job exits and aborts. The standard out and standard error is saved in the directory from where the job was submitted as pbssample.simpl.o5723

Interactive Jobs

Submitting an interactive job is done at the command line by adding the argument -I to qsub.

Example: Interactive version of the PBS-script in Batch Jobs

[perl@tornado perl]$ qsub -I -lnodes=24,walltime=00:10:00
qsub: waiting for job 5972.tornado to start
qsub: job 5972.tornado ready

[perl@n11 perl]$ cd $PBS_O_WORKDIR
[perl@n11 perl]$ mpprun hello_world.icc.scampi

The output are displayed directly to the shell prompt.

Alternatively, you may run mpprun directly if you want to run a MPI application. mpprun will make an implicit call to qsub with a default wall time of 1 hour and automatically execute the application.

Example: Interactive job by calling mpprun directly.

[perl@tornado example]$ mpprun -np 24 hello_world
Using PBS to spawn an interactive job
spawn /usr/pbs/bin/qsub -I -lwalltime=1:00:00,nodes=24
qsub: waiting for job 5977.tornado to start
qsub: job 5977.tornado ready

[perl@n11 perl]$ exec /usr/local/bin/mpprun -Norder=defult hello_world

Automatic cleanup and file stage in/out

1. When a job is finished, the batch queueing system kills all running processes belonging to the user that owns the exiting job on the compute node.

2. Before a new job is started all contents of /disk/local is removed on the compute node.

If you want to be guaranteed not to loose what you have written to /disk/local you should use the PBS functionality called stage out and copy your data to another file system (/home or /nobackup/) when your job exits. Since a job can exit in several ways (normally, segmentation fault, killed by pbs) you can not otherwise be guaranteed that your data always will be copied if you do this by yourself (e.g. in your application or in the PBS-script).

• File stage out only can by done from your master node (i.e. your highest ranking node in the job).

• PBS file stage in is not available on tornado, but this may as well be done manually in the PBS script or in your application.

Please make sure that it works before using file stage out in your production jobs. See 'man qsub' for more details.

Example: copy /disk/local/data0 and /disk/local/data1 (output written from “mpiapplication”) from the master compute node in my job to tornado:/home/perl when the job exits.

#!/bin/sh
#PBS -l nodes=16:ppn=2,walltime=01:00:00 
#PBS -W stageout=/disk/local/data0@tornado:/home/perl/,/disk/local/data1@tornado:/home/perl/

#Manual stage in:
cp /home/perl/tmp/test.gz /disk/local/

mpprun mpiapplication

Frequently used commands

Read the man-pages for more information about each listed command below. Commonly used commands are marked bold.

Frequently used PBS user commands:

qsub

qstat

qdel

Less frequently used PBS user commands:

qalter

qhold

qrerun

qrls

qsig

Moab commands:

Moab is the scheduler. It takes care of how the jobs are prioritized, and where these jobs are run. Moab supports advance reservations, QOS levels, backfill, and allocation management.

showq

showbf

showres

showstart

checkjob

Scheduling policy

• The maximum allowed walltime is 7 days. This is specified as -lwalltime=168:00:00 in PBS.

Bonus

On Tornade we use the same bonus system as on Monolith. Its purpose is to lower the priority of projects and users that has consumed their alloted time. Jobs from bonus users are only scheduled whenever there is no other, normal priority, job to run. Use projinfo to check the amount of time your group has used the current month.

Example output when using projinfo for a user belonging to the group rossby:

Project        Used[h]        Allocated[h]
  User
-----------------------------------------
rossby        38947.1              148608
  sm_wyser     2628.2
  sm_aulle     2210.4
  sm_meier    31504.1
  sm_psamu     2604.5

Interactive development slots

Between 0700 – 1800 on Monday to Friday we have reserved 16 nodes for shorter jobs with a walltime of 1 hour or less (see reservation with command showres or look at: http://www.nsc.liu.se/cgi-bin/tornadostatus). This will make it much easier to develop and test applications without having to wait a long time for some other large jobs to finish.

Storage & File systems

On Tornado there are different types of storage classes depending on level of reliability, size, speed, location, etc.

• Every user has at least one directory under /nobackup. To check which, use the command “ls -d /nobackup/*/yourusername”. Example:

[perl@tornado ~]$ ls -d /nobackup/*/perl
/nobackup/rossby1/perl /nobackup/rossby2/perl

• On /home and /nobackup/* there is a file called diskinfo containing each users total disk usage (+ previous day in diskinfo.old). This file is updated every night and is much quicker to use than du. Example:

[root@tornado ~]# sort -nk1 /home/diskinfo 


Diskinfo for file system/directory /tornado_home
[Measures in Megabytes of file space]
Thu Oct  6 06:05:01 CEST 2005
1       andro
1       msa
1       slurm
1       sm_doesc
1       sm_stran
1       sm_uhans
1       tmp
1       x_jenma
2       sp
14      perl
24      x_heiko
549     x_gorbr
985     sm_wyser
1457    sm_meier
3550    cap
4699    x_jenbr
9600    x_steso
27365   faxen
29267   sm_aulle
155367  sm_psamu

Compiling

For a quick introduction how to compile and run jobs see Quick getting started example.

GCC and both 64-bit and 32-bit Intel Compiler for Linux 8 and 9 are installed

We recommend using the 64-bit Intel compilers version 9 (used as default).

Compiling MPI applications

NSC Cluster Environment (NCE) is NSC's modification of GCC and Intel compilers which makes it easier to compile MPI-applications. Rather than having to specify a quite long list of libraries to be linked which differs between different MPI implementations, you only have to compile with these additional arguments:

-Nscampi

The options should be used both at compile-time (to specify the path to the include files) and at link-time (to specify the correct libraries).

Example: compiling the MPI-program, mpiprog.c with icc for ScaMPI (the same goes for gcc, g++, g77 and ifort):

icc mpiprog.c -o mpiprog -Nscampi

Intel compiler, useful compiler options

Below are some useful compiler options, please do "man ifort" or "man icc" for more. For additional information, see:

• http://www.intel.com/software/products/compilers/flin/ for Fortran.

• http://www.intel.com/software/products/compilers/clin/ for C/C++.

a) Optimisation

There are three different optimization levels in Intel's compilers:

-O0

-O1,-O2 

-O3

A recommended flag for general code is -O2 -mp and for best performance "-O3 -xP" which will enable software vectorisation. As always however, aggressive optimisation runs a higher risk of encountering compiler limitations.

b) Debugging

-g

-traceback

-fpe<n>

-mp

c) Profiling

-p

d) Options that only apply to Fortran programs

-assume byterecl

-r8

-i8

-zero 

-save

-CB

e) Large File Support (LFS)

To read/write files larger than 2GB you need to specify some flags at compilation:

Fortran: no additional flags needed.

CC/C++: LFS is obtained by specifying the flags below when compiling and linking:

-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 

f) Miscellaneous options

Little endian to Big endian conversion in Fortran is done through the F_UFMTENDIAN environment variable. When set, the following operations are done:

• The WRITE operation converts little endian format to big endian format.

• The READ operation converts big endian format to little endian format.

F_UFMTENDIAN = big 

F_UFMTENDIAN ="big;little:8" 

g) NCE options (locally supplied by NSC)

-Nscampi

Math libraries

Intel Math Kernel Library

MKL Version 7.2 is installed on Tornado.

For complementary information look at Intel® Math Kernel Library 7.2 for Linux* Technical User Notes. Additional documentation is available locally on Tornado at: /usr/local/intel/l_mkl_p_7.2.1.003/mkl721/doc/

The Math Kernel Library includes the following groups of routines:

• Basic Linear Algebra Subprograms (BLAS):

  • vector operations

  • matrix-vector operations

  • matrix-matrix operations

• Sparse BLAS (basic vector operations on sparse vectors)

• Fast Fourier transform routines (with Fortran and C interfaces)

• LAPACK routines for solving systems of linear equations

• LAPACK routines for solving least-squares problems, eigenvalue and singular value problems, and Sylvester's equations

• Vector Mathematical Library (VML) functions for computing core mathematical functions on vector arguments (with Fortran and C interfaces).

Full documentation can be found at http://www.intel.com/software/products/mkl/

Directory structure

mkl is located in $MKL_ROOT, defined at login. Semantically, MKL consists of two parts: LAPACK and processor specific kernels. The LAPACK library contains LAPACK routines and drivers that were optimized as without regard to processor so that it can be used effectively on processors from Pentium to Xeon. Processor specific kernels contain BLAS, FFTs, CBLAS, VML that were optimized for the specific processor. Threading software is supplied as a separate dynamic link library - libguide.so, when linking dynamically to MKL.

Linking with MKL

To use LAPACK and BLAS software you must link two libraries: LAPACK and one of the processor specific kernels (i.e. libmkl_p4n on tornado). Please use -L$MKL_ROOT instead of hardcoding the path (e.g. do not use -L/usr/local/intel/l_mkl_p_7.2.008/mkl72/lib/32). This will ensure that correct libraries are used when switching modules between different mkl versions.

Example (LAPACK library, Xeon EM64T processor kernel):

ld myprog.o -L$MKL_ROOT -lmkl_lapack -lmkl_p4n

Example (Dynamic linking. DLL dispatcher will load the appropriate dll for the processor dynamic kernel):

ld myprog.o -L$MKL_ROOT -lmkl -lguide -lpthread

Example (Dynamic linking using ifort):

ifort -L$MKL_ROOT -lmkl_lapack -lmkl_p4n example.o -o example

Quick getting started example

All the files in this example are available here.

For more detailed information, read Compiling and The Batch Queueing System.

Recommendations

• Use mpprun to start your MPI applications.

• No world writable home-directories, if you need this, create world writable subdirectories instead

• If you want your job to start fast, use a wall clock time that is less than or equal to 1 hr since there is 16 nodes reserved for jobs less than, or equal to 1 hr.

• Use projinfo to look at how much cpu time your group has used the current month

• Under /nobackup/*/ and /home there is a file called diskinfo which contains disk usage for each user. Example: sort -nk1 /nobackup/rossby2/diskinfo

• To plot data have a look at octave or gnuplot.

• A status overview of Tornado is available at status.nsc.liu.se

• Use the local disks (/disk/local/) for temporary storage instead of using the NFS mounted area /home. /home consists of two RAID 0 disks, it is globally mounted over NFS to every compute node and will perform poorly compared to the local disks attached to the compute nodes. Notice that /disk/local are not backed up and should be considered as temporary scratch areas.

• File transfer is available using scp and sftp.

• Backup of /home is made every night.

• For questions, contact support@nsc.liu.se or smhi-support@nsc.liu.se (only for users from SMHI).