Systems  
Status displays
System status
Retired systems
 
 
 
 
 
 
 
 
 
 
 
 

Tornado User Guide

Description

Hardware

Software

Policy

Getting an account on Tornado

Getting a grid certificate

Security

The batch queueing system

Storage and file systems

Compiling

Math libraries

Quick getting started example

Recommendations

Description

Tornado is a climate research cluster owned by Rossby Centre (80%) and MISU (20%). It is located in NSCs computer facility in Linköping and is maintained by NSC.

Hardware

Tornado consists of several different types of nodes. It has 129 compute nodes, 2 analysis nodes, 1 login node and system/disk servers. Every node in Tornado is a Dell PowerEdge 1850 Server (except for the new login node which is a Dell PowerEdge 1950), but they are not strictly identical, the analysis nodes have for example a total of 12GB RAM and the new login node has a Quad Core (4x) 3.0GHz Xeon CPU w/ 4GB RAM. These nodes are connected in a InfiniBand network.

  • 64-bit Dual Intel Xeon EM64T CPU @ 3.40GHz w/ 2MB cache/CPU.

  • New login node: Quad Core Intel Xeon EM64T CPU @ 3.0GHz w/ 2MB cache/CPU.

  • Every node except for the analysis nodes and the login node have 2GB ECC DDR-2 RAM. The analysis nodes have a total of 12GB RAM, and the login node has 4GB or RAM.

  • InfiniBand HCA: Mellanox Technologies MT25208 InfiniHost III Ex.

  • Mellanox 144 port Infiniband Switch.

  • 3Com Gigabit Ethernet switch.

Node Configuration:


tornado.nsc.liu.se

Login Node



a1 - a2

Analysis Nodes

12 Gb RAM.


n1 – n129

Compute Nodes




Policy

A short guideline on how to use the different types of nodes on tornado:

Policy for the login node:

  • Suitable stuff: Editing, Compiling, File transfers.

  • You may run tiny test runs (< 60 CPU-seconds _and_ < 2 CPU:s _and_ < 1 GiB memory, _else_ use a compute node).

Policy for the analysis nodes:

  • Do _not_ use the analysis nodes if your application is capable of running elsewhere.

  • Things suitable for the analysis nodes: Anything that requires > 2GiB of memory.

Policy for the compute nodes:

  • n1-n129 must be allocated throught the batch queueing system.

  • Things suitable for compute nodes:

    • Jobs (>= 60 CPU:seconds _or_ >= 2 CPU:s _or_ >= 1 GiB memory).

    • Parallell applications.

    • Compiling (using interactive jobs).

    • Test runs (interactive/regular jobs utilizing the development reservation).

Tip:

  • Use the command top to get realtime information about running processes on the node (default sorted by cpu-usage, press shift-m to sort by memory usage).

  • Use nice to run applications with a lower priority (e.g. nice -n 19 myapplication executes myapplication with the lowest priority). Use renice to alter priority of already running processes.

Software:

Operating System

Centos 4.

CentOS 4 is a 64-bit operating system (a rebuild of Redhat Enterprise Linux 4), it has the capability to run both 64-bit and 32-bit binaries.

Resource Manager

Torque


Scheduler

Moab


Compilers

Intel C/C++/Fortran for Linux

GCC


Math Library

Intel Math Kernel Library


MPI

Scali MPI Connect



Numerical Applications

Matlab*

Octave

Gnuplot


Matlab uses Linköping University site license. It is available from the login node and the analysis nodes (a1-a2) and all the compute nodes. See list of toolboxes below.

Octave and GnuPlot are recommended for plotting data.



*Matlab has the following toolboxes:

Local toolboxes (see /usr/local/matlab/toolboxes)

  • mexnc-2.0.14-a64-R14

  • netcdf_toolbox-1.0.8

  • snctools-2.0.16

Commercial toolboxes

  • MATLAB Compiler

  • Chemical engineering

  • Communication Toolbox

  • Communication blockset

  • Control Systems Toolbox

  • Data aquisition Toolbox

  • DSP blockset

  • Fuzzy Logic Toolbox

  • Image Processing Toolbox

  • Image Aquisition Toolbox

  • Instrument control Toolbox

  • Mapping toolbox

  • Neural Networks Toolbox

  • Non-linear Control Toolbox

  • Optimization Toolbox

  • PDE toolbox

  • Real-Time workshop

  • Real-Time Windows target

  • Robust Control Toolbox (inkl My-analysis och LMI control)

  • Signal Processing Toolbox

  • Simmechanics Toolbox

  • SimPowersystems

  • Simulink

  • Spline Toolbox

  • Stateflow Toolbox

  • Statistics Toolbox

  • Symbolic Math toolbox

  • System Identification Toolbox

  • Virtual reality Toolbox

  • Wavelet toolbox

  • xPC Target

If you wish to have some specific software installed, contact support@nsc.liu.se or smhi-support@nsc.liu.se (only for SMHI users) specifying Tornado + relevant information in the subject header.

Environment

We use module to handle the environment when there exists several versions of the same software installed, e.g. different compilers. This application sets up the correct paths to the binaries, man-pages, libraries, etc. for the currently selected module. Here is a list of the most useful arguments to module:


module

lists the available arguments


module list

lists currently loaded modules


module avail

lists the available modules for use


module load example

loads the environment specified in the module named example


module unload example

unloads the environment specified in the module named example

A default environment is automatically declared when you log in. Check using module list

Example: Switch from the default 64-bit icc compiler to a 32-bit icc compiler:

Example:
[perl@tornado ~]$ module list
Currently loaded modules:
  1) icc
  2) ifort
  3) mkl
  4) pbs
  5) scali
  6) matlab
  7) dotmodules
  8) default

[perl@tornado ~]$ module avail

In directory /etc/cmod/modulefiles:

  +default                     -ifort/8.1.032_EM64T
  +dotmodules                  -ifort/8.1.033_EM64T
  -icc/8.1.027_EM64T           -ifort/8.1.035_IA32
  -icc/8.1.029_EM64T           -ifort/9.0.024_EM64T
  -icc/8.1.030_IA32            -ifort/9.0.024_IA32
  -icc/8.1.032_EM64T           -ifort/9.0.027_EM64T (def)
  -icc/8.1.033_EM64T           -ifort/default
  -icc/8.1.033_IA32            -matlab/default
  -icc/8.1.035_IA32            -mkl/7.2.1p3 (def)
  -icc/9.0.023_EM64T           -mkl/7.2.1p3_IA32
  -icc/9.0.023_IA32            -mkl/default
  -icc/9.0.026_EM64T (def)     -pbs/default
  -icc/default                 -root
  -ifort/8.1.026_IA32          -scali/default
  -ifort/8.1.027_EM64T         -totalview/7.1 (def)
  -ifort/8.1.029_EM64T         -totalview/default
  -ifort/8.1.029_IA32

[perl@tornado ~]$ module unload icc
[perl@tornado ~]$ module load icc/8.1.030_IA32
[perl@tornado ~]$ module list
Currently loaded modules:
  1) ifort
  2) mkl
  3) pbs
  4) scali
  5) matlab
  6) dotmodules
  7) default
  8) icc/8.1.030_IA32

Tip: The environment is specified in the files located under /etc/cmod/modulefiles.

Resource Name Environment Variable

If you are using several NSC resources and copying scripts between them, it can be useful for a script to have a way of knowing what resource it is running on. You can use the NSC_RESOURCE_NAME variable for that:

[username@tornado ~]$ echo "Running on $NSC_RESOURCE_NAME"
Running on tornado

Getting an account on Tornado

  1. Contact support@nsc.liu.se or smhi-support@nsc.liu.se (for SMHI users only) with topic “New account on Tornado” or similar. We already have a list of people who are authorized for an account (given by our contact persons from Rossby and MISU - Anders Ullerstig and Michael Burger).

  2. Supply the following information: Name, address, e-mail and telephone number.

  3. We will send an e-mail back containing your account information + password. Please change this password as soon as possible

  4. Read the security section below.

Getting a grid certificate

In order to get access to NSC's file storage you must have a valid grid certificate. The correct procedure to get a certificate is described at (please read both "User Guide - Introduction" and "Getting a Grid Certificate"):

http://www.nsc.liu.se/systems/storage.html#User Guide - Introduction

The layout of the storage system is also described and there is also a detailed tutorial of how to use the storage system including a lot of examples.

Accessing Tornado

Log in to Tornado using ssh:

ssh username@tornado.nsc.liu.se

Use scp and sftp for transferring of files between Tornado and other computers:

Example: copy the file named example to Tornado:

scp ./example user@tornado.nsc.liu.se:~/documents

Example: connect to Tornado using sftp:

sftp user@tornado.nsc.liu.se
Connecting to tornado.nsc.liu.se...
sftp>


Security




  • Use different passwords for different systems (Very important). When a system is compromised and passwords stolen, the thing that causes the most grief is when the stolen password can be used for more than one system. A user that has accounts on many different computers and gets his/her shared password stolen will allow the intruders to easily cross administrative domains and further compromise other systems.

  • Do not share user accounts with other people. If they are eligible for an account on Tornado they will have to apply for an account of their own, if not, they should not be able to use Tornado.

  • Do not use weak passwords. Not using a trivial password (your name, account, dogs name, etc.) is VERY important. However, using really hard passwords it not even close to as important as not sharing them between systems.

  • Avoid chains of ssh sessions (see figure above). To login to a system and then continue from that system to a third (as illustrated above) is bad and should be avoided.

  • Check “Last login: DATE from MACHINE”. When logging into a system it only takes half a second to read the “last login” information. If you can't verify the information, contact support@nsc.liu.se / smhi-support@nsc.liu.se as soon as possible.

SSH public-key authentication

There is an alternative to traditional passwords. This method of authentication is known as key-pair or public-key authentication. While a password is simple to understand (the secret is in your head until you give it to the ssh server which grants or denies access), a key-pair is somewhat more complicated.

Our recommendation is to use whichever method you feel comfortable with. If you invest some time to learn about key-pairs you will receive several benefits, including better security and easier work flow.

A key-pair is as the name suggests a pair of cryptographic keys. One of the keys is called the private key (this one should be kept secure and protected with a pass phrase) and a public key (this one can be passed around freely as the name suggests).

After you have created the pair, you have to copy the public key to all systems that you want to ssh to. The private key is kept as secure as possible and protected with a good pass phrase. On your laptop/workstation you use a key-agent to hold the private key while you work.

  • Can be much more secure than regular password authentication

  • Can be less secure if used incorrectly (understand before use)

  • Allows multiple logins without reentering password/pass phrase

  • Allows safer use of ssh chains

How to use SSH public-key authentication instead of regular password authentication is described in chapter 4 in SSH tips, tricks & protocol tutorial by Damien Miller.

Short description of the necessary steps involved using SSH public-key authentication (read Damien Miller's guide above for more details):

  • Generate a key-pair on your desktop/laptop, choose a good pass phrase and make sure private key is secure (once).

  • Put your public key into ~/.ssh/authorized_keys on desired systems.

  • Load your private key into your key-agent (ssh-add with OpenSSH).

  • Run ssh all you want without reentering your pass phrase and without the risk of anyone stealing your password.

The Batch Queueing System

In order to utilize the cluster as efficient as possible a batch queueing system is used (Torque). Instead of running your applications interactively you submit “jobs” to the batch queuing system which automatically starts your job when there are resources available. A job can have several attributes but among the most important ones are:

  • The number of compute nodes that the applications should use.

  • The maximum allowed time for the application to run.

This makes it easier to ensure that the cluster is not idling at any time and we can easily specify that no more than one job at a time may use a node, preventing inefficient task switching and swapping. By also using an job scheduler (Moab) we can also easily implement policies, that allows us to prioritize between different types of jobs.

Tip: A graphical overview of the scheduling status is available at:

http://www.nsc.liu.se/cgi-bin/tornadostatus

Submitting Jobs

For a quick introduction how to compile and run jobs see Quick getting started example

There are two ways to submit jobs to the batch queue system, either as an interactive job or as a batch job. Interactive jobs are most useful for debugging as you get interactive access to the input and the output of the job when it is running. But the normal way to run the applications is by submitting them as batch jobs since otherwise you would have to manually start your application.

qsub is the name of the command for submitting jobs, either if it is a batch job or an interactive job.

Important: qsub takes a submit script as a parameter and not standard in or any binary application. The PBS-script must end its lines with a newline (\n) as is the default on Unix platforms. On Windows, line-endings are terminated with a combination of a carriage return (\r) and a newline(\n) - this will not work with qsub.

Example of useful arguments to qsub (more important arguments are marked bold), read the man page for additional arguments and details:


-I

Run the job interactively.


-j oe

Join standard out (o) and standard error (e) to the same file. As default the standard out and standard error is saved in two different files.


-l nodes=n:ppn=2

n specifies the number of nodes to run the job on, ppn=2 specifies that both the processors on each node will be used (if not specified only one will be used. n is an integer in the interval [1,129].


-l walltime=hh:mm:ss

Your estimation of the maximum execution time for the job.


-m abe

Send mail to the local user@tornado.nsc.liu.se when the job begins (b), exits (e) and aborts (a). You may combine these as you wish. If not specified, mail is only sent when the job is aborted.


-M email1@smhi.se,email2@misu.se

List of e-mail addresses to send mail to. If not specified mail is sent to the local user@tornado.nsc.liu.se


-N myjobname

Name of the job, consisting of up to 15 printable, non white space characters with the first character alphabetic. If not specified the default name will be jobid.tornado.

Batch Jobs

Two sample PBS-scripts are available for download: pbssample.simple and pbssample.advanced.

  1. Create a PBS-script. This is a shell-script with additional declarations for the arguments to qsub, the arguments to qsub are declared as #PBS pbsargument, e.g. -N myjobname is specified as #PBS -N myjobname.

    Example of a PBS-script named pbssample.simple using 24 nodes and is submitted to riskjobb. The wall clock time is 10 minutes and an e-mail will be sent when the job exits normally or exits with an error:

    #!/bin/sh
    
    # Request 24 nodes using both processors for the job and request 10 minutes of wall-clock time.
    #PBS -l nodes=24:ppn=2,walltime=00:10:00
    
    # Request regular output (stdout) and error output (stderr) to the same 
    # file.
    #PBS -j oe
    
    # Send mail when the job exits normally (e) or aborts with an error (a).
    #PBS -m ae
    
    # Goto the directory from which you submitted the job.
    cd $PBS_O_WORKDIR
    
    # Start the job with mpprun on the nodes that the batch queue system have
    # allocated for your job.
    mpprun helloworld
  2. Submit the job, by specifying the PBS-script as the only argument to qsub:

    [perl@tornado example]$ qsub pbssample.simple
    5723.tornado
  3. Check the status of the job with qstat. by specifying qstat -n you get information about allocated nodes. For even more details add -f as an argument. You should also try using showq.

    [perl@tornado example]# qstat
    Job id           Name             User             Time Use S Queue
    ---------------- ---------------- ---------------- -------- - -----
    5723.tornado       pbssample.simpl  perl             00:00:00 R riskjobb  
    
    
    [perl@tornado example]# qstat -n
    
    tornado: 
                                                                Req'd  Req'd   Elap
    Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
    --------------- -------- -------- ---------- ------ --- --- ------ ----- - 
    5724.tornado      perl     riskjobb pbssample.    --   24  --    --  01:00 R 00:00
       n72/0+n71/0+n70/0+n69/0+n68/0+n67/0+n66/0+n65/0+n64/0+n63/0+n62/0+n61/0
       +n60/0+n59/0+n58/0+n57/0+n56/0+n55/0+n54/0+n53/0+n52/0+n51/0+n50/0+n49/0  
  4. Since we specified -m ae, an e-mail will be sent to the local user at tornado when/if the job exits and aborts. The standard out and standard error is saved in the directory from where the job was submitted as pbssample.simpl.o5723

Interactive Jobs

Submitting an interactive job is done at the command line by adding the argument -I to qsub.

Example: Interactive version of the PBS-script in Batch Jobs


[perl@tornado perl]$ qsub -I -lnodes=24,walltime=00:10:00
qsub: waiting for job 5972.tornado to start
qsub: job 5972.tornado ready

[perl@n11 perl]$ cd $PBS_O_WORKDIR
[perl@n11 perl]$ mpprun hello_world.icc.scampi

The output are displayed directly to the shell prompt.

Alternatively, you may run mpprun directly if you want to run a MPI application. mpprun will make an implicit call to qsub with a default wall time of 1 hour and automatically execute the application.

Example: Interactive job by calling mpprun directly.

[perl@tornado example]$ mpprun -np 24 hello_world
Using PBS to spawn an interactive job
spawn /usr/pbs/bin/qsub -I -lwalltime=1:00:00,nodes=24
qsub: waiting for job 5977.tornado to start
qsub: job 5977.tornado ready

[perl@n11 perl]$ exec /usr/local/bin/mpprun -Norder=defult hello_world

Automatic cleanup and file stage in/out

  1. When a job is finished, the batch queueing system kills all running processes belonging to the user that owns the exiting job on the compute node.

  2. Before a new job is started all contents of /disk/local is removed on the compute node.

If you want to be guaranteed not to loose what you have written to /disk/local you should use the PBS functionality called stage out and copy your data to another file system (/home or /nobackup/) when your job exits. Since a job can exit in several ways (normally, segmentation fault, killed by pbs) you can not otherwise be guaranteed that your data always will be copied if you do this by yourself (e.g. in your application or in the PBS-script).

  • File stage out only can by done from your master node (i.e. your highest ranking node in the job).

  • PBS file stage in is not available on tornado, but this may as well be done manually in the PBS script or in your application.

Please make sure that it works before using file stage out in your production jobs. See 'man qsub' for more details.

Example: copy /disk/local/data0 and /disk/local/data1 (output written from “mpiapplication”) from the master compute node in my job to tornado:/home/perl when the job exits.

#!/bin/sh
#PBS -l nodes=16:ppn=2,walltime=01:00:00 
#PBS -W stageout=/disk/local/data0@tornado:/home/perl/,/disk/local/data1@tornado:/home/perl/

#Manual stage in:
cp /home/perl/tmp/test.gz /disk/local/

mpprun mpiapplication



Frequently used commands

Read the man-pages for more information about each listed command below. Commonly used commands are marked bold.

Frequently used PBS user commands:
qsub

Submits a job to the PBS queuing system.

qstat

Show status of PBS batch jobs.

qdel

Delete a PBS job from the queue.

Less frequently used PBS user commands:
qalter

Modifies the attributes of a job.

qhold

Requests that the PBS server place a hold on a job.

qrerun

Reruns a PBS batch job.

qrls

Release hold on PBS batch job.

qsig

Requests that a signal be send to the session leader of a batch job.

Moab commands:

Moab is the scheduler. It takes care of how the jobs are prioritized, and where these jobs are run. Moab supports advance reservations, QOS levels, backfill, and allocation management.

showq

List all jobs visible to the scheduler.

showbf

Show resources available for immediate access.

showres

Show current reservations.

showstart

Makes a qualified guess about when a job will start.

checkjob

Display numerous scheduling details for a job.

Scheduling policy

  • The maximum allowed walltime is 7 days. This is specified as -lwalltime=168:00:00 in PBS.

Bonus

On Tornade we use the same bonus system as on Monolith. Its purpose is to lower the priority of projects and users that has consumed their alloted time. Jobs from bonus users are only scheduled whenever there is no other, normal priority, job to run. Use projinfo to check the amount of time your group has used the current month.

Example output when using projinfo for a user belonging to the group rossby:

tornado 1% projinfo

Project        Used[h]        Allocated[h]
  User
-----------------------------------------
rossby        38947.1              148608
  sm_wyser     2628.2
  sm_aulle     2210.4
  sm_meier    31504.1
  sm_psamu     2604.5

Interactive development slots

Between 0700 – 1800 on Monday to Friday we have reserved 16 nodes for shorter jobs with a walltime of 1 hour or less (see reservation with command showres or look at: http://www.nsc.liu.se/cgi-bin/tornadostatus). This will make it much easier to develop and test applications without having to wait a long time for some other large jobs to finish.

Storage & File systems

On Tornado there are different types of storage classes depending on level of reliability, size, speed, location, etc.

A detailed description of NSC's file storage area is available at:

http://www.nsc.liu.se/systems/storage.html

This includes a description of its layout, how to get access, and a tutorial on how to use it.

Name

Type

Size

Reliability

Description

/home

nfs

1000GB

backed up

Your home directory. Writing large amount of data here will affect all logged in users. Store source code and vital data that must be backed up.

/nobackup/rossby1,

/nobackup/rossby2,

...

/nobackup/misu1

/nobackup/misu2

...

nfs

2-4 TB each

not backed up



If your job writes a large amount of data to disk you should use this area. Move vital data to /home or make a copy on another file system since it is not backed up.

Additional storage will be added by time. Check available file systems with the command df

Each file system is 2 – 4 TB large and consists of 1 or 2 RAID5 volumes of 6 – 7 harddrives each.


/disk/local

local hard drive to each compute node

55GB

not backed up

Should be used for temporary storage of your job output data. Fastest area since it is not shared between any other nodes. All contents on /disk/local is erased when the next job starts on the node. Use the stage-out facility in PBS to automatically copy data to another file system when job exits.

n/a

hierarchical file archive

Magnitude of ~100s of TB total

not backed up

NSC long term storage. Large area. Long access time. Accessible using migftp or globus-url-copy. It is described at:

http://www.nsc.liu.se/systems/storage.html



  • Every user has at least one directory under /nobackup. To check which, use the command “ls -d /nobackup/*/yourusername”. Example:

[perl@tornado ~]$ ls -d /nobackup/*/perl
/nobackup/rossby1/perl /nobackup/rossby2/perl



  • On /home and /nobackup/* there is a file called diskinfo containing each users total disk usage (+ previous day in diskinfo.old). This file is updated every night and is much quicker to use than du. Example:

[root@tornado ~]# sort -nk1 /home/diskinfo 


Diskinfo for file system/directory /tornado_home
[Measures in Megabytes of file space]
Thu Oct  6 06:05:01 CEST 2005
1       andro
1       msa
1       slurm
1       sm_doesc
1       sm_stran
1       sm_uhans
1       tmp
1       x_jenma
2       sp
14      perl
24      x_heiko
549     x_gorbr
985     sm_wyser
1457    sm_meier
3550    cap
4699    x_jenbr
9600    x_steso
27365   faxen
29267   sm_aulle
155367  sm_psamu



Compiling

For a quick introduction how to compile and run jobs see Quick getting started example.

GCC and both 64-bit and 32-bit Intel Compiler for Linux 8 and 9 are installed

We recommend using the 64-bit Intel compilers version 9 (used as default).

Note that since you may run both 64-bit binaries and 32-bit binaries on CentOS 4 the corresponding libraries are installed in different locations (/lib64 and /usr/lib64 for the 64-bit versions and /lib /usr/lib for the 32-bit versions).



Compiling MPI applications

NSC Cluster Environment (NCE) is NSC's modification of GCC and Intel compilers which makes it easier to compile MPI-applications. Rather than having to specify a quite long list of libraries to be linked which differs between different MPI implementations, you only have to compile with these additional arguments:


-Nscampi

Compile with ScaMPI support.

The options should be used both at compile-time (to specify the path to the include files) and at link-time (to specify the correct libraries).

Example: compiling the MPI-program, mpiprog.c with icc for ScaMPI (the same goes for gcc, g++, g77 and ifort):


icc mpiprog.c -o mpiprog -Nscampi



Intel compiler, useful compiler options

Below are some useful compiler options, please do "man ifort" or "man icc" for more. For additional information, see:

a) Optimisation

There are three different optimization levels in Intel's compilers:


-O0

Disable optimizations.


-O1,-O2 

Enable optimizations (DEFAULT).


-O3

Enable -O2 plus more aggressive optimizations that may not improve performance for all programs.

A recommended flag for general code is -O2 -mp and for best performance "-O3 -xP" which will enable software vectorisation. As always however, aggressive optimisation runs a higher risk of encountering compiler limitations.

b) Debugging


-g

Generate symbolic debug information.


-traceback

Generate extra information in the object file to allow the display of source file traceback information at runtime when a severe error occurs.


-fpe<n>

Specifies floating-point exception handling at run-time.


-mp

Maintains floating-point precision (while disabling some optimizations).

c) Profiling


-p

Compile and link for function profiling with UNIX gprof tool.

d) Options that only apply to Fortran programs


-assume byterecl

Specifies (for unformatted data files) that the units for the OPEN statement RECL specifier (record length) value are in bytes, not longwords (four-byte units). For formatted files, the RECL unit is always in bytes.


-r8

Set default size of REAL to 8 bytes.


-i8

Set default size of integer variables to 8 bytes.


-zero 

Implicitly initialize all data to zero.


-save

Save variables (static allocation) except local variables within a recursive routine; opposite of -auto.


-CB

Performs run-time checks on whether array subscript and substring references are within declared bounds.

e) Large File Support (LFS)

To read/write files larger than 2GB you need to specify some flags at compilation:

Fortran: no additional flags needed.

CC/C++: LFS is obtained by specifying the flags below when compiling and linking:


-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 

f) Miscellaneous options

Little endian to Big endian conversion in Fortran is done through the F_UFMTENDIAN environment variable. When set, the following operations are done:

  • The WRITE operation converts little endian format to big endian format.

  • The READ operation converts big endian format to little endian format.


F_UFMTENDIAN = big 

Convert all files.


F_UFMTENDIAN ="big;little:8" 

All files except those connected to unit 8 are converted.

g) NCE options (locally supplied by NSC)


-Nscampi

Compile with ScaMPI support.

Math libraries

Intel Math Kernel Library

MKL Version 7.2 is installed on Tornado.

For complementary information look at Intel® Math Kernel Library 7.2 for Linux* Technical User Notes. Additional documentation is available locally on Tornado at: /usr/local/intel/l_mkl_p_7.2.1.003/mkl721/doc/

The Math Kernel Library includes the following groups of routines:

  • Basic Linear Algebra Subprograms (BLAS):

    • vector operations

    • matrix-vector operations

    • matrix-matrix operations

  • Sparse BLAS (basic vector operations on sparse vectors)

  • Fast Fourier transform routines (with Fortran and C interfaces)

  • LAPACK routines for solving systems of linear equations

  • LAPACK routines for solving least-squares problems, eigenvalue and singular value problems, and Sylvester's equations

  • Vector Mathematical Library (VML) functions for computing core mathematical functions on vector arguments (with Fortran and C interfaces).

Full documentation can be found at http://www.intel.com/software/products/mkl/

Directory structure

mkl is located in $MKL_ROOT, defined at login. Semantically, MKL consists of two parts: LAPACK and processor specific kernels. The LAPACK library contains LAPACK routines and drivers that were optimized as without regard to processor so that it can be used effectively on processors from Pentium to Xeon. Processor specific kernels contain BLAS, FFTs, CBLAS, VML that were optimized for the specific processor. Threading software is supplied as a separate dynamic link library - libguide.so, when linking dynamically to MKL.

Linking with MKL

To use LAPACK and BLAS software you must link two libraries: LAPACK and one of the processor specific kernels (i.e. libmkl_p4n on tornado). Please use -L$MKL_ROOT instead of hardcoding the path (e.g. do not use -L/usr/local/intel/l_mkl_p_7.2.008/mkl72/lib/32). This will ensure that correct libraries are used when switching modules between different mkl versions.

Example (LAPACK library, Xeon EM64T processor kernel):


ld myprog.o -L$MKL_ROOT -lmkl_lapack -lmkl_p4n

Example (Dynamic linking. DLL dispatcher will load the appropriate dll for the processor dynamic kernel):


ld myprog.o -L$MKL_ROOT -lmkl -lguide -lpthread

Example (Dynamic linking using ifort):


ifort -L$MKL_ROOT -lmkl_lapack -lmkl_p4n example.o -o example

Quick getting started example

All the files in this example are available here.

  1. Get an account on Tornado.

  2. Log in to Tornado using ssh:

    ssh username@tornado.nsc.liu.se
  3. Compile an MPI application using the preferred Intel compiler (see Compiling for more details):

    icc -Nscampi helloworld.c -o helloworld
  1. Run the application as a batch job (see Submitting jobs for more details):

    1. Create a PBS-script, e.g. pbsexample. This file contains information about how many nodes you wish to use, how long you expect the job to run, how to start the application.

    2. Submit the job.

      qsub pbsexample
    3. When the job is finished, an e-mail will be sent to the e-mail address specified in ~/.forward.

For more detailed information, read Compiling and The Batch Queueing System.

Recommendations

  • Use mpprun to start your MPI applications.

  • No world writable home-directories, if you need this, create world writable subdirectories instead

  • If you want your job to start fast, use a wall clock time that is less than or equal to 1 hr since there is 16 nodes reserved for jobs less than, or equal to 1 hr.

  • Use projinfo to look at how much cpu time your group has used the current month

  • Under /nobackup/*/ and /home there is a file called diskinfo which contains disk usage for each user. Example: sort -nk1 /nobackup/rossby2/diskinfo

  • To plot data have a look at octave or gnuplot.

  • A status overview of Tornado is available at status.nsc.liu.se

  • Use the local disks (/disk/local/) for temporary storage instead of using the NFS mounted area /home. /home consists of two RAID 0 disks, it is globally mounted over NFS to every compute node and will perform poorly compared to the local disks attached to the compute nodes. Notice that /disk/local are not backed up and should be considered as temporary scratch areas.

  • File transfer is available using scp and sftp.

  • Backup of /home is made every night.

  • For questions, contact support@nsc.liu.se or smhi-support@nsc.liu.se (only for users from SMHI).






Page last modified: 2009-05-25 11:06
For more information contact us at info@nsc.liu.se.