Systems  
Status displays
System status
Retired systems «
 
 
 
 
 
 
 
 
 
 
 
 

Monolith FAQ

  • Bonus system

      The "bonus" accounting system is now running on the Monolith for academic users and for SMHI (including Rossby) users. This means that for each project, used CPU time is continuously verified against the allocated time . The first day of each month the usage for all projects is set to 0 since the project allocation is on a monthly basis and the usage does not carry over from month to month. However, if a job is partly done during one month and partly during the next month, the total time will be divided correctly between the two months. You can find out the latest usage statistics from your project by typing the command /usr/local/bin/projinfo

      The bonus system continuously surveys all active accounts on the system. When a project has used up the allocated CPU hours for the current month, this is detected by the bonus system and the following actions will take place:

      1. All members of the project will be informed (through email) that the allocated CPU hours for the current month are consumed and that jobs will only be executed when there is "free" time on the system.

      2. Jobs from the project (with no time left) is only executed when the usage of the system is below a certain threshold. This is done by queuing all batch requests from the project in the bonus queue. The bonus jobs are always stopped. When the load is light however, one bonus job will be eligible for execution.

      This has the dual benefit of making sure everyone gets his/her share of the resources as well as making it possible for users with no time left to use the system at off peak periods.

      Test project accounts generally will not be allowed to run under the bonus system. When the allocated hours are consumed for a particular month, no runs are therefore possible until the following month.

  • C

      Version 3.3 of the GCC compiler suite is now installed on Monolith. Do "module add gcc/3.3" to switch to that version. It doesn't support the -N switch as of now, though.

      Version 3.4 of the GCC compiler suite is now installed on Monolith. Do "module add gcc/3.4.0" to switch to that version.

  • Cache

      How much cache memory is available per processor on Monolith?

      First level cache: 8K (data)
      Second level cache: 512K

      http://www.geek.com/procspec/procspec.htm has a quite good overview of cache and speed of processors. The processors in Monolith are "P7-Xeon-Prestonia" with a clock frequency of 2.2 GHz.

  • Default shell

      The default shell is bash. The command chsh is available, but we have not provided any dot-files for the other shells.

  • Editors

      The editors emacs and vi are available on /usr/bin/, and now also xemacs is available.

  • Email

      Email is available. We recommend that all users have a forward from Monolith to your normal email address, and that you create a .forward file in your home directory where you put your normal e-mail address. An easy way of doing this is to edit a file .forward in your login-directory with the content 
      e-mail_address
      and a more advanced way is to simply type the command: 
      echo e-mail_address  > $HOME/.forward
      where in both alternatives e-mail_address should be substituted with your normal complete e-mail address. By doing this you will receive notifications at your normal e-mail address instead of locally on Monolith.

  • Error messages

      The meaning of the message 
      job 83609 violates active HARD MAXPS limit of 1 for user x_nscnn
      is that you can no longer run jobs longer than one second, i.e. you cannot run jobs on Monolith. Probably your project has ended and you do not any longer belong to any project on Monolith. You may still login to Monolith, to take home your files, but may not run jobs there.

  • File systems

      The Monolith system has two levels of file systems:
      • a local file system that is directly attached to each compute node, called /disk/local.
      • global file systems mounted through NFS, called /disk/global and /home. In order to improve performance we now have five /disk/global systems and two /home systems. We have added numbers to distinguish them. You will be asked to use a specified /disk/global and /home, for example /disk/global5 and /home.

      Performance for the global NFS mounted file systems is much less than for the local file system and if too heavily used also impacts overall system performance.

      So, in order to improve the situation we urge you to please use /disk/local for local files. This will improve the performance of your job and relieve the system.

      The rules for the /disk/local file system are simple:

      • 20 GB is always available for each job.
      • After the job is completed, the file system is cleaned so you need to copy data you like to save before the end of the job.
      • It is a purely local file system and can not be accessed from other nodes.

      In addition, we have modified the G98 execution scripts setting the environment variable "GAUSS_SCRDIR" to /disk/local. This means that G98 jobs are now forced to use /disk/local for scratch files (Gau-xxx.rwf, Gau-xxx.scr, Gau-xxx.int, Gau-xxx.d2e and Gau-xxx.inp).

  • File transfer

      File transfer with scp to and from Monolith is available. File transfer out with ftp is available, but not recommended.

      If you have problems with scp due to an old version at the receiving end you may use scp -oProtocol=1.

  • Fortran

      Intel (ifc) Fortran 90/95 as well as PGI 4.0 (pgf77 and pgf90) and g77 compilers are available on the system.

        /usr/local/intel/6.0_old/
  /usr/local/intel/6.0/
  /usr/local/intel/7.0/
  /usr/local/intel/7.1/
  /usr/local/pgi/4.0/
  /etc/cmod/modulefiles
      You switch between the Intel compilers with

      • switch from 7.1 to 6.0: 
        module unload intel; module load intel/6.0

      As soon as you log out you will have Intel 7.1 as default when you log back in.

      We have installed new versions of the Intel compilers.

      To use them or test them, please use the module system: 
      	module unload intel; module load intel/8.0
      The Fortran compiler has been profoundly changed, also in the name of the compiler and the names of option flags. The new compiler name is 'ifort'. (The old name 'ifc' may still be used.) Please look for more information about the compiler on URL = http://www.intel.com/software/products/compilers/flin/whatsnew.htm.

      Run-Time Error Messages are given in http://www.intel.com/software/products/compilers/flin/docs/f_ug1/ug1l_rt_errors.htm.

      Release notes and other documentation are located as files on Monolith:

      • Fortran: /usr/local/intel/8.0/l_fc_p_8.0.034/doc
      • C/C++: /usr/local/intel/8.0/l_cc_p_8.0.055/doc

      The Intel® Fortran Compiler 8.0 allocates more temporaries on the stack than previous Intel Fortran compilers. If a program has inadequate stack space at runtime, it will terminate with a Segmentation fault or Signal 11. The stack space can be increased with the ulimit -s unlimited command (in bash shell) or limit stacksize unlimited (in csh shell) on Linux.

      When using ifort (Intel 8.0) and MKL for linking the Lapack routine dgeev (with ifort test_dgeev.f90 -L$MKL_ROOT -lmkl_lapack -lmkl_p4 -lpthread -lguide -openmp) you may get the following error message: 

      "ilaenv.o(.text+0x52a): Undefined reference to 's_copy' "
      This is because libg2c is not linked automatically. Add the following: 
      -L/usr/lib/gcc-lib/i386-redhat-linux/2.96/ -lg2c
      See http://support.intel.com/support/performancetools/libraries/mkl/linux/link_error.htm for further information, including the reason.

      New Portland Group Compiler

      For the Portland compilers pgf77, pgf90, and pghpf the default limit for file size is 2 G Bytes. You can remove this limit through using the Large File Support routines by adding the switch -Mlfs to your compilation command.

      We have also installed a new version, 5.0-2, of the PGI compilers. To use them or test them, please use the module system: 

      	module unload pgi; module load pgi/5.0
      Release notes and other documentation (Fortran and C/C++) are located as files on Monolith: /usr/local/pgi/linux86/5.0/doc.

      With the command "module add gcc/3.3" you get the new version of g77.

      If you use scratch-files from Fortran we recommend that these are assigned names via FILE=filename in the OPEN statement.

  • Interactive jobs

      You get an interactive job on one node (two processors) four one hour with the command 

      qsub -I -lwalltime=1:00:00,nodes=1
      If you start a job on several nodes the environment variable PBS_NODEFILE will point to a file with all the nodes in your job.

      You will be automatically logged into the node when your interactive batch job starts. If nothing happens you may do "showstart" (from another window) on the batch job to see when it will start. During the day there are 8 nodes reserved for interactive jobs with a maximum time limit of 1 hour.

  • Login

      The login-command from Unix/Linux is ssh -l x_nscnn monolith.nsc.liu.se, where x_nscnn has to be replaced with your username ("account") on Monolith. The difference between telnet and ssh is that telnet prompts for the user name on the distant computer but ssh assumes it to be the same as on the local computer.

  • Plotting

  • Processors

      Both processors are now available on each compute node. There are 198 nodes available for users, meaning that there are 396 processors in total available for computational tasks. As an example, if you need 32 processors in your job, you allocate 
      	#PBS -l nodes=16:ppn=2
      Nodes on Monolith are allocated on a node basis. This means that you will be accounted for two processors on a node, even if you use only one. If you e.g. run parameter variations on a problem as one-processor jobs, it would be a good idea to run two variations in each job, thus using both processors.

      • Monolith will always be requested and allocated on a node basis. Example: 
          With one processor/node:     you need 32 processors ->
	#PBS -l nodes=32:ppn=1
  With two processors/node:    you need 32 processors ->
	#PBS -l nodes=16:ppn=2
      • How to write a run script to start two jobs (in separate directories) at the same time.

        Is the following script correct? 

         #PBS -l nodes=1:ppn=2
 cd /directory_one/
 ./aa
 cd /directory_two/
 ./aa
        That is almost correct, but if you do as shown above, the first "./aa" computation will go to completion before you start with the next, i.e. they will not compute in parallel. You have to "fork away/put in parallel/put in the background" at least the first of them by ending the command line with a '&' sign. This also means that you have to let the script "wait" for the completion of those computations, that you have forked away.

        Summary: 

        #PBS -l nodes=1:ppn=2
cd /directory_one/
./aa &
cd /directory_two/
./aa &
wait

  • Secure shell

      We have installed ssh both in to and out from the computer. It is on /usr/bin. Remember that if you have different user names on Monolith and your local machine you have to use the command ssh -l username machine_name.

      We have also installed scp and sftp.

      With some environments you may have problems with giving the password to ssh and scp. If that is the case, you may try to terminate the password character string with "Control-J" instead of the usual "Enter" or "Return".

      You may use the SSH crypto challenge authentication mechanism to avoid sending a not encrypted password.

  • Scampi Trace

      A simple way to trace the communications is to use SCAMPI_TRACE. We do not yet know how much the performance is degraded.

      A description is available in FAQ for ScaMPI.

      We tested interactively with the following script 

      qsub -I -l walltime=1:00:00 -l nodes=2:ppn=2
MPI_HOME=/opt/scali
export MPI_HOME
SCAMPI_TRACE='-b -v'
export SCAMPI_TRACE
cd mpi
/opt/scali/bin/mpimon a.out -- `cat $PBS_NODEFILE`

  • Software

      The final list of provided mathematics libraries is yet to be determined but SCALAPACK will soon be available:

      • BLAS (optimized and reference) on /usr/local/lib/
      • LAPACK on /usr/local/lib/
      • SCALAPACK

      If you need other libraries you can either build them yourself or ask for our assistance building and/or installing them globally on the system.

      Several chemistry application programs have now been installed on Monolith, including Gaussian 98 and Dalton 1.2, see the Chemistry page.

      For Gaussian you may use up to 850 MB with "shared memory" but only 512 MB with Linda.

      We also have the FLUENT 6.1 software available on Monolith, but only for users within Linköping University. FLUENT is a flow and heat transfer modeling software suited to a wide range of applications. For further information see the Software/Physics page.

      The programs gv and xmgrace are now available on the Monolith frontend login-1.

  • Stageout

      Question

      I used to call a script in a job file to run the code and after finishing to copy the output file into the home directory. It works properly if there is enough time to execute the code in the given time limit.

      However, if the job is killed by the queue system then not just the executing code is killed but the whole script is killed, therefore, the output files vanishes because they were in the volatile /disk/local directory.

      Reply

      To copy out a file after the job has aborted or terminated you can use the PBS stageout facility. Example: 
      #!/bin/sh
#PBS -lwalltime=1
#PBS -lnodes=1:ppn=2
#PBS -W stageout=/disk/local/file1@localhost:/disk/global/x_nscnn/file1
#PBS -W stageout=/disk/local/file2@localhost:/disk/global/x_nscnn/file2

cat >/disk/local/file1 <<EOF
This is file one
EOF
cat >/disk/local/file2 <<EOF
This is file two
EOF

sleep 10000
      In this example /disk/local/{file1,file2} will be copied to /disk/global/x_nscnn/ when the job is finished (or aborted because of time limit).

      There is a corresponding stagein facility. There are more information in the man-page for qsub on Monolith.

  • Status

  • TotalView 6.3

      The TotalView debugger is a source level debugger with a graphic user interface and features for debugging distributed programs, multiprocess programs, and multithreaded programs.

      Totalview can be used to debug "live" programs as well as postmortem debug on core files: 

      totalview [ filename [ corefile ]] [ options ]
      where "filename" specifies the name of an executable to be debugged and "corefile" specifies the name of a core file. The executable must be compiled with source line information (usually the -g compiler switch) in order to give full debug capabilities.

      On Monolith please note the following:

      • Postmortem debug can be used from the login node.
      • Live debug is only supported for interactive jobs and requires a special procedure since X-forwarding is not automatic through PBS:
        • Start an interactive job, for example: "qsub -I -lwalltime=1:00:00,nodes=4". This will create an interactive job and the environment variable PBS_NODEFILE will contain a list of the nodes you interactive job is running on.
        • In a separate window, do a "ssh" to one of the nodes in the interactive job. From this window you can now start "totalview executable" and do a live debug.
      • Options to the totalview command are described in the TotalView User's Guide. Online documentation is located at http://www.etnus.com/Support/docs/index.html.

  • Used time

      You can obtain info on how much time you have used the present month with the command projinfo.

      You run it on your service node on Monolith, normally the computer monolith.nsc.liu.se and it tells you how many CPU hours you have been accounted for during the current month. An example: 

      [y_user@login-1 y_user]$ projinfo

Project        Used[h]        Allocated[h]
  User
-----------------------------------------
p2004099       2313.2                5000
  y_user       2313.2
      This output means that the batch system until now has accounted your project 1590 CPU hours this month, one hundred of these hours for the jobs of x_user and 1490 CPU hours for the jobs of y_user.

      The accounting is accumulated at the end of each batch job, according to how many computing nodes you have asked for (please remember that each node has two CPUs, meaning that you are accounted for two CPUs for each node) and how long time the job actually run.

      An example on this, assuming the following definitions in your job script: 

      #PBS -l nodes=4:ppn=2
#PBS -l walltime=10:00:00
      If we also assume that the job completes after three of the requested ten hours, you are accounted for 24 cpu hours (4 * 2 * 3).

      If would be the same, if you requested nodes=4:ppn=1, because the system makes reservations only down to the granularity of nodes, not CPU:s. (This may change later.) And it does not matter if you really run your job on all those nodes and all those processors, you are still accounted for them all.

      As you see from the example, you are not accounted for making long reservations, only for how long time your job really runs. The down side with long reservations is that longer jobs get a lower priority in job scheduling, so you might wait longer for the job to start.

      The maximum wall clock run time for a job is 144 hours.

  • Vampir 4.0
      VAMPIR is a graphical tool for analyzing the performance and message passing characteristics of parallel programs that use the MPI message passing library.

      The VAMPIR package has two parts:

      1. Vampir-trace which is a library that you link in to your application. This will produce a trace file.
      2. Vampir is used to analyze the trace file.

      The full user documentation can be found at: /usr/local/tools/vampir/3.0/doc/

      Vampir-userguide.pdf for Vampir
      Vampirtrace-userguide.pdf for Vampir-trace

      or at /usr/local/tools/vampir/4.0/lam/doc/ or at /usr/local/tools/vampir/4.0/mpich/doc/.

      There are also man pages for Vampir and the Vampir-trace library routines.

      Follow these steps to start using Vampir:

      1. Compile and link your MPI code for tracing. Be sure to obey the order of the various libraries: libVT.a must be linked before the MPI library.

        Examples on linking with Vampir trace using Fortran:

        • With ScaMPI and Intel compiler: 
          ifc code.f -L$PAL_ROOT/lib -I/opt/scali/include -L/opt/scali/lib -lfmpi \
 -lVT -lmpi -lpthread -lPEPCF90 -ldwarf -lelf
        • With MPICH and Portland Group compiler: 
          pgf90 simple.f -I/usr/local/mpich-1.2.4/pgi-4.0/include -L$PAL_ROOT/lib \
 -L/usr/local/mpich-1.2.4/pgi-4.0/lib -lfmpich -lVT -lmpich -ldwarf -lelf
      2. Define the environmental variable VT_PROGNAME to be the name of the executable.
      3. Run the executable as usual. In addition to the usual output this will generate a VAMPIRtrace output file which will have the extension ".bvt".
      4. Analyze the resulting VAMPIRtrace output file by running "vampir", specifying the .bvt file: "vampir a.out.bvt" (make sure you have the DISPLAY environment variable defined).

      An excellent VAMPIR tutorial is available at http://www.arsc.edu/support/howtos/usingvampir.html.

      A note for existing Vampir users. The release 3.0 of Vampir has many new features, checkout http://www.pallas.com/e/products/vampir/index.htm for a short description. If you for any reason wish to use Vampir 2.5 you can do so by doing: 

      module unload vampir; module load vampir/2.5

      4.0 has some new features:

      • A new file format (STF) with scalability and compact format
      • Ability to trace LAM MPI applications as well as applications using the Global Array toolkit.

      Default for 4.0 is to have MPICH/ScaMPI applications as target.

      For LAM, do: 

      module unload vampir/4.0.mpich ; module load vampir/4.0.lam
      If you want to use Vampir/Vampirtrace 3.0: 
      module unload vampir/4.0.mpich ; module load vampir/3.0
  • Walltime
      The walltime limit of jobs is 144 hours, i.e. six days.





    Page last modified: 2004-07-16 11:47
    For more information contact us at info@nsc.liu.se.