User Area
User support
Guides, documentation and FAQ.
Getting access
Applying for projects and login accounts.
We provide some utilities that make it easier to compile, link and run programs at NSC systems. These are the build environments, compiler wrappers, the parallel job launcher mpprun, and the inspection tool dumptag. The purpose of these utilities is to help the handling of different mpirun-type programs and library paths somewhat transparently. Specifically, what they do is to add library paths to the 'rpath' of the binary so they can be found at runtime without, e.g., setting $LD_LIBRARY_PATH, and they 'tag' the binary so that at runtime mpprun knows which mpi program to start the binary with.
To build software at NSC you should normally first load a build environment module. This gives you access to compilers, an mpi library, and some core numerical libraries (for Intel build environments, Intel MKL). If you try to run the system gcc (e.g., simply issuing gcc or make) without first loading a build environment module you get an error message that points you to loading an appropriate module. NSC suggest using the recommended version of the Intel build environment, but there also is a gcc-based one if you need it.
To load the intel build environment, first do as follows to find out the recommended version:
[rar@tetralith1 ~]$ module load buildenv-intel
*************** NO MODULE LOADED *****************
***** Please also specify desired version ********
This is a notice to inform you that NSC has deprecated the use
of default modules for this software. You will now instead need
to specify version number/name as well.
The recommended version of build-env intel is 2018a-eb, i.e., add it with
module load buildenv-intel/2018a-eb
Lmod has detected the following error: No module loaded.
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
buildenv-intel/recommendation /software/sse/modules/buildenv-intel/recommendation.luaFrom reading that message, one now knows to load the recommended version as:
[rar@tetralith1 ~]$ module load buildenv-intel/2018a-ebThis command gives access to compilers and libraries for building software. For Intel compilers it is recommended to use:
For C: mpiicc (note the double i signifies the mpi-wrapped icc compiler.) or for non-MPI software icc.
For C++: mpiicpc or for non-MPI software icpc.
For Fortran: mpiifort or for non-MPI software ifort.
(Note: there is normally no harm in using the mpi-wrapped compilers even for non-MPI software.)
Let us say that you have an mpi c program called hello_world_mpi.c with no complicated dependencies. You can build this as follows:
[rar@tetralith1 ~]$ module load buildenv-intel/2018a-eb
[rar@tetralith1 ~]$ mpiicc -o hello_world_mpi hello_world_mpi.c
[rar@tetralith1 ~]$ ls
hello_world_mpi hello_world_mpi.cIf you are using the buildenv-gcc build environment, e.g.
[rar@tetralith1 ~]$ module load buildenv-gcc/2018a-ebThe relevant compilers are:
For C: mpicc or for non-MPI software gcc.
For C++: mpic++ or for non-MPI software g++.
For Fortran: mpifort or for non-MPI software gfortan.
Note: If you wish to switch between intel and gcc buildenvs it is important to module purge before loading the new buildenv. This helps to ensure a clean switch of environments.
Many software packages require satisfaction of a number of dependency libraries before they can be built. On Tetralith and Sigma we have build environments that supply a large number of such dependency libraries (see info below). On other systems there are some select dependency libraries and tools available to be found as usual with module avail. However, in most cases you will have to build the needed libraries yourself using the provided compilers.
On Tetralith and Sigma NSC provides two types of build environment modules:
Build environments ending in -bare only come with access to a bare minimum of dependency libraries (for the Intel compilers this is typically what is distributed alongside the compilers by Intel), everything else you need to build yourself.
Build environements ending in -eb have been created using the EasyBuild buildtool. Upon loading these modules, a subsequent module avail will show a number of modules for dependencies and tools that can significantly help to build software.
NSC recommends using an -eb build environment.
The following example shows the loading of an -eb build environment. First load the module. Notice the message that points out that upon loading this module you get access to dependency libraries.
[rar@tetralith1 ~]$ module load buildenv-intel/2018a-eb
***************************************************
You have loaded a buildenv module
***************************************************
The buldenv-intel module makes available:
- Compilers: icc, ifort, etc.
- Mpi library with mpi-wrapped compilers: intel mpi with mpicc, mpifort, etc.
- Numerical libraries: intel MKL
It also makes a set of dependency library modules available via
the regular module command. Just do:
module avail
to see what is available.
NOTE: You should never load build environments inside submitted jobs.
(with the single exception of when using supercomputer time to compile code.) Then check what dependency libraries you have access to:
[rar@tetralith1 ~]$ module avail
---- /software/sse/easybuild/prefix/modules/all/MPI/intel/2018.1.163-GCC-6.4.0-2.28/impi/2018.1.163 -----
ABINIT/8.8.2-nsc1 SCOTCH/6.0.4-nsc1
Amber/16-AmberTools-17-patchlevel-8-12-nsc1 Siesta/4.0.1-nsc1
Boost/1.66.0-Python-2.7.14-nsc1 Siesta/4.1-b3-nsc1
CDO/1.9.2-nsc1 UDUNITS/2.2.25-nsc1
CGAL/4.11-Python-2.7.14-nsc1 X11/20171023-nsc1 (D)
...
------------------- /software/sse/easybuild/prefix/modules/all/Compiler/GCCcore/6.4.0 -------------------
CMake/3.9.1-nsc1 Python/2.7.14-bare-nsc1 gperf/3.1-nsc1
CMake/3.9.4-nsc1 SQLite/3.21.0-nsc1 intltool/0.51.0-Perl-5.26.0-nsc1
CMake/3.9.5-nsc1 Szip/2.1.1-nsc1 libdrm/2.4.88-nsc1
CMake/3.10.0-nsc1 Tcl/8.6.8-nsc1 libffi/3.2.1-nsc1
...
--------------------------------- /home/rar/EasyBuild/modules/all/Core ----------------------------------
Anaconda2/5.0.1-nsc1 (D) gettext/0.19.8.1-nsc1 ncurses/6.0-nsc1
...
---------------------------- /software/sse/easybuild/prefix/modules/all/Core ----------------------------
Anaconda2/5.0.1-nsc1 GCC/6.4.0-2.28 (D) intel/2015a
Anaconda3/5.0.1-nsc1 GCCcore/6.4.0 (L) intel/2018a (L,D)
EasyBuild/3.5.3-nsc17d8ce4 (L) foss/2015a ncurses/6.0-nsc1
Eigen/3.3.4-nsc1 foss/2018a (D)
...
----------------------------------------- /software/sse/modules -----------------------------------------
ABINIT/recommendation (D)
ABINIT/8.8.2-nsc1-intel-2018a-eb
Amber/recommendation (D)
...
Where:
L: Module is loaded
D: Default Module
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".Many of the sections in the above output of the module avail commands were not visible until you loaded the buildenv module. Now that the module is loaded, you access these as usual with the appropriate module load commands.
For example, if you have a program called use_matheval.c that requires linking with the Matheval library using -lmatheval, you can do so by first loading the buildenv module as shown above, and then do this:
[rar@tetralith1 ~]$ module load libmatheval/1.1.11-nsc1
[rar@tetralith1 ~]$ mpiicc -o use_matheval use_matheval.c -lmathevalIn case the software that you compile has makefiles (or equivalent) which require you to specify paths to dependencies, the module makes those paths available as environment variables. The easiest way to discover this is by the module show command, where you can see exactly what variables are being set.
Here we investigate the libmatheval module:
[rar@tetralith1 ~]$ module show libmatheval/1.1.11-nsc1
------------------------------------------------------------------------------------
/software/sse/easybuild/prefix/modules/all/Compiler/GCCcore/6.4.0/libmatheval/1.1.11-nsc1.lua:
------------------------------------------------------------------------------------
help([[
Description
===========
GNU libmatheval is a library (callable from C and Fortran) to parse
and evaluate symbolic expressions input as text.
More information
================
- Homepage: http://www.gnu.org/software/libmatheval/
]])
whatis("Description: GNU libmatheval is a library (callable from C and Fortran) to parse
and evaluate symbolic expressions input as text.")
whatis("Homepage: http://www.gnu.org/software/libmatheval/")
conflict("libmatheval")
prepend_path("CPATH","/software/sse/easybuild/prefix/software/libmatheval/1.1.11-GCCcore-6.4.0-nsc1/include")
prepend_path("LIBRARY_PATH","/software/sse/easybuild/prefix/software/libmatheval/1.1.11-GCCcore-6.4.0-nsc1/lib")
prepend_path("PKG_CONFIG_PATH","/software/sse/easybuild/prefix/software/libmatheval/1.1.11-GCCcore-6.4.0-nsc1/lib/pkgconfig")
setenv("EBROOTLIBMATHEVAL","/software/sse/easybuild/prefix/software/libmatheval/1.1.11-GCCcore-6.4.0-nsc1")
setenv("EBVERSIONLIBMATHEVAL","1.1.11")
setenv("EBDEVELLIBMATHEVAL","/software/sse/easybuild/prefix/software/libmatheval/1.1.11-GCCcore-6.4.0-nsc1/easybuild/Compiler-GCCcore-6.4.0-libmatheval-1.1.11-nsc1-easybuild-devel")Note how the environment variable $EBROOTLIBMATHEVAL is made to points to the root of libmatheval. Here one finds, e.g., $EBROOTLIBMATHEVAL/lib to be the path of the compiled library files (.so and .a), and $EBROOTLIBMATHEVAL/include are the include files. As you can see, these paths are also added to the $LIBRARY_PATH and $CPATH environment variables, so they get picked up automatically during compile time. This is why in the example above we did not have to add -L "$EBROOTMATHEVAL/lib", as that path was already present in $LIBRARY_PATH. Also, note that the modules do not set $LD_LIBRARY_PATH. In contrast to how this works in many other supercomputer centers, NSCs compiler wrappers go through $LIBRARY_PATH and any paths it finds under /software are added to the rpath of the compiled binary, i.e., these libraries will be found at runtime without having to set $LD_LIBRARY_PATH.
Upon compiling a program you can use the dumptag utility to see how the end result is 'tagged' by the compiler wrapper. Simply do:
[rar@tetralith1 ~]$ dumptag hello_world_mpi
File name /home/rar/compiler-demo/simple-compile-mpi/hello_world_mpi
NSC tags ---------------------------------
Build date 180618
Build time 164441
Built with MPI openmpi 2_1_2__eb
Linked with gcc 2018a__eb
Tag version 5
------------------------------------------You can see that this is a binary built by gcc, and expects to be started using the openmpi mpirun-type program.
In case you desire interactive access to test and debug your compiled program, that can be done as interactive node jobs. More detailed information is available in the section about running applications.
For example, to reserve one interactive development node (i.e., 32 cores on Tetralith) and run your own compiled program mympiprogran, do:
[rar@tetralith1 ~]$ interactive -N1 --exclusive --reservation=devel -t 1:00:00
Waiting for JOBID 38222 to start
[rar@n76 ~]$ mpprun ./use_matheval
(... program output ...)
[rar@n76 ~]$ exit
[rar@tetralith1 ~]$ When your compiled program is ready to be used, you can do so with a submit script as if it was any other type of NSC software. Note that you should not load any build environment or dependency library modules in your submit script. If you have compiled your program according to the instructions above, that should not be necessary, all needed libraries should be found without doing so.
(Note: while NSC provided dependency libraries are handled by the compiler wrappers, the default behavior is to not do anything with libraries that you have compiled yourself, i.e., under your /home or /proj directories. However, you can change this behavior by setting the NSC_LD_FLAG environment variable. See below for more details on how the compiler wrappers work).
Here is an example submit script:
#!/bin/bash
#
#SBATCH -J myjobname
#SBATCH -t 00:30:00
#SBATCH -n 64
#
mpprun ./mympiappNote:
Use the mpprun command rather than mpirun or similar. The mpprun inspects your program and runs the appropriate mpirun-type command for you. More details about the mpprun command is available below.
The submit script should not contain anything like module load openmpi/1.5.4 mkl/11.1 or module load libmatheval/1.1.11-nsc1. (This is frequently necessary on other supercomputer centers.)
The compiler wrappers are currently enabled for the Intel and GNU compiler suites. The main feature of the compiler wrapper is to embed the paths to libraries in the binary using RPATH. So at runtime there is no need to set the LD_LIBRARY_PATH environment variable or load modules. The wrapper also provides a few extra compiler flags in addition to native compiler options to help to link programs using MPI or MKL or both.
The path to every library that you include from the /software/ directory is embedded, but not libraries from your own home directory. The underlying reason is that we assume that libraries that you have installed yourself are used for active software development, and you might want to exert full control of these. You can change this behavior by setting the NSC_LD_FLAG environment variable (see below).
The compiler wrapper is enabled by default when you load a compiler module. The name of the compiler wrapper is the same as the name of the original compiler. So for Intel Fortran name of the compiler wrapper is ifort, for GNU C compiler, gcc and so on. In most cases, the compiler wrapper will be transparent to users.
These are the extra arguments provided by the wrapper in addition to those provided by the underlying compiler:
-Nhelp Show help related to the compiler wrapper
-Nmkl Link with MKL
-Nmpi Link with MPI (but, better just use mpi<compiler> instead.)
-Nverbose Be more verbose about what the wrapper is doing.The wrapper also understands a few environment variables:
NSC_COMPILER_FLAG
Works same as the command line flags -N*
To compile an mpi program:
export NSC_COMPILER_FLAG="-Nmpi"
gcc mpitest.c ## for gcc compiler
NSC_LD_FLAG
0 no rpathing
1 rpath libraries in the /software/ (default)
2 rpath libraries in all folders except /usr /lib /lib64 /tmp /opt
example:
export NSC_LD_FLAG=0
gcc mpitest.c ## for gcc compiler
The program will be compiled with mpi but no rpathing will
be done
NSC_LD_EXTRA_LIBPATH
A colon separated list of additional paths for rpathing. If
NSC_LD_FLAG is set to 0 this flag will be ignored. NSC provides the MPI job launching tool called mpprun. We strongly recommend that you use mpprun instead of mpirun or similar to start an MPI job. The command mpprun should be available by default when you log in to a cluster.
As mentioned before, the main benefit of mpprun is that it handles differences in how to correctly start an MPI program automatically. If an MPI binary is built according to NSC recommendations, then mpprun can detect the correct MPI library, and the corresponding command to start an MPI job, e.g.mpirun or mpiexec.hydra. What happens then is that mpprun calls the native MPI launcher from within itself. If the OpenMP environment variable controlling the number of threads is unset when launching an MPI application with mpprun, mpprun will by default set OMP_NUM_THREADS=1. mpprun also writes some useful information about the job in the system log file. For the full list of mpprun options use mpprun --help.
Here is example job script using mpprun. It runs an MPI application on two nodes on Tetralith (64 cores):
#!/bin/bash
#
#SBATCH -J myjobname
#SBATCH -t 00:30:00
#SBATCH -n 64
#
mpprun ./mympiappMpprun also works in interactive sessions. Below, we ask for an interactive session on two nodes and test an MPI program:
[kronberg@tetralith1 mpi] interactive -n64 --exclusive -t 00:10:00 --reservation=devel
Waiting for JOBID 77079 to start
...
[kronberg@n1137 mpi]$ mpprun mpitest_c
mpprun INFO: Starting impi run on 2 nodes (64 ranks)...
Hello, world, I am 16 of 64
[...]
Hello, world, I am 31 of 64
Hello, world, I am 6 of 64
[kronberg@n1137 mpi]$ mpprun mpitest_c_openmpi
nberg@n1137 mpi]$
[kronberg@n1137 mpi]$ exit
[screen is terminating]
Connection to n1137 closed.
[kronberg@tetralith1 mpi]$The dumptag utility can be used to inspect which build information are encoded inside a binary, e.g. name and version of the MPI library, which MKL that was used, build date etc. The dumptag utility comes with mpprun, so you don't need load a module to have access to it.
Sample out of an MPI program that was compiled with Intel 12.1.4 compilers and Intel MPI 4.0.3.008, but without any MKL library:
[kronberg@tetralith1 mpi]$ dumptag mpitest_c -- NSC-tag
---------------------------------------------------------- File name:
/home/kronberg/mpi/mpitest_c
Properly tagged: yes
Tag version: 4
Build date: 121024
Build time: 131142
Built with MPI: impi 4_0_3_008
Built with MKL: no (or build in an unsupported way)
Linked with: intel 12_1_4
---------------------------------------------------------------------If you would have compiled with e.g. OpenMPI instead, it would have been visible in the "Built with MPI" section:
[kronberg@tetralith1 mpi]$ dumptag mpitest_c_openmpi
-- NSC-tag ----------------------------------------------------------
File name: /home/kronberg/mpi/mpitest_c_openmpi
Properly tagged: yes
Tag version: 4
Build date: 121024
Build time: 131446
Built with MPI: openmpi 1_6_2_build1
Built with MKL: no (or build in an unsupported way)
Linked with: intel 12_1_4
---------------------------------------------------------------------
[kronberg@tetralith1 mpi]$(This segment is mostly for advanced users, and applies to Tetralith and Sigma only.)
For users seeking help of a more automated build tool to build complex software, EasyBuild is available on Tetralith and Sigma. The EasyBuild website contains more information about EasyBuild.
To use EasyBuild, load the easybuild buildtool module, e.g.,:
[rar@tetralith1 ~]$ module load buildtool-easybuild/3.5.3-nsc17d8ce4This makes the command eb available. At NSC the default configuration of EasyBuild is to build software in your home directory under the subdirectory EasyBuild. This can, however, be easily changed by modifying $EASYBUILD_PREFIX after loading the easybuild module. Note that loading the easybuild module also sets a number of environment variables on the form $EASYBUILD_* that affects the behavior of EasyBuild (you can see them by module show buildtool-easybuild/<version>). The NSC setup is made to make easybuild able to use existing centrally built EasyBuild packages at NSC. However, note that NSC EasyBuild modules frequently uses different names than the standard ones, which means that you need to edit .eb files to point to the correct package names. (The reason NSC names differ is that we typically add an -nsc(number) suffix to versions to be able to upgrade builds without removing old versions, something that is necessary to not break existing compiled software when libraries are accessed using rpath.)
You can find prepared .eb files in the easybuild-easyconfigs repository.
EasyBuild at NSC comes with a tool, nsc-eb-helper.py with the purpose to simplify mass-edit big batches of dependency '.eb' files. However, this tool is still in an early stage of development and sparsely documented, so if mass-editing .eb dependencies is something you have a desire to do and you cannot get this tool to work, contact NSC support.
User Area
Guides, documentation and FAQ.
Applying for projects and login accounts.
National Supercomputer Centre
Linköping University
581 83 LINKÖPING
SWEDEN
Org.nr: 202100-3096
VAT.nr: SE202100309601
