WIEN2K Installations on Tetralith & Sigma

The WIEN2k installations are generally maintained by Weine Olovsson (weiol@nsc.liu.se).

"The program package WIEN2k allows to perform electronic structure calculations of solids using density functional theory (DFT). It is based on the full-potential (linearized) augmented plane-wave ((L)APW) + local orbitals (lo) method, one among the most accurate schemes for band structure calculations. WIEN2k is an all-electron scheme including relativistic effects and has many features."

To use the NSC installations one needs to be covered by a valid license, for more information see this link.

Note: for a single k-point there's at the moment no recipe how to run over several nodes.

Initial setup

There are some settings which are needed by WIEN2k. This can be done by running userconfig after loading the WIEN2k module, which will put the settings into .bashrc. For most purposes, you can provide the minimal settings:

export SCRATCH=./
export EDITOR="vi"

where the editor also can be set to e.g. "emacs". Without these settings e.g. setting up your calculation with init_lapw might crash.

Example batch script

A batch script for running WIEN2k may look like below, here running MPI over 32 cores on one Tetralith node (this can be useful if one got 1 k-point, also fewer cores might be enough, see benchmark at the end of the page):

#!/bin/bash
#SBATCH -A SNIC-xxx-yyy
#SBATCH -J jobname
#SBATCH -n 32
#SBATCH -t 12:00:00

module add WIEN2k/19.1-nsc1-intel-2018b-eb
# export SCRATCH=$SNIC_TMP  

# set .machines for parallel job
# lapw0 running on one node
echo -n "lapw0: " > .machines
echo -n $(hostlist -e $SLURM_JOB_NODELIST | tail -1) >> .machines
echo "$i:8" >> .machines

# run one mpi job on each node (splitting k-mesh over nodes)
for i in $(hostlist -e $SLURM_JOB_NODELIST)
do
  echo "1:$i:32 " >> .machines
done
echo granularity:1 >> .machines
echo extrafine:1   >> .machines

run_lapw -p

# if you need the vector files for continuing calculations, e.g.
# for DOS, spectrum etc. copy them to your current directory:
# cp $SNIC_TMP/* .

(Note that you should edit the jobname and the account number before submitting.)

Available modules

WIEN2k/19.1-nsc1-intel-2018b-eb

Installation of 19.1 release 25/6/2019, including some extra patches. This version includes OpenMP threading (instructions in preparation) and ELPA 2019.05.001.

WIEN2k/18.2-nsc2-intel-2018a-eb

A small update of nsc1 version with fix for running over several nodes.

WIEN2k/18.2-nsc1-intel-2018a-eb

Installation of 18.2 release 17/7/2018. This version includes ELPA 2015.11.001.

Further information

  • See the userguide for test examples and much information.

  • If setting export SCRATCH=$SNIC_TMP note that it might be problematic when using more than one node (if needed, contact NSC support).

  • If the vector files are needed for further processing, you need to retrieve them from the temporary directory as the job finishes.

  • For very heavy jobs with several 100s of inequivalent atoms, it might be necessary to specify in the job script:

     echo "lapw2_vector_split 4" >> .machines     #typical values 2, 4, 6, 8
  • In general, if you have several k-points it makes sense to use the k-point parallelization in WIEN2k. The most efficient way is to calculate 1 k-point per computer core (if memory use isn't too big).

  • For many jobs it can be most efficient to run on less than a full node, e.g. allocating 4, 8 or 16 cores on Tetralith/Sigma, see benchmarks below.

  • For tiny jobs (e.g. small unit cells) it is recommended to do a serial run:

    ...
    #SBATCH -n 1
    ...
    run_lapw
  • The use of the ELPA library can be switched to ScaLAPACK, by changing the 2nd line in the file .in1 or .in1c, from ELPAto SCALA:

    7.00     10   4   ELPA pxq BL 64 (R-MT*K-MAX,MAX L IN WF,V-NMT,LIB)
    -->
    7.00     10   4   SCALA pxq BL 64 (R-MT*K-MAX,MAX L IN WF,V-NMT,LIB)

k-point parallelization

k-point parallelization works over one or several nodes, e.g. using 32 cores/node on Tetralith. It is probably the most efficient way to run WIEN2k if you have several k-points. For example:

...
#SBATCH -n 32
...
for i in $(hostlist -e $SLURM_JOB_NODELIST)
do
  for j in {1..32}
  do
    echo "1:$i:1 " >> .machines
  done
done
echo granularity:1 >> .machines
echo extrafine:1   >> .machines
...

k-point parallelization and MPI

It is possible to combine k-point parallelization with mpi. For instance, if you have 2 k-points instead of 1 as in the example at the top, it might be more efficient to split the calculations into two:

...
#SBATCH -n 32
...
for i in $(hostlist -e $SLURM_JOB_NODELIST)
do
  echo "1:$i:16 " >> .machines
  echo "1:$i:16 " >> .machines
done
echo granularity:1 >> .machines
echo extrafine:1   >> .machines    

A different example, if there are more k-points. Here the list of k-points is split over two Tetralith nodes (32x2 cores) and then split in 8 parts on each node:

...
#SBATCH -n 64
...
for i in $(hostlist -e $SLURM_JOB_NODELIST)
do
  echo "1:$i:4 " >> .machines
  echo "1:$i:4 " >> .machines
  echo "1:$i:4 " >> .machines
  echo "1:$i:4 " >> .machines
  echo "1:$i:4 " >> .machines
  echo "1:$i:4 " >> .machines
  echo "1:$i:4 " >> .machines
  echo "1:$i:4 " >> .machines
done
echo granularity:1 >> .machines
echo extrafine:1   >> .machines
...

Benchmark and scaling test on Tetralith

Here, I will demonstrate some results for running WIEN2k on the Tetralith cluster. The calculations below are using WIEN2k 18.2 with ELPA corresponding to the WIEN2k/18.2-nsc1-intel-2018a-eb module (though use the latest recommended one). They are for regular self-consistent DFT PBE calculations, setting RKMAX=7.

Disordered Ga2O3 with 120 inequivalent atoms, 1 k-point (normalized 1 job on 4 cores, 50401 s).

grep :RKM *.scf
:RKM  : MATRIX SIZE 26231LOs:1032  RKM= 7.00  WEIGHT= 1.00  PGR:

Disordered NaCl with 64 inequivalent atoms, 1 k-point (normalized 1 job on 1 core, 2013 s).

grep :RKM *.scf
:RKM  : MATRIX SIZE  5048LOs: 416  RKM= 7.00  WEIGHT= 1.00  PGR:

FeOCl with 12 inequivalent atoms, 416 irreducible k-points (normalized 1 job k-point parallel on 8 cores, 12563 s).

grep :RKM *.scf
:RKM  : MATRIX SIZE  3049LOs: 184  RKM= 7.00  WEIGHT= 4.00  PGR:  

The first two cases are for larger systems where 1 k-point might be suitable for the self-consistent calculations, while the third case is for a smaller system with more k-points.

Ga2O3 scaling

Fig. 1: Scaling on Tetralith Ga2O3 supercell with 120 atoms for 1 k-point.

NaCl scaling

Fig. 2: Scaling on Tetralith for NaCl supercell with 64 atoms for 1 k-point.

FeOCl scaling

Fig. 3: Scaling on Tetralith for FeOCl supercell with 12 atoms for 416 k-points.

From the scaling in the first two cases, Fig. 1 and 2, it's clear that increasing the number of cores in use for the mpi parallelism doesn't increase the performance much. However, it might still be needed to allocate a full Tetralith node (32 cores) in order to have access to memory needed for the calculation. In such case, it's still more efficient to run on 32 cores since it gives best performance, even if the scaling isn't so good. In the last Fig. 3, the mpi parallelism is contrasted with the full k-point parallel scheme, effectively running several serial jobs. Here, it's clear that it's much more efficient to use k-point parallelism if possible.


User Area

User support

Guides, documentation and FAQ.

Getting access

Applying for projects and login accounts.

System status

Everything OK!

No reported problems

Self-service

SUPR
NSC Express