Quantum ESPRESSO Installations on Tetralith & Sigma

The Quantum ESPRESSO installations are generally maintained by Weine Olovsson (weiol@nsc.liu.se).

"Quantum ESPRESSO is an integrated suite of Open-Source computer codes for electronic-structure calculations and materials modeling at the nanoscale. It is based on density-functional theory, plane waves, and pseudopotentials."

Available modules

Most of the old installations from CentOS 7 are now provided as -compat-el7 after the upgrade to Rocky Linux 9. This means that the installations are automatically run within a CentOS 7 container, with the same (or very similar) performance.

QuantumESPRESSO/7.3-hpc1-intel-2023a

Most recent installation, done after update to Rocky Linux 9. 7.3 includes wannier90 3.1.0 and EPW 5.8. Want is installed. Most QE tests go through. Compiled with OpenMP.

QuantumESPRESSO/7.2-nsc1-intel-2018b-eb-compat-el7

Last standard installation on CentOS 7. 7.2 includes wannier90 3.1.0 and EPW 5.7. Want is installed. For EPW, 101 tests out of 102 goes through. Compiled with OpenMP. In this installation, Thermo_pw 1.8.1 is included.

QuantumESPRESSO/7.1-nsc1-intel-2018b-eb-compat-el7

Standard installation. 7.1 includes wannier90 3.1.0, gipaw 7.1, EPW 5.5 and Yambo 4.5.2. Standard tests go through. Want is installed. Compiled with OpenMP. In this installation, Thermo_pw 1.7.1 is included.

QuantumESPRESSO/7.0-nsc1-intel-2018b-eb-compat-el7

Standard installation. 7.0 includes wannier90 3.1.0, gipaw 7.0, EPW 5.4.1 and Yambo 4.5.2. Standard tests go through. Want is installed. Compiled with OpenMP. In this installation, Thermo_pw 1.6.1 is included.

QuantumESPRESSO/6.8-nsc1-intel-2018b-eb-compat-el7

Standard installation. 6.8 includes wannier90 3.1.0, gipaw 6.8, EPW 5.4 and Yambo 4.5.2. Standard tests go through. Want is installed. Compiled with OpenMP. In this installation, Thermo_pw 1.6.0 is also included.

QuantumESPRESSO/6.7-nsc1-intel-2018b-eb-compat-el7

Standard installation. 6.7.0 includes wannier90 3.1.0 and EPW 5.3.1. Standard tests go through. Want is installed, but not gipaw. Compiled with OpenMP.

QuantumESPRESSO/6.6-nsc1-intel-2018b-eb-compat-el7

Standard installation. 6.6 includes wannier90 3.1.0, gipaw 6.6, EPW 5.3.0 and Yambo 4.5.1. Standard tests go through. For EPW, there are some failed tests All done. ERROR: only 84 out of 91 tests passed. Yambo isn't tested. want not installed. Compiled with OpenMP.

QuantumESPRESSO/6.5-nsc1-intel-2018b-eb-compat-el7

Standard installation. 6.5 includes wannier90 3.0.0, gipaw 6.5 and want. Standard tests go through. Compiled with OpenMP.

QuantumESPRESSO/6.4.1-nsc1-intel-2018b-eb-compat-el7
QuantumESPRESSO/6.4-nsc1-intel-2018b-eb-compat-el7

Standard installations. 6.4.1 includes the patch backports-6.4.1.diff, wannier90 3.0.0, gipaw 6.4.1, want, EPW 5.1.0 and Yambo 4.2.1 (stable) and 4.3.2 (developer version). Standard tests goes through. For EPW, the tests epw_mob_ibte and epw_mob_ibte_sym fail. Yambo isn't tested. Compiled with OpenMP. Version 6.4 doesn't contain gipaw or Yambo.

QuantumESPRESSO/6.3-nsc2-intel-2018b-eb-compat-el7

Standard installation. Includes wannier90 2.1.0, gipaw 6.3 and want. Use this version rather than previous build nsc1. Compiled with OpenMP.

QuantumESPRESSO/6.3-nsc2-intel-2018b-eb-DEBUG

A debug version of the installation above. Use this if regular module crashes and you need more details for debugging. Compiled with OpenMP.

QuantumESPRESSO/6.3-nsc1-intel-2018a-eb

This installation includes wannier90 2.1.0, gipaw 6.2.1, EPW 5.0.0 and Yambo 4.2.1.

QuantumESPRESSO/6.1-nsc1-intel-2018a-eb

This installation includes wannier90 2.1.0, gipaw 6.1 and WEST 3.1.0.

How to run

Load the desired QE module, e.g.:

module add QuantumESPRESSO/6.8-nsc1-intel-2018b-eb-compat-el7

Launch the QE binary with "mpprun":

mpprun pw.x -input file.in

The path to the default pseudopotential directory included with Quantum ESPRESSO is stored in the $PSEUDO_DIR environment variable.

Example batch script

A minimum batch script for running QE looks like this:

#!/bin/bash
#SBATCH -n 32
#SBATCH -t 4:00:00
#SBATCH -J jobname
#SBATCH -A naiss-xxx-yyy

module add QuantumESPRESSO/6.8-nsc1-intel-2018b-eb-compat-el7
mpprun pw.x -input file.in

(Note that you should edit the jobname and the account number before submitting.)

Parallelization settings

Following the Quantum ESPRESSO manual: use k-point parallelization -npool=k-points whenever you can, and use -ntg=1/2/4 when you run on many processors (more cores than FFT gridpoints in the Z direction). Scalapack parallelization works, but set -ndiag to a value much less than the total number of cores.

OpenMP threading

Several Quantum ESPRESSO installations are compiled with support for optional OpenMP threading. It might e.g. be useful for increasing the efficiency when reducing the number of tasks (mpi ranks) per node in order to save memory.

To use the OpenMP threading, compare with the example job script from the top of the page, now setting e.g. 2 OpenMP threads. Note the number of tasks (mpi ranks), here 16 tasks x 2 OpenMP threads = 32 (cores/node). Also note the setting of OMP_NUM_THREADS.

#!/bin/bash
#SBATCH -n 16
#SBATCH -c 2
#SBATCH -t 4:00:00
#SBATCH -J jobname
#SBATCH -A SNIC-xxx-yyy

module add QuantumESPRESSO/6.8-nsc1-intel-2018b-eb-compat-el7
export OMP_NUM_THREADS=2
mpprun pw.x -input file.in

Note: for the special case of two OpenMP threads, OMP_NUM_THREADS=2, and running on more than one node, use the following command instead of mpprun:

srun --mpi=pmi2 -m block:block pw.x -input file.in

Possible problems

Some systems might have convergence problems, e.g. showing the message "problems computing cholesky". What one can try is to use alternative gcc builds (and an intel build) of Quantum ESPRESSO, which can be found by checking among the installations in:

/software/sse/manual/QuantumESPRESSO
/software/sse/manual/QuantumESPRESSO/6.7/elpa1

as well as available DEBUG builds (e.g. see above). Another possibility is to switch in &electrons from diagonalization='david' to 'cg', though note that this typically gives a much slower convergence.

Special instructions for phonon calculations

OBS: This section is under revision & might change! Due to heavy I/O in QE phonon calculations one needs to use the local scratch disk at each node. Below is an example of one way to do this, assuming that the scf and phonon calculations are done in the same run.

Set the output directory in the scf input file for pw.x and also for the phonon input file for ph.x (i.e. in the two input files):

outdir='TMP'

In case you run on a single node #SBATCH -n 32 (since Tetralith / Sigma has 32 cores per node), in the job script, before running pw.x create the temporary folder and insert the correct path for scratch disk:

mkdir $SNIC_TMP/tmp
# below assumes outdir='TMP' set in your input files, to get correct path!
sed -i.bak -e "s|TMP|${SNIC_TMP}/tmp|g" input.scf.in
sed -i.bak -e "s|TMP|${SNIC_TMP}/tmp|g" input.ph.in

Now everything is set up for running on a single node.

If you want to run a parallel job on several nodes, you need to adjust the job script before running ph.x. In this example it is assumed that prefix='Ti' and that the scf calculation is done in the same job, with its temporary folder in the job working directory:

export WORKDIR=$PWD
mkdir -p $WORKDIR/tmp
# below assumes outdir='TMP' set in your input files, to get correct path!
sed -i.bak -e "s|TMP|${WORKDIR}/tmp|g" input.scf.in
sed -i.bak -e "s|TMP|${SNIC_TMP}/tmp|g" input.ph.in 

mpprun pw.x -input input.scf.in

The phonon calculations are set up right after the scf run in the job script:

mkdir $WORKDIR/tmp/_ph0
mkdir $WORKDIR/tmp/_ph0/Ti.save
cp $WORKDIR/tmp/Ti.save/charge-density.dat $WORKDIR/tmp/_ph0/Ti.save/
cp -r $WORKDIR/tmp $SNIC_TMP/

for i in $(hostlist -e $SLURM_JOB_NODELIST | tail -n +2)
do
jobsh $i <<HERE
rsync -a $WORKDIR/tmp $SNIC_TMP
HERE
done

mpprun ph.x -input input.ph.in

This would also work with scf and phonon runs done as separate jobs, since the 'TMP' folder from scf is in the local work directory.