ABAQUS Installations on Tetralith and Sigma

Official homepage: https://www.3ds.com/products-services/simulia/products/abaqus

The Abaqus FEA software suite offers various tools for stress analysis, heat transfer, fluid mechanics etc.

Installed ABAQUS Versions

ABAQUS License

ABAQUS is a commercial program, that requires a license. NSC does not provide any ABAQUS licenses. That means, you have to connect to a license server that runs at your department or university. Typically, the license server is run by the IT department, e.g. at Chalmers.

To connect to the license server, you require the name of the license server and a port number. Both are provided by the IT department that runs the license server.

How to specify the ABAQUS License Information

To specify the ABAQUS license server information, you have to create a file abaqus_v6.env in your home directory ($HOME). The file abaqus_v6.env has to contain the license information as following:

abaquslm_license_file="port@server-name"

port is a five digit number, and server-name is the name of the license server. This format relates to a FLEXlm license file. ABAQUS also supports the DSLS license system, which requires different settings.

How to run

Load the ABAQUS module corresponding to the version you want to use, for example

module load abaqus/2020-nsc1

This gives you access to the ABAQUS software.

To start the graphical interface abaqus cae, you have the following two options:

1. On a login node, or a compute node with a GPU, start abaqus cae as following:

   vglrun abaqus cae

2. On a standard compute node, that does not have a GPU, start abaqus cae as following:

   abaqus cae -mesa

vglrun provides GPU support for the graphical interface. The option mesa indicates, that the graphical interface has to be rendered using software rendering, without hardware accelerated GPU support.

To run a simulation without the grahical interface, ABAQUS is started in the following way:

abaqus job=job-name cpus=number-of-cores interactive

Common ABAQUS Options

The following options are commonly used as arguments, when starting an ABAQUS simulation

ABAQUS Environmental Files

The ABAQUS environmental settings allow you to control ABAQUS job execution. Many advanced options can be specified, that would otherwise have to be given on the command line. The ABAQUS environmental settings are saved in the configuration file abaqus_v6.env. The environmental settings are processed in a hierarchical order:

1. The user-level environment settings: Create the file abaqus_v6.env in your home directory ($HOME) to apply settings to all ABAQUS jobs that run in your account. This file typically contains the license server information, namely the variable abaquslm_license_file.

2. The job-level environment settings: These settings are applied to only the designated ABAQUS job. Create the file abaqus_v6.env in the current directory, where the ABAQUS input file resides and from where the job is started.

Environment settings can be specified more than once. The last value processed will be the one used for the setting if it is defined at more than one level.

Environmental File Syntax

The following is a brief overview of the syntax rules of abaqus_v6.env, see e.g. Reference Documentation for more details:

• A string value must be enclosed in a pair of double or single quotes
• Comments are preceded by a number sign (#)
• Blank lines are ignored
• Triple quoted (""") strings can span more than one line
• Lists must be enclosed in parentheses (( )) or square brackets ([ ]). Individual items in the list are separated by commas

Hostlist for Multiple Nodes

To run ABAQUS on multiple compute nodes, ABAQUS requires a list of host machine names including the number of cores to be used on each machine/compute node. The hostlist has to be specified in the ABAQUS variable mp_host_list. Since the hostlist is a job-level environmental setting, it should be defined in abaqus_v6.env within the current job directory.

The hostlist has the following format:

mp_host_list=[['node1',number-of-cores],['node2',number-of-cores]]

In this example, we consider two compute nodes. node* specifies the name of the compute node on Tetralith/Sigma, for example n143. number-of-cores specifies the number of cores, that is used on that node.

Hostlist for Interactive Jobs

For interactive jobs, that typically use a small number of nodes, we can easily edit the hostlist by hand. For example, for an allocation of two compute nodes with 32 cores each (interactive -N2, or interactive -n64):

mp_host_list=[['n98',32],['n143',32]]

In an interactive session, you can determine the names of the compute nodes via the command "squeue -u your-user-name". The node names are listed under "NODELIST(REASON)". Alternatively, you can display the variable $SLURM_NODELIST

echo $SLURM_NODELIST

Hostlist for Batch Jobs

When submitting a job within a SLURM batch script, we need a more automated way to create the hostlist. The names of the compute nodes, as well as the number of cores can be retrieved from the SLURM environment. The following command automatically generates the hostlist for you, and saves it in the intermediate variable mp_host_list:

mp_host_list="["$(hostlist --repeat-slurm-tasks="$SLURM_TASKS_PER_NODE" "$SLURM_NODELIST" -e | \
                sort | uniq -c | awk '{print "[\047"$2"\047," $1"]"}' | paste -sd, -)"]"

To save the hostlist in the file abaqus_v6.env, you can use

echo "mp_host_list=${mp_host_list}" > abaqus_v6.env

A complete example of a SLURM batch script is given below.

MPI Environmental Variables

MPI environmental variables can be set using the ABAQUS variable mp_mpirun_options:

mp_mpirun_options="-v -genv VAR_NAME VAR_VALUE"

Multiple variables can be set using several -genv options:

mp_mpirun_options="-v -genv VAR_NAME_1 VAR_VALUE_1 -genv VAR_NAME_2 VAR_VALUE_2"

For example, to set the MPI environment variable I_MPI_DEBUG=5, we use

mp_mpirun_options="-v -genv I_MPI_DEBUG 5"

General Environmental Variables

To modify more general environmental variables, you have to use

import os

Typically, import os is placed in the first line of abaqus_v6.env. Then you can modify any environmental variable as following:

os.environ['VARIABLE'] = 'VALUE'

How to Get Basic Testcases

ABAQUS includes several basic testcases. These can be retrieved via the command abaqus fetch, for example

abaqus fetch job=c1

Example Batch Script

To run ABAQUS in batch mode, it has to be submitted via a SLURM batch script, as the following example illustrates:

#!/bin/bash
#SBATCH -n 64
#SBATCH -t 00:20:00
#SBATCH -J slurm_jobname
#SBATCH -A SNIC-xxx-yyy

module load abaqus/2020-nsc1

ABAQUS_JOBNAME=c1

# -------------------------------------------------------------------------

envFile=abaqus_v6.env

# Set the hostlist
mp_host_list="["$(hostlist --repeat-slurm-tasks="$SLURM_TASKS_PER_NODE" "$SLURM_NODELIST" -e | \
          sort | uniq -c | awk '{print "[\047"$2"\047," $1"]"}' | paste -sd, -)"]"

# Set number of parallel tasks
tasks=$(hostlist --repeat-slurm-tasks="$SLURM_TASKS_PER_NODE" "$SLURM_NODELIST" -e | wc -l)

# Write the hostlist to the environment file
echo "mp_host_list=${mp_host_list}" > $envFile

echo "ask_delete=OFF" >> $envFile

# Run abaqus
abaqus job=$ABAQUS_JOBNAME cpus=$tasks scratch=. interactive

Submit the script via the command: sbatch <script file>. Here, we assume that the license information is provided in abaqus_v6.env in the home directory.

You have to adjust the batch script as following:

1. Specify the number of cores via the option #SBATCH -n<cores>
2. Specify the required time #SBATCH -t <time>
3. Change the SLURM jobname #SBATCH -J <job name>
4. Specify your account name #SBATCH -A <account name>
5. Adjust the name of your ABAQUS_JOBNAME (testcase)