Intel ITAC installations at NSC

Directory list:

VersionDescription Intel ITAC version Intel ITAC version

Level of support

Tier 2 NSC has only limited experience with this software, but we will try to help as much as possible. We have ran some tests, for example, if the program comes with a test suite, but they may be far from exhaustive. We will try to install and test new versions, as soon as we can.

Please see the page describing our software support categories for more information.
You can also contact for further information.

Official homepage:

Intel Trace Analyzer and Collector (ITAC) is a powerful tool for understanding and visualizing MPI application behavior. It can be difficult to understand all the data that is being presented by ITAC, so we really recommendation checking out Intel’s documentation.

And please contact NSC Support if you have any questions or problems.

How to run (MPI tracing only)

ITAC works by recording (“tracing”) subroutine calls while the program is running. It is recommended that you start by tracing MPI calls, and not the rest of the program. In that case, there is no need to recompile program. Just load the itac module, and launch your program with mpprun and the “-trace” flag.

module load itac/
mpprun -trace ./<my_program>

After the program has exited, you will find lots of files called myprogram.stf.*. These are the trace files that you can analyze with the ITAC graphical user interface called “traceanalyzer”. We recommend that you login to triolith3 with the ThinLinc client instead of using normal X11 forwarding, as the performance is much better. More about ThinLinc here. Start traceanalyzer like this:

traceanalyzer <my_program>.stf

This will present the GUI.

Getting links to source code locations

ITAC can show from where, inside source code, a specific MPI call was made. To make this work, you need to have your program compiled with debugging support (typically the “-g” compiler flag), and turn on the ITAC’s stack sampling.

export VT_PCTRACE=5
mpprun -trace ...

The number you set VT_PCTRACE to controls have many levels of the stack that is recorded. Start with 5, it is often ok. The source code locations of each MPI call can be reached from the “Event View” graph: right-click on the red area corresponing to a call and choose “Details on function”. In the dialog window that appears, there is a button with a file icon in the column called “Show source”. Click on it.

How to run (full tracing)

To do full tracing of every subroutine calls, not just MPI, you need to compile your binary with additional flags for using ITAC. The compiler flag (only available with Intel’s compilers) is called “-tcollect”. So in a typical makefile you would have to insert:

CFLAGS  += -tcollect -I$(VT_ROOT)/include
FFLAGS  += -tcollect -I$(VT_ROOT)/include

And recompile. Then run with mpprun “-trace”.

Please note that doing full tracing can be very slow and consume vast amounts of disk space. The amount of data increases by the number of MPI ranks, so try to work with the smallest job that can reproduce your problem.

Example batch script

For running a tracing task with stack sampling in the batch queue:

#SBATCH -t 4:00:00
#SBATCH -J jobname
#SBATCH --exclusive
#SBATCH -A SNIC-xxx-yyy

module load itac/
export VT_PCTRACE=5
mpprun -trace ./your_exec