|8.1.0.024||Intel ITAC version 8.1.0.024|
|8.1.3.037||Intel ITAC version 8.1.3.037|
Official homepage: http://software.intel.com/en-us/intel-trace-analyzer
Intel Trace Analyzer and Collector (ITAC) is a powerful tool for understanding and visualizing MPI application behavior. It can be difficult to understand all the data that is being presented by ITAC, so we really recommendation checking out Intel’s documentation.
And please contact NSC Support if you have any questions or problems.
ITAC works by recording (“tracing”) subroutine calls while the program is running. It is recommended that you start by tracing MPI calls, and not the rest of the program. In that case, there is no need to recompile program. Just load the itac module, and launch your program with mpprun and the “-trace” flag.
module load itac/8.1.3.037 mpprun -trace ./<my_program>
After the program has exited, you will find lots of files called myprogram.stf.*. These are the trace files that you can analyze with the ITAC graphical user interface called “traceanalyzer”. We recommend that you login to triolith3 with the ThinLinc client instead of using normal X11 forwarding, as the performance is much better. More about ThinLinc here. Start traceanalyzer like this:
This will present the GUI.
ITAC can show from where, inside source code, a specific MPI call was made. To make this work, you need to have your program compiled with debugging support (typically the “-g” compiler flag), and turn on the ITAC’s stack sampling.
export VT_PCTRACE=5 mpprun -trace ...
The number you set VT_PCTRACE to controls have many levels of the stack that is recorded. Start with 5, it is often ok. The source code locations of each MPI call can be reached from the “Event View” graph: right-click on the red area corresponing to a call and choose “Details on function”. In the dialog window that appears, there is a button with a file icon in the column called “Show source”. Click on it.
To do full tracing of every subroutine calls, not just MPI, you need to compile your binary with additional flags for using ITAC. The compiler flag (only available with Intel’s compilers) is called “-tcollect”. So in a typical makefile you would have to insert:
CFLAGS += -tcollect -I$(VT_ROOT)/include FFLAGS += -tcollect -I$(VT_ROOT)/include LFLAGS += -L$(VT_LIB_DIR) -lVT $(VT_ADD_LIBS)
And recompile. Then run with mpprun “-trace”.
Please note that doing full tracing can be very slow and consume vast amounts of disk space. The amount of data increases by the number of MPI ranks, so try to work with the smallest job that can reproduce your problem.
For running a tracing task with stack sampling in the batch queue:
#!/bin/bash #SBATCH -N 4 #SBATCH -t 4:00:00 #SBATCH -J jobname #SBATCH --exclusive #SBATCH -A SNIC-xxx-yyy module load itac/8.1.3.037 export VT_PCTRACE=5 mpprun -trace ./your_exec