likwid-mpirun

Information

likwid-mpirun A tool to start and monitor MPI applications with LIKWID. It can be used as supplement of the MPI implementations' startup programm like mpirun or mpiexec with some enhancements for pinning of OpenMP thread in hybrid jobs. Moreover, likwid-mpirun can insert calls to likwid-perfctr to measure hardware performance counters for each MPI process and its threads, including Marker API. Since the modules system is widely used on clustered systems, likwid-mpirun checks the modules system to get MPI and OpenMP types. The hostfile must be in the format of the loaded MPI implementation.

Options

Option	Description
-h, –help	Print help message
-v, –version	Print version information
-d, –debug	Print debug information
-n, -np, –n, –np <arg>	Specify the number of processes for MPI
–nperdomain <domain>:<arg>	Schedule <arg> MPI processes for each affinity domain starting with <domain>, e.g S:2 translates in two MPI processes per socket. likwid-mpirun assumes that all participating hosts have the same topology.
–hostfile <file>	Specify the file that should be used as hostfile. If not set, likwid-mpirun checks the PBS_NODEFILE, LOADL_HOSTFILE and SLURM_HOSTFILE environment variable
–pin <expr>	For hybrid pinning specify the thread pinning expression for each MPI process. The format is similar to CPU_expressions separated by '_' for multiple processes. If -np is not set, the number of MPI processes is calculated using the pinning expressions.
-s, –skip <arg>	'arg' must be a bitmask in hex. Threads with the ID equal to a set bit in bitmask will be skipped during pinning Example: 0x1 = Thread 0 is skipped.
–mpi <mpitype>	Specify the type of the MPI implementation. likwid-mpirun tries to read the MPI implementation from the modules system. If not recognized automatically, possible values are intelmpi, openmpi and mvapich2.
–omp <omptype>	Specify the type of OpenMP implementation. likwid-mpirun tries to read the OpenMP implementation using ldd and the modules system. If not recognized automatically, possible values are intel and gnu
-g, –group <eventset>	Use likwid-perfctr to measure performance data for the MPI processes and OpenMP threads. <eventset> can be either a performance group or a custom event string. For details see performance_groups.
-m, –marker	Activate the Marker_API for the measurements with likwid-perfctr.
-O	Print results in CSV format (conform to RFC 4180)
-f	Configure events even if the counter is already in use.

Examples

• likwid-mpirun -np 32 ./a.out
  Runs ./a.out with 32 MPI processes distributed over the hosts in PBS_NODEFILE
• likwid-mpirun -nperdomain S:1 ./a.out
  Runs ./a.out using one MPI process per socket over the hosts in PBS_NODEFILE, LOADL_HOSTFILE or SLURM_HOSTFILE.
  The total amount of processes is calculated by <numberOfSocketDomains> * <processCountPerDomain> * <hostsInHostfile>
• likwid-mpirun –hostfile host.list -pin S0:2_S1:2 ./a.out
  Runs ./a.out using two MPI processes per host in host.list.
  The first MPI process on each host and its 2 threads are pinned to the first two CPUs on socket S0,
  the second MPI process on each host and its 2 threads are pinned to the first two CPUs on socket S1
• likwid-mpirun -nperdomain S:2 -g MEM ./a.out
  Runs ./a.out with 2 MPI processes per socket on each host in PBS_NODEFILE, LOADL_HOSTFILE or SLURM_HOSTFILE and measure the MEM performance group
  Only one process per socket measures the Uncore/RAPL counters, the other one(s) only core-local counters.

*/