|
DisCostiC v2.0.0
Distributed Cost in Cluster: a cross-architecture simulator for massively parallel MPI programs
|
Part of the main README.
The system parameters, either hypothetical or actual, can be tuned by editing the config.cfg file available in the current directory. Machine descriptions are YAML files in machine files. Network descriptions are numeric .cfg tables in network files.
The resource allocation, intercluster characteristics, and runtime modalities are all included in the cluster model.
| Metadata information | Description |
|---|---|
task_per_node | Number of processes placed on one node |
number_of_processes | Number of simulated processes. Launch the simulator with this count plus one. |
number_of_timesteps | Number of iterations. collective_mode = 1 forces this to one. |
heteregeneous | Second system (0: off, 1: primary plus secondary processes). Documented in Heterogeneous mode. |
dim_x, dim_y, dim_z | Problem size. High-dimensional parameters are ignored for low-dimensional problems. |
Each network file is one scope (intra-chip, inter-chip, inter-node, or inter-cluster). A row is number_of_simultaneous_process, msg_size_in_bytes, latency_in_ns, inverse_bandwidth, comm_eagerlimit_in_bytes, G_in_ns, comm_o_in_ns, comm_g_in_ns, comm_O_in_ns. config.cfg selects the model with comm_model (0: LogGP, 1: Hockney) and the WaitIO mode with waitio_mode (socket, file, or hybrid).
| Column | Meaning |
|---|---|
latency_in_ns | Latency parameter L, in nanoseconds |
inverse_bandwidth | Inverse bandwidth, GB/s interpreted as bytes per nanosecond |
comm_eagerlimit_in_bytes | Message size at which the model switches from eager to rendezvous |
comm_o_in_ns, comm_g_in_ns, comm_O_in_ns | LogGP overhead, gap, and per-byte overhead |
The network files shipped in this tree are:
| Network file | Scope |
|---|---|
| A64FX_interchip.cfg | A64FX inter-chip |
| A64FX_internode.cfg | A64FX inter-node |
| A64FX_intrachip.cfg | A64FX intra-chip |
| A64FX_waitio_socket_interchip.cfg | A64FX WaitIO socket inter-chip |
| A64FX_waitio_socket_internode.cfg | A64FX WaitIO socket inter-node |
| A64FX_waitio_socket_intrachip.cfg | A64FX WaitIO socket intra-chip |
| BroadwellEP_E5-2630v4_interchip.cfg | Broadwell EP E5-2630 v4 inter-chip |
| BroadwellEP_E5-2630v4_internode.cfg | Broadwell EP E5-2630 v4 inter-node |
| BroadwellEP_E5-2630v4_intrachip.cfg | Broadwell EP E5-2630 v4 intra-chip |
| IcelakeSP_Platinum-8360Y_interchip.cfg | Ice Lake SP Platinum 8360Y inter-chip |
| IcelakeSP_Platinum-8360Y_internode.cfg | Ice Lake SP Platinum 8360Y inter-node |
| IcelakeSP_Platinum-8360Y_intrachip.cfg | Ice Lake SP Platinum 8360Y intra-chip |
| IcelakeSP_Platinum-8360Y_waitio_socket_interchip.cfg | Ice Lake SP Platinum 8360Y WaitIO socket inter-chip |
| IcelakeSP_Platinum-8360Y_waitio_socket_internode.cfg | Ice Lake SP Platinum 8360Y WaitIO socket inter-node |
| IcelakeSP_Platinum-8360Y_waitio_socket_intrachip.cfg | Ice Lake SP Platinum 8360Y WaitIO socket intra-chip |
| SapphireRapids_Platinum-8470_interchip.cfg | Sapphire Rapids Platinum 8470 inter-chip |
| SapphireRapids_Platinum-8470_internode.cfg | Sapphire Rapids Platinum 8470 inter-node |
| SapphireRapids_Platinum-8470_intrachip.cfg | Sapphire Rapids Platinum 8470 intra-chip |
| waitio_socket_intercluster.cfg | WaitIO socket inter-cluster |
The machine files in YAML format, the selection of compiler settings and the performance model of computation are included in the node model.
| Metadata information | Description |
|---|---|
name | Name of the YAML formatted processor file |
sockets_per_node | Number of sockets in one node |
ccNUMA_domains_per_socket | Number of ccNUMA domains in one socket |
cores_per_ccNUMA_domain | Number of cores in one ccNUMA domain |
FP_instructions_per_cycle | Floating point instructions (ADD, MUL) per cycle |
FP_operations_per_instruction_(SP/DP) | Single or double precision floating point operations per instruction |
clock_frequency | Clock frequency in GHz |
memory_bandwidth | Memory bandwidth in GB/s |
compiler-flags | STD and SIMD optimizations of the compiler |
pmodel | Performance model for computation (Roofline, ECM) |
For illustration, a few examples of available machine files are given below:
| Machine files (YAML format) | Description |
|---|---|
| A64FX | 48 core Fujitsu A64FX FX1000 CPU @ 1.8 GHz |
| IcelakeSP_Platinum-8360Y | 36 core Intel(R) Xeon(R) Icelake SP Platinum 8360Y CPU @ 2.4 GHz |
| CascadelakeSP_Gold-6248_SNC | 20 core Intel(R) Xeon(R) Cascadelake SP Gold 6248 CPU @ 2.5 GHz |
| SkylakeSP_Gold-5122 | 4 core Intel(R) Xeon(R) Skylake SP Gold 5122 CPU @ 3.6 GHz |
| SkylakeSP_Gold-6148 | 20 core Intel(R) Xeon(R) Skylake SP Gold 6148 CPU @ 2.4 GHz |
| SkylakeSP_Gold-6148_SNC | 20 core Intel(R) Xeon(R) Skylake SP Gold 6148 CPU with SNC enabled @ 2.4 GHz |
| SkylakeSP_Platinum-8147 | 24 core Intel(R) Xeon(R) Skylake SP Platinum 8174 CPU @ 3.1 GHz (2.7 GHz used) |
| BroadwellEP_E5-2630v4 | 10 cores/socket Intel Xeon E5-2630 v4 @ 2.20 GHz |
| SapphireRapids_Platinum-8470 | 52 cores/socket Intel Xeon Platinum 8470 @ 2.00 GHz |
| ARMv8_ThunderX2-CN9980 | 32 cores/socket ThunderX2 CN9980; clock in the file depends on SKU (2.0-2.5 GHz) |
| Zen5_EPYC-9965 | 192 cores/socket AMD EPYC 9965 @ 2.03 GHz |
| Zen2_EPYC-7452 | 32 cores/socket AMD EPYC 7452 @ 2.35 GHz |
| Zen_EPYC-7451 | 24 cores/socket AMD EPYC 7451 @ 2.3 GHz |
| Zen_Ryzen7-1700X | 8 cores/socket AMD Ryzen 7 1700X @ 2.90 GHz |
| BroadwellEP_E5-2697v4_CoD | 18 core Intel(R) Xeon(R) Broadwell EN/EP/EX E5-2697 v4 CPU with CoD enabled @ 2.3 GHz |
| HaswellEP_E5-2695v3 | 14 core Intel(R) Xeon(R) Haswell EN/EP/EX E5-2695 v3 CPU @ 2.3 GHz |
| HaswellEP_E5-2695v3_CoD | 14 core Intel(R) Xeon(R) Haswell EN/EP/EX E5-2695 v3 CPU with CoD enabled @ 2.3 GHz |
| IvyBridgeEP_E5-2660v2 | 10 core Intel(R) Xeon(R) IvyBridge EN/EP/EX E5-2660 v2 CPU @ 2.2 GHz |
| IvyBridgeEP_E5-2690v2 | 10 core Intel(R) Xeon(R) IvyBridge EN/EP/EX E5-2690 v2 CPU @ 3.0 GHz |
| SandyBridgeEP_E5-2680 | 8 core Intel(R) Xeon(R) SandyBridge EN/EP E5-2680 CPU @ 2.7 GHz |
./discostic --help and ./discostic --version print the text from help() and version() in include/macro.hpp. The version line is v1.0 (initial release). The parameter list is:
P2P_mode is what --help prints. The shipped config.cfg leaves P2P_mode commented out and sets benchmark_kernel to the full header name. heteregeneous, collective_mode, and number_of_timesteps are read by the simulator and are not listed by --help. The spelling compiler-flags in the help text does not match compiler_flags in config.cfg.
Static analysis is the most reliable for regular communication and known problem sizes. Unstructured or rank-unique patterns need a hand-written DSEL, and some kernels (for example HPCG SpMV) need manual extensions.
To perform the static analysis, the following procedure must be followed before running the DisCostiC batch script:
Converters in this tree include Convert-HEAT.py, Convert-HPCG.py, Convert-STREAM.py, and Convert-POISSONNS.py.
| Files | Description |
|---|---|
Convert-<benchmark_kernel>.py | A helper script that takes the original code and, through annotation, locates code loops and communication and identifies user-defined variables, such as dim_x and dim_y in the Cartesian stencil heat.c code, and ultimately generates DSEL code. |
requirements.txt | Each (sub)dependency is listed and pinned using "==" to specify a particular package version; see requirements.txt. This project makes use of the lightweight Python tree data structure anytree==2.8.0 and type hints for Python 3.7+ typing_extensions==4.4.0. These dependencies are later installed (normally in a virtual environment) through pip using the pip install -r staticanalysis/requirements.txt command. The generated tree's syntax is the same as the original C/C++ code. |
For the specified program, this will produce the following files:
| Files | Location | Description |
|---|---|---|
<benchmark_kernel>.c | nodelevel/kernels | It comprises only the generated computational loop kernel of the MPI parallel program. |
<benchmark_kernel>_<kernel_mode>.hpp | test | It includes the entire generated code expressed in DisCostiC DSEL language. |
📝 Essential DisCostiC routines 💭
The goal is to offer convenient, compact and practically usable application programming interfaces (APIs) with appropriate abstractions. It provides information about the call tree and attributes for communication (volume, mode, and protocol) and computation (data access pattern, flop count).
📝 Code blueprint as DisCostiC input 💭
The following DSEL code example shows the simplest illustration of how the domain knowledge of the applications is expressed in the DisCostiC language:
Collective calls use the signature in routine 8 above. Mode flags, algorithms, and the multi-kernel and heterogeneous placements are in README_COLLECTIVE.md.
📝 Code documentation: suite of C++ data structures and enumerated types 💭
The HTML and LaTeX documentation can be generated by doxygen. To build the documentation, navigate to the Doxyfile file available in the current directory and run the command:
After doxygen Doxyfile, the HTML tree is doc/html/index.html. The generated HTML is committed under doc/html/ and published at https://rrze-hpc.github.io/DisCostiC-Sim/doc/html/index.html. The documentation shipped with this tree is the Markdown guides in doc/.
| Metadata information | Description |
|---|---|
bufSize | Number of bytes (data size) transmitted in the communication operation (no real buffer size for comp, just added for completeness) |
DepOperations | Dependencies for blocking routines, i.e., other operations that depend on this current operation |
IdepOperations | Dependencies for non-blocking routines, i.e., other operations that depend on the current operation |
depCount | Number of dependencies for this current operation |
label | Index/identifier of this current operation for each rack |
target | Rank of target/partner (source for recv or destination for send or no real target for comp but added for completeness) |
rank | Owning rank of this current operation |
tag | Tag of this current operation (no real tag for comp, just added for completeness) |
node | Node or processing element for this current operation |
network | Type of network for this current operation |
time | Ending time of this current operation |
starttime | Starting time of this current operation |
type | Type of this current operation |
| Metadata information | Description |
|---|---|
mode | mode of this current calling operation |
| Metadata information | Description |
|---|---|
model | Analytic first-principle performance model for computation and communication |