DisCostiC v2.0.0
Distributed Cost in Cluster: a cross-architecture simulator for massively parallel MPI programs
Loading...
Searching...
No Matches
🧐 MPI-parallelized DisCostiC implementation

Part of the main README.

DisCostiC is itself an MPI program. Launch it with number_of_processes + 1 ranks: one controller rank (Q_0) and one simulator rank (Q_i) for each simulated application rank (P_i). MPI_Comm_split puts the controller on its own side of the communicator. The remaining ranks advance the discrete-event schedule and exchange serialized operation metadata with blocking or non-blocking MPI. The controller merges trace output and prints verbose logs.

Simulator process count follows the simulated rank count. Very large runs may need distributed event scheduling; that limit is listed under Planned features.

The underlying principle of parallel simulation is that each operation's entire data is transmitted via blocking or non-blocking MPI routines to the processes it communicates with.

Terms Description
simulated processes P_i processes from the application point of view
simulator processes Q_i processes from the parallel simulation framework point of view
master simulator process Q_0 master process from the parallel simulation framework point of view

Initialization 💭 In the MPI implementation, only the master process of the default communicator makes any print calls for debug and verbose output reporting, and all other processes of the new communicator initialize root operations only once in each run.

MPI-parallelized

int ierr, process_Rank, size_Of_Cluster, sucesss, N_process_Rank, N_size_Of_Cluster;
MPI_Comm newcomm;
ierr = MPI_Init(NULL, NULL);
ierr = MPI_Comm_rank(MPI_COMM_WORLD, &process_Rank);
MPI_Comm_split(MPI_COMM_WORLD, (process_Rank) < 1 ? 1 : 2, 1, &newcomm);
ierr = MPI_Comm_rank(newcomm, &N_process_Rank);
ierr = MPI_Comm_size(newcomm, &N_size_Of_Cluster);
ierr = MPI_Comm_rank(MPI_COMM_WORLD, &process_Rank);
ierr = MPI_Comm_size(MPI_COMM_WORLD, &size_Of_Cluster);
std::queue<int> test;
MPI_Status status;
MPI_Request request;
DisCostiC_Timetype starttimeProcess;
DisCostiC::DisCostiC_OP operation, operation_og;
int iter = 0;
iter = ((rankLocalIt->getNumOps()) / 3);
if (rank == N_process_Rank)
{
operation_og = copy(op, operation_og);
operation = op;
starttimeProcess = operation.starttime;
}
DisCosTiC::DisCosTiC_OP copy(DisCosTiC::DisCosTiC_OP op_og, DisCosTiC::DisCosTiC_OP op_copy)
Definition DisCosTiC.cpp:79

Serialized

op.numOpsInQueue = numOpsInQueue++;
queue.emplace(op);

Start of simulation 💭 Do loop over the number of iterations and the number of steps per iteration. Processes connected to the new communicator, or "newcomm," are only used in the main simulation.

MPI-parallelized

rank = N_process_Rank;
bool cycle = true;
int iter_num = 1;
do{
do{

Serialized

do{
queue.pop();

Within each ccNUMA domain, runtime corrections are made by monitoring the number of processes that are concurrently engaged in each computation and communication step.

SEND operation 💭 The currently active process sends the operation object or array to the process listed as "operation.target," which is specific to the communication pattern simulation program.

MPI-parallelized

operation.type = DisCostiC::Operation_t::RECV;
operation.starttime = operation.time;
int temp = oSuccessor[operation.rank][operation.node];
double arr[13];
*serialize(operation, temp, arr);
if (operation.mode == DisCostiC::Mode_t::NONBLOCKING){
MPI_Isend(&arr, 13, MPI_DOUBLE, operation.target, operation.tag, newcomm, &request);
}else{
MPI_Send(&arr, 13, MPI_DOUBLE, operation.target, operation.tag, newcomm);
}
finishedRankList.push_back(operation.rank);
operation.type = DisCostiC::Operation_t::RECV;
cycle = true;

Serialized

std::swap(operation.rank,operation.target);
queue.emplace(operation);

RECV operation 💭 The currently running process receives the operation object or array from the process listed as "operation.target".

MPI-parallelized

double brr[13];
if (operation.mode == DisCostiC::Mode_t::NONBLOCKING){
MPI_Irecv(&brr, 13, MPI_DOUBLE, MPI_ANY_SOURCE, operation.tag, newcomm, &request);
}else{
MPI_Irecv(&brr, 13, MPI_DOUBLE, MPI_ANY_SOURCE, operation.tag, newcomm, &request);
}
*deserialize(operation, temp, brr);
oSuccessor[operation.rank][operation.node] = brr[12];
cycle=true;
operation.numOpsInQueue = numOpsInQueue++;
finishedRankList.push_back(operation.rank);
operation.type = DisCostiC::Operation_t::MSG;

Serialized

finishedRankList.push_back(operation.rank);
DisCostiC::DisCostiC_queueOP matchedOP;
if(M.listmatch(operation, &UMQ[operation.rank], &matchedOP))
else {
DisCostiC::DisCostiC_queueOP nOp;
nOp.bufSize = operation.bufSize;
nOp.src = operation.target;
nOp.tag = operation.tag;
nOp.label = operation.label;
recvQueue[operation.rank].push_back(nOp);
}

MSG operation 💭 If MSG is not found, the cycle will continue (cycle = true) until it is found.

MPI-parallelized

if ((earliestFinish = operation.time) <= ...){
...
operation.numOpsInQueue = numOpsInQueue++;
...
iter_num++;
cycle = false;
}else{
...
cycle = true;
}

Serialized

if ((earliestFinish = operation.time) <= ...){
...
DisCostiC::DisCostiC_queueOP matchedOP;
if(M.listmatch(operation, &recvQueue[operation.rank], &matchedOP)) {
...
}else {
#ifdef USE_VERBOSE
std::cout << "[MSG ==> NOT FOUND] in receive queue - add to unexpected queue"<< std::endl;
#endif
DisCostiC::DisCostiC_queueOP nOp;
nOp.bufSize = operation.bufSize;
nOp.src = operation.target;
nOp.tag = operation.tag;
nOp.label = operation.label;
nOp.starttime = operation.time; // when it was started
UMQ[operation.rank].push_back(nOp);
}
}else{
...
queue.emplace(operation);
}

Adding new sorted operations in queue in order 💭

The parallel path does not add a separate snippet in this step.

Serialized

newOps=false;
newOps = true;
op.numOpsInQueue=numOpsInQueue++;
queue.emplace(op);

End of while statement 💭 The end of the do loops over the number of iterations and the number of steps per iteration. The necessary information gathered from all processes of the new communicator is communicated to the default communicator's master process.

MPI-parallelized

while (cycle);
cycle = true;
operation_og.time = oSuccessor[operation.rank][operation.node];
operation = copy(operation_og, operation);
} while (iter_num <= iter);
double arr[13];
double brr[13];
*serialize(operation, oSuccessor[rank][operation.node], arr);
*finalize(oSuccessor[rank][operation.node], numRanks, numOpsInQueue, rank, operation.node, brr);
MPI_Send(&brr, 13, MPI_DOUBLE, 0, 1, MPI_COMM_WORLD);
}
else if (process_Rank == 0)
{
bool cycle = true;
double brr[13];
while (test.size() < numRanks)
{
MPI_Recv(&brr, 13, MPI_DOUBLE, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
if (status.MPI_TAG == 1)
{
test.emplace(status.MPI_SOURCE);
oSuccessor[brr[3]][brr[4]] = brr[0];
numOpsInQueue = brr[2];
}
}
}
ierr = MPI_Comm_rank(MPI_COMM_WORLD, &process_Rank);
MPI_Barrier(MPI_COMM_WORLD);
double * finalize(double a, double b, double c, double d, double e, double arr[5])
Definition DisCosTiC.cpp:98

Serialized

while(!queue.empty() || newOps)