MPI Reduction of Results

[JDTruj2018]

I would like to add cleaner output for MPI runs.

The current workflow outputs data to stdout from each rank, which may be interleaved. This is not human readable when using > ~10 ranks, and requires some parsing (see lines 207-218 here: https://github.com/lanl/spatter/blob/main/scripts/scaling.sh).

                mv ${JSON}.orig ${JSON}

		mkdir -p ${ranklist[i]}r

		num_patterns=`cat ${JSON} | grep -o -P 'kernel.{0,20}' | wc -l`
		num_patterns="$((num_patterns-1))"

		for pattern in $(seq 0 ${num_patterns}); do
			cat mpi_${ranklist[i]}r${CFILE}.txt | grep "^${pattern} " | awk '{print $3}' > ${ranklist[i]}r/${ranklist[i]}r${CFILE}_${pattern}p.txt.tmp
			cat ${ranklist[i]}r/${ranklist[i]}r${CFILE}_${pattern}p.txt.tmp | awk '{$1=$1};1' > ${ranklist[i]}r/${ranklist[i]}r${CFILE}_${pattern}p.txt
			rm ${ranklist[i]}r/${ranklist[i]}r${CFILE}_${pattern}p.txt.tmp
		done

Here is an example of the current output with 4 ranks of skylake:

Pattern Length: 16 Max Pattern Value: 15                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                
Running Spatter version 0.4                                                                                                                                                                                                                                                                     
Compiler: GNU ver. 9.4.0
Compiler Location: /projects/opt/centos8/x86_64/gcc/9.4.0/bin/gcc
Backend: Aggregate Results? YES

Run Configurations
[ {'name':'CUSTOM', 'kernel':'Gather', 'pattern':[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15], 'pattern_gather':[], 'pattern_scatter':[], 'delta':8, 'deltas_gather':[], 'deltas_scatter':[], 'length':1024, 'agg':10, 'wrap':1, } ]

config  time(s)      bw(MB/s)    
Pattern Length: 16 Max Pattern Value: 15

Pattern Length: 16 Max Pattern Value: 15


Running Spatter version 0.4
Compiler: GNU ver. 9.4.0
Compiler Location: /projects/opt/centos8/x86_64/gcc/9.4.0/bin/gcc
Backend: Aggregate Results? YES

Run Configurations
[ {'name':'CUSTOM', 'kernel':'Gather', 'pattern':[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15], 'pattern_gather':[], 'pattern_scatter':[], 'delta':8, 'deltas_gather':[], 'deltas_scatter':[], 'length':1024, 'agg':10, 'wrap':1, } ]

config  time(s)      bw(MB/s)    

Running Spatter version 0.4
Compiler: GNU ver. 9.4.0
Compiler Location: /projects/opt/centos8/x86_64/gcc/9.4.0/bin/gcc
Backend: Aggregate Results? YES

Run Configurations
[ {'name':'CUSTOM', 'kernel':'Gather', 'pattern':[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15], 'pattern_gather':[], 'pattern_scatter':[], 'delta':8, 'deltas_gather':[], 'deltas_scatter':[], 'length':1024, 'agg':10, 'wrap':1, } ]

config  time(s)      bw(MB/s)    
Pattern Length: 16 Max Pattern Value: 15


Running Spatter version 0.4
Compiler: GNU ver. 9.4.0
Compiler Location: /projects/opt/centos8/x86_64/gcc/9.4.0/bin/gcc
Backend: Aggregate Results? YES

Run Configurations
[ {'name':'CUSTOM', 'kernel':'Gather', 'pattern':[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15], 'pattern_gather':[], 'pattern_scatter':[], 'delta':8, 'deltas_gather':[], 'deltas_scatter':[], 'length':1024, 'agg':10, 'wrap':1, } ]

config  time(s)      bw(MB/s)    
0       1.89e-05     6934.292667 

Min         25%          Med          75%          Max         
0       1.683e-05    7786.147083 

Min         25%          Med          75%          Max         
0       1.534e-05    8543.901962 

Min         25%          Med          75%          Max         
8543.9       8543.9       8543.9       8543.9       8543.9      
H.Mean       H.StdErr    
0       1.532e-05    8553.380318 

Min         25%          Med          75%          Max         
8553.38      8553.38      8553.38      8553.38      8553.38     
H.Mean       H.StdErr    
8553.38      0           
8543.9       0      
6934.29      6934.29      6934.29      6934.29      6934.29     
H.Mean       H.StdErr    
6934.29      0           
7786.15      7786.15      7786.15      7786.15      7786.15     
H.Mean       H.StdErr    
7786.15      0         

I would like to add an MPI reduction and output format for MPI runs when USE_MPI is enabled that is more readable and easily parsable.