The ASCI sPPM Benchmark Code

Contents

Building Considerations

The problem size per process is established in iq.h. The layout of processes, number of threads, and number of time steps to run are established in run/inputdeck. The rest of the build is controlled by the Makefile.

For purposes of most of this benchmark, the code is to be run in double precision (64 bit) on a 192x192x192 per process job with no dumps or other output files being written, and it is to run for 20 timesteps. Any threading is to be done with OpenMP. The exception is the final run, which can use pthreads rather than OpenMP, if desired, can use single precision rather than double precision and can use a different problem size.

To build and run the code with the above general constraints,

• in run/inputdeck you should have "stoptime 20 1.0"
• in iq.h you should have IQX, IQY, IQZ, and IQ all set to 192
• in Makefile you should have "FPSIZE= -DREAL=double" and "MOPT= -DBOBOUT=0 -DDUMPS=0 -DNOCHDIR=1"

• FOPT3 is used for Fortran compiler options, including whatever option is needed to indicate that default real variables and constants are to be taken as if declared double precision.
• COPT is used for C compiler options, including a setting of the F2C macro name to indicate the linkage convention between Fortran and C (see the comments in c_io.c).
• TOPT can be used for options to be passed to the Fortran compiler, the loader, and possibly the C compiler.

Note that it may be necessary to modify timers.c in order to use appropriate timer routines.

To set up for builds that are to use OpenMP, Makefile should include "THMODE= -DTHREADED=1 -DOPENMP=1," OMPOPT should include any options needed by the Fortran compiler and/or loader to compile and load OpenMP code, COMPOPT should include any C compiler options needed for OpenMP, and THLD can be used for any additional loader options needed for OpenMP. The number of threads to use is determined at execution time by nthreads in the input deck. For builds that do not use OpenMP (or pthreads) you should set "THMODE= -DTHREADED=0."

Performance optimizations to the sPPM source are encouraged as long as they don't specialize the functionality. However, the hydro kernel (i.e., the sppm.m4 source file) must be used UNMODIFIED except for the insertion of compiler directives. Alternative coding can be selected through the m4 defines (ifsupr, ifcvmg, ifinln, and ifcray) found in the sppm.h file. Also note that changes outside of sppm.m4 can affect it. For example, it is possible to use the IQ parameter in iq.h to affect the array sizes in these routines. The goal is to emphasize higher-level optimizations and/or compiler technology instead of relying on hand-tweaked specialized code.

The benchmark must use MPI message passing between nodes. It cannot overlap the boundary communications in the BDRYS routines (in bdrys.m4) with the directional sweeps in the CALCHYD routines (in runhyd3.m4) because we want to measure communication separately; however, overlapped communication and computation will be allowed for the final TeraFLOP demonstration.

The wall-clock execution time required to execute sPPM should be reported. The minimum hardware memory required to run each problem should also be reported.

The code is extensively instrumented. The individual steps in each double timestep are timed. Each double timestep has both "delta" and cumulative timings. Processing and I/O for each restart and graphics dump is timed. Code totals are also printed. Node 0 prints all of this information to stdout. Also, every node prints its own timing information to its own output file (in the node's subdirectory). The cpu/wall ratio gives an indication of multithreaded efficiency. The graphics image I/O time includes necessary reformatting and scaling to write "bricks of bytes."

Only the nodelayout (npx, npy, and npz), nthreads, and (possibly) stoptime should be changed to run the benchmark problems.

Sample output files are provided in the Sample_Outputs subdirectory for your reference in determining correctness of answers.

We have run on SGI -IA64(Linux), SGI (Irix), IBM (AIX), Compaq Sierra AlphaServer (Tru64 Unix) and SUN (Solaris) systems.

The critical numbers you should duplicate are the Courant numbers and energies in each timestep print. They should be duplicated exactly, because they are only printed to 6 significant digits. We have seen consistent results from all machines to date.