Contents

• Code Description
• Files in this Distribution
• Building the Code
• Running the Code
• Input File Format
• Output Format
• Parallelism and Scalability Expectations
• Timing Issues
• References
• Release and Modification Record
• Code

Code Description

The Sphinx Parallel Microbenchmark Suite

A. General Description

Sphinx, an integrated parallel microbenchmark suite, consists of a harness for running performance tests and extensive tests of MPI, Pthreads and OpenMP. It was adapted from the Special Karlsruhe MPI (SKaMPI) Benchmark suite¹ by Bronis R. de Supinski and other members of the PSE/ASDE project, including John May and Bor Chan. LLNL adaptations include extensive tests of the Pthreads interface² and on-going integration of the LLNL OpenMP Performance Suite. In addition, several new MPI tests have been added, primarily focusing on the performance of collective operations, including the first widely available tests that accurately measure the operation latency of fan-out collective operations such as MPI_Bcast³. Sphinx was a significant aspect of the ASCI PSE Milepost Run Time Systems Performance Testing; a paper on this effort should be available in the near future. The entire suite is implemented in C and has been run on a wide variety of platforms.

The Sphinx test harness provides a flexible mechanism for running performance tests. The action being measured, such as a message pingpong, is accessed through a function pointer. Different threads or tasks can execute different functions, which supports the measurement of highly complex parallel actions. The Sphinx harness provides a flexible mechanism for varying parameters, or independent variables, of the action. The harness times repeated calls (iterations in Sphinx terminology) of the action. Sphinx measures several timings (or repetitions) and outputs for their arithmetic mean for a given set of parameters for the action. The timings are stopped when the standard deviation of the repetitions is less than user-determined percentage of their mean, given that a minimum number of repetitions have been measured. Since this cut-off may never be achieved, the harness guarantees test termination through a user-specified maximum number of repetitions. Results that do not achieve the cut-off are annotated as "UNSETTLED" in the output. Sphinx includes a facility that can optionally be used to correct for harness overhead including that of the function call to the action routines. Results that have anomalous corrections are annotated "UNRELIABLE" in the output. Anomalous corrections include the overhead exceeding the measurement value; complex tests, such as our accurate tests of the operation latency of fan-out collectives, can have more complicated anomalous correction conditions.

Sphinx is highly portable. The primary portability issue involves the use of processor bindings in the Pthreads tests. The mechanism for binding threads to processors varies widely between platforms, both in interface and in capabilities. Sphinx includes a module that abstracts most of these issues. However, care needs to be exercised as several platforms have failed to exhibit appropriate binding behavior. Another possible portability snag is that the code currently must be linked with MPI and Pthreads libraries; future changes will support a "no Pthreads" version; a "no MPI" version may also be implemented. Other issues for portability include the current Makefile mechanism; this will probably be replaced by an autoconf script in the future.

B. Coding

Sphinx is implemented in C. It includes tests of three popular parallelization mechanisms: MPI, Pthreads and OpenMP. The OpenMP tests are excluded by default; they are included only if the macro _HAVE_OMP_ is defined. Generally, the choice of which tests are available in the executable is decided by the makefile target chosen.

C. Parallelization

The code tests MPI, Pthreads and OpenMP. The MPI tests include tests of the full range of MPI collective communications and pingpong tests of a variety of MPI send and receive combinations; other MPI tests are planned. Tests of most Pthreads functions are included and of all OpenMP constructs, plus the auxilliary OpenMP locking functions.

To build the code, type "make ARCH_COMPILER_MPI_OPTION" where ARCH_COMPILER_MPI_OPTION names the platform/compiler/MPI implementation that you want. For example, to build the IBM SP version that uses IBM's MPI library, type "Make IBM" while "Make IBM_MPICH" uses MPICH instead. For the full list of currently supported ARCH_COMPILER_MPI_OPTION choices, see the Makefile. It may be necessary to edit the Makefile.ARCH.COMPILER.MPI_OPTION file depending on the location of your compiler, etc. or to add a new one.

The basic command for running the code is:

sphinx_version input_filename

If input_filename is omitted then a simple MPI pingpong test is run. The exact command to run the code depends on the system (i.e. use poe on IBM SPs). Use the machine's parallel job start up mechanism to determine the number of MPI tasks. The OpenMP function omp_get_max_threads determines the limit on OpenMP threads although a lower limit can be specified in the input file.

This section details the input file format for Sphinx and the tests included in this distribution. Sphinx determines parameters for a run by reading and parsing an input file. The parameters for the run determine which tests to perform and variables that control the harness operation, such as the number of iterations per repetition. The input file format consists of several different "modes" that determine what run parameters are being specified. The tests to perform are specified in one or more MEASUREMENTS modes. MEASUREMENTS modes consist of a list of tests to perform; test entries include test-specific parameters. The input file format may seem complex due to the flexibility provided by the test harness. However, experienced users can quickly create the desired input file, due in part to the flexibility of the input file format.

In general, Sphinx input file processing is very forgiving - mode or parameter identifiers must contain a string that uniquely determines them but can contain random other characters, allowing input file processing to complete even with most typos. Further, the determining string is not case specific. Each test in Sphinx has several different possible parameters. A sensible default is used if the input file doesn't specify anything for a given mode or parameter. The default values can be overridden for the entire input file or for a specific test.

The Sphinx input file format is fairly free form; an "@" as the first character of a line changes the input mode and modes can occur in any order. Most of the modes are optional; the only mode that is required is at least one MEASUREMENTS mode. The last occurence of a mode determines the value for that mode except for the COMMENT and MEASUREMENTS modes; for these two modes, multiple occurences are concatenated to form a single value. The following table describes the Sphinx input file modes (names listed in ALL CAPS for historic reasons):

Many of the modes have list variants, as indicated. These allow the specification of different defaults for independent variables X0, X1, X2,... These variants are needed since Sphinx supports multiple independent variables per test, such as varying both the message size and the number of tasks for a MPI collective test. If a test uses more independent variables than specified in a corresponding list, then the non-list default is used for the additional defaults.

The MEASUREMENTS MODE is a structured text field. It describes the tests that will be run for the input file. The format is a series of test descriptions; blank lines between test descriptions are discarded. A test description consists of a name followed by a left curly brace ({) (optionally on a new line), followed by parameters specific to the test; a right curly brace (}) marks the end of the description. Each parameter field of a test description must be on a separate line; the general format of parameter field line is "parameter_name = value". The following table describes the parameter fields of the test description:

All fields of a test description other than the type field are optional. Test descriptions often consist only of a name, a {, a Type = X line and a }. Properly specified defaults enable this simplicity.

A measurement name is text string without spaces and need not have any relation to the type. This is unfortunate; future versions may augment names in the output file with a type specific string.

If the same name is used for several test descriptions, Sphinx will automatically extend the second and later occurences with a unique integer. This mechanism ensures that all test descriptions result in a test run.

Some independent variables do not alter anything for some test types. In general, an effort has been made to allow variation of these variables although some may lead to internally detected errors. In any event, care should be taken with independent variable selections to ensure that test descriptions test interesting variations and to ensure that overall run-time is not excessive.

The preceding table provides only a brief description of the tests and their results. The referenced papers provide further detail. Of course, a complete understanding can only result from careful consideration of the code. For details of the corrections used when the correct for overhead mode is used, consult the code.

Linear scales are reasonably intuitive; a STEPWIDTH of the square root of 2 will result in doubling the previous value with a fixed logarithmic scale. The default stepwidth actually varies with the scale since a STEPWIDTH of 1.00 would not result in any variation with logarithmic scales. The default is the ssquare root of two with logarithmic scales.

The effect of varying work functions should be limited to cache effects. A future paper will present results addressing the validity of this expectation.

The standard OpenMP names for the scheduling options are actually sufficient since Sphinx uses a non-case specific minimum string mechanism to determine the value. Support for the OpenMP runtime schedule option may be added in the future.

The log mechanism inherited from SKaMPI allows multiple runs of the same input file to run the full set of test descriptions to completion. If the log file contains and end of run message, then the log file and output file are moved to file names extended with an integer and a new run of the full set is started. Sphinx includes corrections to some bugs in this mechanism. These corrections ensure that each test description is run to completion exactly once, regardless of its name or the status of partial runs.

Sphinx retains a somewhat cumbersome output mechanism from SKaMPI. In particular, Sphinx can generate output to four different places: an output file, a log file, stdout and stderr. The stream to stdout comprises only simple informational messages; generally it can be redirected to /dev/null without consequence. The stream to stderr includes any error messages, in addition to some informational messages; since most of the errors are also directed to the log file, it can also be redirected to /dev/null without consequence in MOST cases (no promise that all error messages are duplicated). The log file also includes mostly informational messages. It is used for the restart mechanism, which can be very useful when tests are run through a batch system.

The output file is the most interesting of the Sphinx output streams. It is divided into two sections. The first section contains a summary of the default parameters, a listing of several of the text input modes and (often most importantly) a dump of the environment pointer. The second section contains the output records corresponding to the test descriptions in the input file. These output records again are divided into two sections. The first section contains a summary of the test description. The second section contains the results for the data points specified by the test description (run-time settings, such as number of MPI tasks also affect data point selection). Each line of this section is an output result; the first n fields are the independent variable values, the next field is the test timing result in microseconds. The timing result is followed by the number of timings, the standard deviation of the timings that produced the result and then any informational flags for the timing set, such as UNSETTLED, which indicates that the standard deviation of the timings did not achieve the cut-off percentage.

Timing results are in a per "operation" form; thus the standard deviation generally applies to the timing result times the number of iterations of the operation. In addition, results may also have been corrected for overhead, which can be significant, particularly for the accurate tests of fan-out MPI collectives.

This benchmarks have been run on a wide variety of platforms: IBM SPs with four-way, eight-way and sixteen-way SMP nodes; Clusters based on Compaq Alpha four-way and eight-way SMP nodes; A cluster based on SGI Origin 2000 256-way nodes; and several Sun Sparc based systems with SMP nodes. It has also been used with MPICH-g for Grid-based computing tests. The largest runs have used over 1500 MPI tasks. The current test harness supports these large runs although some of the collective tests do not scale well. Care should be exercised in the choice of tests when running large scale MPI tests; even then poorly scaling MPI implementations will be difficult to test at large scales. The Pthreads tests are generally designed to require no more than three active threads at a time; additional tests that use more concurrent threads may be implemented in the future. All of the OpenMP tests scale up to at least 16 threads; their scaling should depend solely on the scaling of the implementation being tested.

The code uses MPI_Wtime to measure wall clock time throughout its tests. Future versions may include the option to use the Unix gettimeofday of call so a no MPI version can be built. The time to run the code depends on the tests selected, the implementations being tested and the scale of the system being tested. The Pthreads tests typically can all be run in a few minutes; the OpenMP and MPI tests can take much longer.

1. R.H. Reussner, "User Manual of SKaMPI, Special Karlsruher MPI-Benchmark," Tech. Report, University of Karlsruhe, 1998.

2. B.R. de Supinski and J. May, "Benchmarking Pthreads Performance," Proc. of the 1999 Intl. Conf. on Parallel and Distributed Processing Techniques and Applications, 1999, pp. 1985-1991.

3. B.R. de Supinski and N. Karonis, "Accurately Measuring Broadcasts in a Computational Grid," Proc. of the 8th Intl. Symp. on High Performance Distributed Computing, 1999, pp. 29-37.

Version 1.0 (April 4, 2001)

and LLNL Disclaimers

UCRL-MI-143659
Revised April 4, 2001

UCRL-CODE-99026