Summary

This document outlines the current design and implementation for a Web-based IMPI conformance testing system. This is a work in progress, comments and suggestions are welcome.

This tester is designed to verify the correct implementation of the IMPI specific protocols and not to test an MPI implementation against the MPI specifications. The full testing of MPI is well beyond the scope of this tester. This version of the IMPI tester operates only within the C implementation of MPI. The testing of IMPI within a Fortran or C++ environment is not yet planned.

To help describe this testing environment, some conformance testing terminology will be used. The implementation of the IMPI protocol to be tested will be referred to as the Implementation Under Test (IUT). The IUT will be running on a System Under Test (SUT). The SUT refers to all the hardware and software needed to run the IUT. The person running the test will be called the tester.

The steps taken by a tester to run the IMPI conformance tests are outlined below. Figure 5.1 shows this testing scheme graphically. The numbers in this figure indicate the order in which the communications channels are established initially. The dashed lines are HTTP Web communication (connection-less communications, stateless). The solid lines are connection-oriented TCP/IP sockets. The dotted line is direct tester interaction with the SUT.

• The Tester connects to the NIST IMPI web page at http://impi.nist.gov/IMPI. (Figure 5.2) with a Java-enabled browser. The current version of the IMPI standard (this document) is available from this web page.
• Follow the IMPI Test Tool link to the main IMPI Test Tool page. This page gives a short description of the major parts of the IMPI testing software: the Test Interpreter, the Test Tool Applet, the Test Scripts, and the Test Server.
• Follow the Test Interpreter link to obtain the current version of the test interpreter. The user must compile and link this interpreter for the IUT on the SUT before continuing with the tests.
• Follow the Test Scripts link to view the source to the test scripts and descriptions of the available tests.
• Follow the Test Tool Applet when you are ready to begin testing. This page gives detailed instructions on running the tests.
• Selecting the Run the IMPI Test Tool Applet link will initiate the IMPI Test Tool (a java client applet) which will be downloaded and run on the tester's browser (see Figure 5.4). This applet establishes a permanent^5.1 TCP/IP connection with an instance of the IMPI Test Manager (the test server) on the NIST IMPI Web server. Test requests and results will be exchanged over this socket.

• The tester can choose the configuration of clients, hosts, and processes to be tested through the Test Tool. This includes choosing where the impirun -server process will be run, how many clients there will be, how many of these clients will be on the NIST side (simulated clients), how many hosts will be on each simulated client, how many processes will be on each of these simulated hosts, and what rank to assign to the first simulated client.

  This does not allow some valid configurations to be specified, such as varying the number of hosts per simulated client. If this becomes necessary for the proper testing of the IMPI protocol, then this interface will be modified to allow more general configurations.

• Once the configuration of clients and hosts is specified, the IMPI Test Tool allows the tester to run the IMPI Startup protocol only, for early testing of an implementation. The Run Startup Only button on the Test Tool initiates this test. This simply runs through the startup on each client with each process printing its rank to stdout for confirmation of the ordering within MPI_COMM_WORLD. Errors that occur during the startup communications will be reported to the tester through the test tool and possibly also through stdout on the SUT. All processes exit without attempting any message passing.

• The complete set of test scripts is made available if the Run Startup & Prepare for Scripts button is pressed instead of the Run Startup Only button.
• Immediately after either the Run Startup Only or the Run Startup & Prepare for Scripts buttons has been pressed, A sample impirun command line is shown to the tester, in the scrolling output window on the Test Tool. This command line includes the specific HOST:PORT needed as a command-line parameter to the IMPI clients. The tester must then run the IMPI test interpreter on the SUT using this command-line, or the equivalent command-line if the IUT uses a different syntax.

  In the future, a command-line template may be made available so that the tester can specify the required syntax for the IUT, with place holders for varying parameters such as the number of processes. This will allow the Test Tool to display the command-lines in the correct syntax for the IUT. This should help avoid errors in starting the IUT correctly for the various tests to be performed.

  

Figure 5.1: IMPI Test Architecture

Figure 5.3 shows the levels of communications protocols involved in this testing. We are most interested in verifying the operation of the IMPI level of this communications stack so the tests will originate in the MPI level (an upper-tester). We will also have a lower-tester which will be able to examine the IMPI protocol packets as they are received from the SUT. This lower-tester will be implemented by instrumenting the Model IMPI implementation directly. The Model IMPI implementation will be part of the IMPI Test Manager.

Once the tester starts the IMPI test interpreter on the SUT, the IMPI startup protocol will begin between the IUT and the Model IMPI implementation running on the NIST IMPI test server. If the startup protocol succeeds, the IMPI Test Manager will present to the tester a menu of options. The format used to display this menu will likely change as the number of available tests increases. If the startup protocol fails, error recovery will be attempted and as much information as possible about the startup negotiations will be provided to the tester.

  

Figure 5.2: IMPI Test Home Page

Assuming the startup protocol succeeded, the tester may select a test or group of tests to be run. The Test Manager will then begin running the tests by sending each test, one at a time, to the processes running on the SUT. Each test will determine the result, either Pass/Fail/Indeterminate, and this result will be sent to the Test Tool for display (see Figure 5.4). Once all of the requested tests have been run, the tester may select more tests or discontinue testing.

For each test, the tester may obtain, through the IMPI web page, the source for the script that is being interpreted. We might also provide links into the IMPI or MPI documents for the text that specifies the part or parts of the protocol that this code is intended to test. In the future, we may also provide a file for downloading which gives a summary of all the tests that have been run, including the results.

The full testing of the IUT must include varying the impirun command-line options. The command-line can specify relative the ranks of the SUT nodes within the MPI_COMM_WORLD group. Also, tests must be made using a single node of the SUT as well as multiple nodes. The tester must therefore follow instructions given in the Test Tool as to the proper impirun command-line to execute each time the test interpreter is started. Although it would be preferable to automate this, we do not yet, and probably will never, have this capability. However, we will at least provide the tester with command-lines in the Test Tool window to cut & paste.

Starting with Java SDK version 1.1, it will be possible to gain access to the disks on the machine running the Test Tool, as well as run external programs on that machine. This will be allowed if the tester certifies that the Test Manager is trusted. How this is will implemented in the browsers is not yet known. Many Web browsers in use are still using Java SDK 1.0, and those that have been updated to version 1.1 have not implemented this feature, so this is not yet an option. Even with this ability, it is not clear that we will be able to automate the running of impirun on the SUT.

Tests will also be provided to allow additional IMPI/MPI implementations to participate in the tests, in addition to the IUT. These tests will be needed to fully exercise the IMPI/MPI protocols.

  

Figure 5.3: IMPI Test Communications Stack