Operational procedures for formal validation testing of PHIGS

All PVT documentation can be found under PHIGS Validation Tests - Overview.

The major sections of this document are:

  1. Why Test PHIGS?
  2. NIST services
  3. Prices for Validation
  4. Glossary and Acronyms
  5. Procedure for PHIGS validation

Why Test PHIGS?

The goal of portability can be achieved only if implementations adhere to the standard. Conformance tests are the means by which suppliers and users of PHIGS can measure objectively how well a putative implementation supports the features of PHIGS. For suppliers, a conformance test suite and validation service provide several benefits: 1) the test suite can enhance internal test facilities and therefore assist in quality assurance during the development of their PHIGS products, 2) validation provides assurance to all potential customers that their product has been tested and approved by a neutral third party, and 3) official validation is needed to allow vendors to bid on certain Federal procurements. For users, the availability of conforming PHIGS products provides a wide choice of graphics software without locking them into a single supplier. A second benefit is that the use of a standard application programmer interface for graphics can reduce training and maintenance costs, since programmers become skilled in an industry-wide standard, rather than in vendor specific techniques.

NIST services

The National Institute of Standards and Technology (NIST) has developed an extensive set of test programs to determine whether implementations of PHIGS conform to the specifications of ANSI/ISO standard 9592.1-1989. The PHIGS Validation Test (PVT) software may be obtained by implementors and users who wish to measure the conformance of a PHIGS system. Currently, the PVT is available in full Fortran and C. NIST maintains the PVT and issues new releases of the current version for the purpose of error correction and minor functional enhancement. The tests are the basis for the PHIGS validation service offered by NIST. Implementors who successfully undergo this process receive certificates of validation and the results are published in the Validated Products List, available from NIST. Federal agencies may require software products procured for graphics applications to be on the Validated Products List.

Prices for Validation

The base price for PHIGS validation is $17,500. If several types of platform or several language bindings are to be tested on-site at the same time, the charge is $5,000 for each additional platform/language combination. For each platform or language tested, the charge is $2,000 for each additional workstation type (beyond the first) on that platform or language. For example, suppose that on platform P901 you want to test the Fortran and C binding for three distinct workstation types, and on platform P902 you want to test just the C binding with two workstation types. The cost would be: 
 $17,500 : P901/Fortran (includes first workstation) 
   4,000 : P901/Fortran two additional workstations 
   5,000 : P901/C (includes first workstation) 
   4,000 : P901/C two additional workstations 
   5,000 : P902/C (includes first workstation) 
   2,000 : P902/C one additional workstation 
---------------------------------------------
 $37,500 : total
The fee for registration of a platform as a result of self-testing is $250 per platform, up to a maximum of $2,000 per client.

Glossary and Acronyms

The following terms and acronyms are used in the description of the validation process:
Implementation under test (IUT):
the implementation of PHIGS to be tested for conformance to the PHIGS standard. An IUT is not simply a subroutine library, but the entire system responsible for accepting valid PHIGS programs and producing graphical output. It typically includes a programming language compiler, subroutine library, device driver, the computer itself, and at least one graphical workstation.
Workstation:
this term is used in the sense defined by the PHIGS standard -- a destination for the display of graphical output data and/or a source of graphical input data.
PHIGS Validation Tests (PVT):
the software used to examine the IUT for conformance to the PHIGS standard; also referred to as the test suite.
Testing laboratory (TL):
the laboratory directly in charge of performing on-site validation, using the PVT
Maintenance authority (MA):
the laboratory responsible for maintenance of the test suite software; the MA for the PVT is NIST.
Control board (CB):
composed of TLs, the board resolves disputes concerning the correctness of the test suite with respect to the PHIGS standard.
Validated products list (VPL):
the list maintained by NIST of computer products that have been validated for conformance to some relevant standard. In some circumstances, products that are found to be in only partial conformance may nonetheless be entered on the VPL (see Registration of client testing results, below). Many Federal procurements accept bids only for products on the VPL.
Validation summary report (VSR):
the report produced as a result of on-site validation by a TL, regardless of the outcome of the testing.
Registration:
a product is registered when it is entered on the VPL, and its VSR is publicly available.
Certificate of validation:
when an IUT is found to be in full conformance with the standard, it is granted a certificate of validation.

Procedure for PHIGS validation

The main steps in having a PHIGS implementation formally validated are:
  1. obtain copy of PVT source code
  2. make approved modifications to source code if needed
  3. perform pre-validation: execute PVT on candidate implementation
  4. submit pre-validation materials to TL
  5. schedule on-site validation by TL
  6. TL performs on-site validation
  7. TL issues VSR
  8. if it meets validation criteria, IUT is added to VPL
  9. if no errors detected, IUT receives certificate of validation
Note that all these steps apply to each distinct implementation under test (IUT) to be validated. The following sections are organized chronologically according to the steps a client must go through in order for an IUT to be registered and to receive a certificate of validation.

Availability and Use of PVT Source Code

The client must obtain a copy of the current version of the PHIGS Validation Tests (PVT). See distribution procedures for details. Currently, the available host languages are full Fortran and C. The suite may be used for purposes other than conformance testing, e.g. for debugging during development of an implementation. The test suite includes documentation on installation and operation. The client may then use the test suite for preliminary evaluation of the PHIGS implementation in question.

Modification of Source Code

Some implementations may require modifications to the PVT source code. The client must submit any proposed modifications to the TL for approval. Generally, no changes to the tests are allowed beyond those specifically mentioned in User's Guide. By definition, standard implementations accept standard programs and produce standard results; since the test programs conform to the standard, an implementation which fails to accept them as is does not conform. For the PVT, these customizations are limited to 1) addressing lack of functionality in the host language (e.g. standard Fortran has no statement to position to end-of-file, nor a random number function) and 2) allowing mechanisms other than the configuration file to supply information needed by <open phigs>, <open archive>, and <open workstation>, since the configuration file assumes static information and this is not mandated by the standard. Note in particular that the modified programs must conform to relevant standards (Fortran, C, PHIGS).

Pre-validation testing by Client

The client performs pre-validation by running the entire test suite with a workstation of category OUTIN designated as the primary workstation. This checks the minimum requirement for a conforming IUT. Additional workstation types are then tested by additional sessions executing the subset of programs appropriate to that workstation's category. Note that in this context, a "workstation type" refers to a physically distinct model of a workstation, not simply to the parameter submitted to <open workstation>. The results of each session must be captured in a distinct global message file. Each message file must contain results from all the required programs for that session. If a program cannot start execution or cannot be executed to completion, this omission must be documented by using the OPRCMT utility program. See the PVT User's Guide, Version 2.1 for operational details.

Questions during pre-validation

Questions regarding the interpretation of the standard and the validity of the tests should generally be resolved during pre-validation between the client and the TL. Any changes to the test suite requested by the client or TL must be approved by the MA.

Resolution of pre-validation disputes

If the client, TL, and MA cannot reach agreement on the validity of a test case, the client must submit a brief written statement to the TL locating as precisely as possible the section of code being challenged and justifying its contention that the test is invalid. The issue is then sent to the PHIGS Testing Control Board (CB) for disposition. The TL and MA may also send comments on the issue to the CB. If a test case is judged to be invalid, either informally by the MA or formally by the CB, the offending code will be withdrawn by the MA, and corrected in later releases. If the test case is sustained as valid, the client either accepts the test result or modifies the IUT to produce the required behavior.

Corrections following pre-validation

Prior to on-site validation, clients may correct implementation errors (behavior not in conformance with the standard) detected during pre-validation. Therefore, the results of on-site validation may differ from the pre-validation results. The client assumes all responsibility for any operational difficulties or delays encountered during on-site validation resulting from such changes.

Payment

If the client wishes to proceed with an official validation, the TL should be notified, and a purchase order or check issued for the appropriate amount.

Schedule on-site validation by the TL

Since the demand for validations may fluctuate, it is recommended that the client schedule a mutually agreeable time with the TL for the on-site validation as far in advance as possible (normally allow 8-16 weeks).

Configuration Information

The client submits to the TL a signed statement detailing the exact configuration in which the pre-validation sessions were executed, including at least: These items should specify all relevant version and release numbers of the system(s) to be tested. The pre-validation and on-site validation must be performed under exactly the same conditions to ensure consistency of results. The VSR and certificate of validation to be issued by the TL pertain only to the configuration(s) actually tested.

TL inspection of pre-validation results

The client submits to the TL for its inspection all of the following for each IUT to be validated: This information (except the last item) is to be supplied in machine readable form via e-mail, UNIX tar tape, DOS diskette or some other mutually agreed format. Documentation of implementation dependent and workstation dependent features may be supplied in machine readable form or on paper. The TL examines the configuration information and pre-validation results. If these are incomplete, the client is notified and must supply the missing information. All pre-validation information must be received by the TL no later than three (3) weeks prior to the start of on-site validation. The TL reserves the right to postpone on-site validation due to incomplete pre-validation information.

On-site validation

TL personnel perform the validation at the site designated by the client in the Scheduling Request. The TL installs a new copy of the PVT source code, including client modifications which have been submitted and approved. There should be a separate account and file directory set up for use by the TL. The client is responsible for providing full support to the TL, including adequate hardware, software and personnel so as to make the PVT operational. In particular, all the hardware and software documented in the Configuration Information must be available. Submission of operational procedures in the pre-validation material should minimize the need for personal assistance. No software changes to the IUT are permitted once on-site testing of an implementation has begun, unless the TL, in its sole discretion, explicitly agrees to waive this requirement. No changes are permitted to the supporting hardware except to repair unforeseen hardware failures (such as a disk crash). Some tests involve visual inspection of graphical output. The TL operator is the ultimate judge of whether the appearance of the screen satisifies the test criteria. Validation results may include notes on non-compiling or aborted tests documented by OPRCMT, and by any other appropriate means (drawings, photographs, etc.). The TL and the client may discuss any disputes that arise during on-site validation; this may involve inspection of the relevant standard and of the test suite. However, no resolution of disputes takes place during validation.

Resolution of validation disputes

Disputes are handled in the same way as during pre-validation. The client and TL and possibly the MA attempt to resolve issues informally; otherwise the question is referred to the CB. The test results are not issued by the TL until all outstanding disputes have been resolved by the CB.

Client release of VSR

The TL prepares a draft VSR for the IUT, describing any errors that may have been detected during validation, and indicating the version and release number of the PVT used. The TL sends the VSR to the client together with a release form stating that the VSR is available to the public and to procuring agencies. If the client does not wish to release the report, no further action is taken by the client or the TL. Otherwise, the client signs the release form and returns it to the TL. The VSR then becomes publicly available.

Registration of validation results

If the VSR indicates that there are uncorrected errors from the previous validation of the IUT, the IUT is not entered on the Validated Products List (VPL) and no further action takes place. If no errors or only "new" errors (not detected in the previous validation) are found, then the IUT is entered on the VPL, and is thereby registered. If no errors were found, the IUT is issued a certificate of validation and its registration stays in effect for two years. If some errors were detected, no certificate is issued and the IUT's registration expires after one year.

Renewal of validation

A certificate of validation for an implementation may be renewed for additional years, year-by-year, if the following conditions are met:

Registration of client testing results

Once an implementation has been validated on-site by the TL, clients may in certain circumstances register additional implementations for entry on the VPL without on-site validation. The target implementation to be registered must be closely related to a reference implementation that has been validated directly and registered within the past year. Only those workstation types which were validated for the reference implementation are eligible for testing with the target implementation. The target and reference implementation may differ in only one of the following ways (not a combination thereof): The client performs all the steps described above, up to but excluding the on-site validation by the TL (that is, self-testing resembles the pre-validation steps of a full validation). The client submits a signed statement to the TL along with the validation materials affirming that those materials are the result of executing the tests in the specified environment. The premise of self-testing is that the target implementation is functionally identical to the reference implementation; therefore, the result files of the two validations must be identical. All self-tested implementations are subject to challenge by the TL and other interested parties (such as procuring Federal agencies). If the TL inspection reveals that a self-tested implementation does not behave in accordance with the submitted validation material, all entries in the VPL for self-tested implementations dependent on the associated reference implementation are stricken.

Testing laboratories

For more information, contact: John Cugini