B Appendix B: Qualification Test Report............................................

B.1 Overview.................................................................................................

B.1.1 New Voting System Qualification Test Report................................

B.1.2 Changes to Previously Qualified Voting System Qualification Test Report        

B.2 Introduction.............................................................................................

B.3 Qualification Test Background.............................................................

B.4 System Identification.............................................................................

B.5 System Overview...................................................................................

B.6 Qualification Test Results and Recommendation..............................

B.7 Appendix - Test Operations and Findings..........................................

B.8 Appendix - Test Data Analysis.............................................................

 

B                                                             Appendix B: Qualification Test Report

This Appendix contains a recommended outline for the Qualification Test Report to be prepared by the test agency. The test report shall be organized so as to facilitate the presentation of conclusions and recommendations regarding system acceptability, a summary of the test operations, a summary of the test results, the test data records, and the analyses that support the conclusions and recommendations. The content of the report may vary based on the scope of review conducted.

A full report is prepared for the initial qualification testing of a voting system. This document consists of five main sections: Introduction, Qualification Test Background, System Identification, System Overview, and Qualification Test Results. 

Detailed information about the test operations and findings, and test data, are included as appendices to the report.

Sections B.2 through B.8 describe the contents of the individual sections of this report.

This report addresses a wide range of scenarios.  After a preliminary review of the submitted changes, the test agency may determined that:

a.   A review of all change documentation against the baseline materials was sufficient for recommendation for qualification, or

b.   All changes must be retested against the previously qualified baseline, or

c.   The scope of the changes are substantial enough such that a complete retest of the software is required. 

The format of this report varies, based on the type of review that was performed.  If only a review of change documentation against the baseline materials was performed the report is quite simple.  It consists of an Introduction, a Version Description, the Testing Approach, and a Results Summary. A more extensive report is prepared, for changes that have extensive impact on the system design and/or operations.

This section details the organization of the test report document and provides background information about the vendor and the software test agency. This section clearly indicates whether testing was conducted for an initial system qualification or system update release.

This section contains the following information:

a.       General information about the qualification test process;  and

b.       A list and definition of all terms and nomenclature peculiar to the hardware, the software, or the test report;

This section gives information about the tested software and supporting hardware, including:

a.       System name and major subsystems (or equivalent);

b.      System Version;

c.      Test Support Hardware; and

d.      Specific documentation provided in the vendor's TDP used to support testing.

This section describes the voting system in terms of its overall design structure, technologies used, processing capacity claimed by the vendor for system components (such as ballot counters, voting machines, vote consolidation equipment) and mode of operation. It may also identify other products that interface with the voting system.

This section provides a summary of the results of the testing process, and indicates any special considerations that affect the conclusions derived from the test results. This summary includes:

a.       The acceptability of the system design and construction based on the performance of the system hardware, software and communications, and on the source code inspection;

b.       The degree to which the hardware and software meet the vendor's specifications and the standards, and the acceptability of the vendor's technical and user documentation;

c.       General findings on the maintainability of the system including, where applicable, notation of specific maintenance activities that are determined to be difficult to perform.

d.       Identification and description of any deficiencies that remain uncorrected after completion of the qualification test and that has caused or is judged to be capable of causing the loss or corruption of voting data, providing sufficient detail to support a recommendation to reject the system being tested. (Similarly, any deficiency in compliance with the security, accuracy, data retention, and audit requirements are fully described); and

e.       A specific recommendation to the NASED ITA Committee for approval or rejection.

Of note, any uncorrected deficiency that does not involve the loss or corruption of voting data shall not necessarily be cause for rejection. Deficiencies of this type may include failure to fully achieve the levels of performance specified in Volume I, Sections 3 and 4 of the Standards, or failure to fully implement formal programs for qualify assurance and configuration management described in Volume I, Sections 7 and 8. The nature of the deficiency is described in detail sufficient to support the recommendation either to accept or to reject the system, and the recommendation is based on consideration of the probable effect the deficiency will have on safe and efficient system operation during all phases of election use.

This appendix provides additional detail about the test results to enable the understanding of test results and recommendation. This information is organized in a manner that reflects the Qualification Test Plan. Summaries of the results of hardware examinations, operating and non-operating hardware tests, software module tests, software function tests, and system-level tests (including security and telecommunications tests, and the results of the Physical and Functional Configuration Audits) are provided.

This appendix provides summary records of the test data and the details of the analysis. The analysis includes a comparison of the vendor's hardware and software specifications to the test data, together with any mathematical or statistical procedure used for data reduction and processing.