U.S. DEPARTMENT OF ENERGY

OFFICE OF CIVILIAN RADIOACTIVE WASTE MANAGEMENT

OFFICE OF QUALITY ASSURANCE

 

 

AUDIT REPORT

 

OF

 

UNITED STATES GEOLOGICAL SURVEY

 

AT

 

DENVER, COLORADO

 

USGS-ARP-98-01

 

OCTOBER 28-30, 1997

 

 

 

 

 

Prepared by: _____________________________ Date: ___________________

Richard L. Weeks

Audit Team Leader

Office of Quality Assurance

 

 

 

 

 

Approved by:_____________________________ Date: ____________________

Donald G. Horton

Director

Office of Quality Assurance

1.0 EXECUTIVE SUMMARY

 

As a result of Performance Based Quality Assurance (QA) Audit USGS-ARP-98-01, the audit team determined that U.S. Geological Survey (USGS) is satisfactorily implementing the Office of Civilian Radioactive Waste Management (OCRWM) QA Program and process controls for the Scientific Expert Elicitation as applied to the evaluation of seismic hazards in support of the AProbabilistic Seismic Hazards Analysis@ (PSHA).

The activity examined during this audit is in accordance with applicable portions of the U. S. Department of Energy (DOE) OCRWM Quality Assurance Requirements and Description (QARD) document DOE/RW-0333P, Revision 7. The audit team considers that development of the hazards analysis has been both adequate and effective to meet the project=s objectives based on implementation of established procedure controls and plans, adequacy of the process with respect to NUREG-1563 (Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program), and evaluation of available technical data. This evaluation was based on the progress made to date towards the final report scheduled for February 1998.

 

The audit team identified one deficiency corrected during the audit which is described in

Section 5.6.1. Additionally, 7 recommendations resulted from the assessment of the process adequacy and program documentation which are detailed in Section 6.0 of this report.

 

2.0 SCOPE

 

The audit was conducted to evaluate the adequacy and effectiveness of USGS controls for the implementation of the Scientific Expert Elicitation process in the development

of the deliverable for the PSHA. The audit was intended to determine the degree to which the resultant products meet the program requirements and management commitments and expectations, as well as to determine that USGS is implementing the work in accordance with pertinent sections of the QARD. In addition, an evaluation of the process used by USGS was compared to NUREG-1563 to measure the implementation of the NUREG-1563 guidance for the expert elicitation.

 

The process/activities/end products were evaluated during the audit, in accordance with the approved Audit Plan.

 

PROCESS/ACTIVITY/END PRODUCT

 

This audit evaluated the in-process documents generated to support the development of the PSHA for Work Breakdown Structure 1.2.3.2.8.3.6. These documents included level IV deliverables associated with the elicitation process workshops.

The performance-based evaluation of process effectiveness and product acceptability was based upon:

 

1. Satisfactory implementation of the critical process steps;

2. Use of trained and qualified personnel working effectively;

3. Documentation that substantiates the quality of the product;

4. Acceptable results and adequate end-product; and

5. Effectiveness of corrective action.

 

Technical Areas

 

The USGS processes for Scientific Expert Elicitation and its associated end products for the PSHA were evaluated for the following critical process steps:

 

 

The audit included a technical evaluation of process effectiveness and/or product acceptability. Details of the technical evaluation are included in Section 5.5.

 

3.0 AUDIT TEAM AND OBSERVERS

 

Team Member Responsible Area

Richard L. Weeks, Audit Team Leader Critical Process Steps and Management

Objectives

William J. Glasser, Auditor Critical Process Steps and Process Implementation

Robert P. Hasson, Auditor Critical Process Steps and Process Adequacy

 

David F. Fenster, Technical Specialist Technical Evaluation of Product

 

A. K. Ibrahim, NRC Observer

 

4.0 AUDIT TEAM MEETINGS AND PERSONNEL CONTACTED

 

The pre-audit meeting was held on October 28, 1997, at the USGS office in Denver, Colorado. Daily debriefing and coordination meetings were held with USGS management and staff, and daily audit team meetings were held to discuss audit status. The audit was concluded with a post-audit meeting held on October 30, 1997, at the USGS office in Denver, Colorado. Personnel contacted during the audit are listed in Attachment 1. The list includes those who attended the pre-audit and post-audit meetings.

 

5.0 SUMMARY OF RESULTS

 

5.1 Program Effectiveness

 

The audit team concluded that, overall, the USGS process controls are effectively being implemented for the areas identified in the scope of this audit.

 

5.2 Stop Work or Immediate Corrective Action Taken

 

There were no Stop Work Orders, immediate corrective actions or related additional items resulting from this audit.

 

5.3 QA Program Implementation of the Elicitation Procedure

 

The audit assessed the implementation of the USGS Procedure YMP-USGS-QMP-3.16 R0, AScientific Expert Elicitation.@ Requirements for selection and training of personnel, preparation and implementation of a project plan, and implementation of procedure requirements were evaluated. Implementation was evaluated for compliance to QA Program elements:

 

2.0 Quality Assurance Program

5.0 Implementing Documents

 

For all audit activities, the audit checklists contain the details of the audit evaluation along with identification of the objective evidence reviewed. The checklists are kept and maintained as QA records.

 

5.4 Adequacy of the Elicitation Process

 

The audit team evaluated the process used by USGS to conduct expert elicitation as compared to NUREG-1563. USGS has an established procedure, QMP 3.16, and Project Plan, YMP-USGS-EE-9701, R/0, AProbabilistic Analysis of Fault Displacement and Vibratory Ground Motion and the Development of Seismic Design Bases for Yucca Mountain,@ to perform the PSHA expert elicitation. The procedural controls established were evaluated for adequacy to the NUREG-1563 guidance. Overall, the process controls established to perform the expert elicitation were considered to be adequate. The procedure and project plan controls in place addressed most of the NUREG-1563 guidance. There were some NUREG-1563 criteria not addressed in the documents for areas such as conflict of interest, conducting the elicitation in a private setting conducive to uninterrupted discussion, documenting the rationale for revisions to elicited judgements, and identifying assumptions and unambiguous terms for the subject-matter experts. Although not addressed in the procedural controls, the NUREG-1563 guidance has been considered by USGS in the performance of the expert elicitation. In discussion with USGS personnel, USGS=s intention is not to revise the procedure and project plan since this elicitation is a one-time project. USGS=s plan is to cancel the procedure once the elicitation project is completed. USGS agreed to address these concerns in the PSHA final report explaining what was implemented and what was not and why. Four recommendations (#1-4) were made to address these concerns by the audit team.

 

The audit team also performed a review of USGS=s response to the Office of Quality Assurance=s (OQA) comments on USGS Procedure QMP 3.16. OQA issued a letter to USGS, (reference letter D. G. Horton to T. H. Chaney, dated August 8, 1997) stating that the review of USGS response to OQA comments on the Scientific Expert Elicitation procedure have raised concerns that the procedure did not meet the intent of NUREG-1563 and that OQA would follow up with a surveillance to evaluate the procedure to NUREG-1563. Based on the review of the procedure, the project plan, and discussion with USGS=s personnel during the audit, resolution of the questionable responses were agreed upon and will be revised by USGS and resubmitted to OQA for acceptance.

 

5.5 Technical Evaluation of the Elicitation Product

 

The technical audit checklist questions were based on technical evaluation that are a necessary part of conducting a probabilistic seismic hazard analysis. The checklist questions and supplementary questions were designed to evaluate how the expert elicitation process evaluated and documented conclusions related to: evaluation of seismic source zones; evaluation of earthquake recurrence and maximum magnitude; evaluation of ground motion/attenuation functions; and, probabilistic seismic hazard calculation for vibratory ground motion.

 

USGS audit participants were well-prepared and well-organized. All responses to checklist and supplementary questions were satisfactory. Based on these responses and on an examination of the objective evidence, the technical work performed was adequate to meet project objectives. We recognize that the PSHA Expert Elicitation has not been completed and that both the fault displacement evaluation and completion of the activity reports are ongoing efforts.

 

As a result of the above technical evaluation, two recommendation (#5 and #6) were made.

 

In conjunction with the technical portion of the audit, an evaluation of the implementation of requirements described in the Project Plan, YMP-USGS-EE-9701, AProbabilistic Analysis of Fault Displacement and Vibratory Ground Motion and the Development of Seismic Design Bases for Yucca Mountain,@ was performed. It was concluded that the requirements described in the Project Plan were effectively implemented. One recommendation (#7) regarding the security of records is being provided.

 

5.6 Summary of Deficiencies

 

5.6.1 Deficiency Corrected During the Audit

 

During review of the implementation of USGS Procedure YMP-USGS-QMP-3.16, R0, the audit team determined that two USGS staff personnel responsible for the management and implementation of the process had not documented their training to the subject procedure. All other personnel involved in this process had been trained (classroom training) and the training documented on an appropriate training record. Both individuals in question had been in attendance at the classroom training as subject matter experts but had not signed the attendance list. Since both individuals had been trained but the training not documented, the required USGS training record form was completed and reflects both the training date and the date the form was filled out as allowed by the USGS procedure. This deficiency is an isolated case and was satisfactorily corrected during the audit.

 

5.6.2 Follow up to Deficiency Report YM-97-D-039

 

During the annual compliance audit of USGS (USGS-ARC-97-15 conducted in April 1997), the subject deficiency was written to identify the untimely approval of the PSHA elicitation process Project Plan and the lack of attestation of qualifications for many of the participants in the process. The audit team determined that the approved project plan has been and is being followed as documented, even prior to the approval date. Training records were complete for all personnel involved in the PSHA process. A memorandum which was required by the Actions to Preclude Recurrence required a minor revision which was completed during the audit. A verification statement will be added to the subject deficiency with a recommendation for closure.

 

6.0 Recommendations

 

1. NUREG-1563 discusses the need for subject-matter experts to be willing to publicly disclose all potential conflicts of interest. Although not addressed in the USGS procedure or plan, USGS did provide documentation that was completed by the subject-matter experts regarding conflict of interest statements. A recommendation is made to document USGS implementation of Aconflict of interest@ in the PSHA final report.

 

2. NUREG-1563 states that the generalists and normative expert be in attendance for the complete elicitation session with each subject-matter expert in a private setting conducive to uninterrupted discussion. A requirement was not established in the procedure to address this guidance. It is recommended that USGS provide a statement in the PSHA final report as to who was in attendance fulfilling the generalists and normative function and where the elicitation took place that indicate implementation of this guidance.

 

3. NUREG-1563 guidance states that for all elicited judgements, the rationale for any revisions are required to be well documented. In discussion with USGS personnel, they do not feel this guidance benefits the experts, but actually Aanchors@ the experts and may possibly cause Abias@ among the experts. A recommendation is made that USGS document their approach regarding this issue in the PSHA final report.

 

4. NUREG-1563 guidance states that unambiguous definitions and specific terms be provided and any assumptions used in the elicitation be explicitly stated. Although not required by the USGS procedure or project plan, USGS personnel revealed that definitions and terms were standardized throughout the elicitation process and documented in the Ground Motion and Seismic Source activity reports. This was accomplished through consensus with the subject-matter experts throughout the elicitation process. Assumptions made by the experts will be included as part of the expert=s elicitation summary. A recommendation is made that USGS provide documentation regarding these issues in the PSHA final report.

 

5. Documentation of expert team decisions as shown on logic trees and described in the draft Seismic Source and Fault Displacement Characterization Project Activity Report, Appendix E, Expert Elicitation Summaries, dated July 15, 1997, appears, in some cases, to require additional detail. For most of the logic tree nodes, the technical basis for each decision is explicit. However, the technical basis is absent or needs to be more explicit at other nodes. We recommend that the management team review the revised draft to be certain that this documentation is complete in the final report.

 

6. In order to ensure that project files are complete, the management team should contact the expert panelists to be certain that any references, data, interpretations not provided by the Project but generated or used by the experts to support decisions, conclusions or interpretations be provided to the management team. Although it was understood that all information used by an expert team was made available to all other expert teams, it should also be understood that this information should be submitted as a project record to support decisions made and documented in a final activity report.

 

7. It is recommended that USGS implement more rigorous controls to ensure the security of the records generated as a result of the PSHA activity. The current system provides only minimum controls for these in-process records in that there is only one location maintaining the records. Based on discussions with personnel responsible for records security, the minimum requirement as described in AP-17.1Q,

Section 5.3e is implemented. This requirement states AKeep records and in-process records in a secured area when not in use (e. g., a locked desk drawer or office).@ Because these records represent a considerable investment, on the part of the Project, and will be retained for a considerable length of time before being submitted to the Records Processing Center, it is suggested that either a 1-hour fire proof storage facility be provided to store the records or dual storage of all records be maintained.

 

7.0 List of Attachments

 

Attachment 1 Personnel Contacted During the Audit

 

Attachment 2 Summary Table of Audit Results

 

 

 

 

 

 

ATTACHMENT 1

 

Personnel Contacted During the Audit

 

 

 

Name

 

 

Organization/Title

 

Pre-audit Meeting

 

Contacted During Audit

 

Post-audit Meeting

 

Tom Chaney

 

USGS/EA Chief

 

X

 

X

 

X

 

Robert Craig

 

USGS/TPO

 

X

 

X

 

X

 

Joyce Golos

 

USGS/Admin Officer

 

X

 

 

 

 

 

Martha Mustard

 

USGS/Hydrologist

 

X

 

X

 

X

 

Bruce Parks

 

USGS/Team Chief, Seismotectronics

 

X

 

X

 

X

 

Richard Quittmeyer

 

M&O/WCFS/SPO Staff

 

X

 

X

 

X

 

Patricia Sheaffer

 

USGS/QA Implemnetation Specialist

 

X

 

X

 

X

 

Donna Sinks

 

OQA/QA Specialist

 

X

 

 

 

 

 

Carl Stepp

 

M&O/WCFS/PSHA Project Director

 

X

 

X

 

 

 

Ardell Whiteside

 

OQA/QA Specialist

 

 

 

 

 

X

 

John Whitney

 

USGS/Tetonics Chief

 

X

 

X

 

X

 

Ivan Wong

 

M&O/WCFS/PSHA Deputy Project Director

 

X

 

X

 

 

 

Legend:

EA Engineering Assurance

TPO Technical Project Officer

WCFS Woodard Clyde Federal Services

OQA Office of Quality Assurance

NRC Nuclear Regulatory Commission

SPO Site Program Operations

PSHA Probabilistic Seismic Hazards Analysis

 

ATTACHMENT 2

 

AUDIT USGS-ARP-98-01 - DETAILED SUMMARY FOR EVALUATION OF EXPERT ELICITATION PROCESS

 

 

 

Activities

 

Process steps/MGMT Objectives

 

Details

(Checklist)

 

 

Deficiencies

 

 

REC

 

Process Eff.

 

Product

Adequacy

 

 

Overall

 

 

 

 

 

Scientific

Expert Elicitation for PSHA

 

 

Process control established for elicitation

 

pgs. 1-6

 

N

 

#1

 

SAT

 

SAT

 

 

 

 

 

 

 

 

 

SAT

 

Definition of Objectives

 

pg 7

 

N

 

 

 

SAT

 

SAT

 

Selection and Training of Experts

 

pgs 3, 8, 9, 11

 

N

 

 

 

SAT

 

SAT

 

Assemble and Disseminate Basic Information

 

pgs 5, 10, 20, 22, 25, 26

 

N

 

 

 

SAT

 

SAT

 

Management of Judgements (elicitation, feedback, and aggregation)

 

pgs 12-17, 20-43

 

N

 

#2, 3

#5, 6

#7

 

SAT

 

SAT

 

Documentation and Preparation of Report

 

 

pgs 6, 18,19

 

N

 

#4

 

SAT

 

SAT

 

 

SUMMARY TABLE OF AUDIT RESULTS

FOR PROCEDURAL COMPLIANCE EVALUATIONS

 

 

QA

Element

 

 

Document Reviewed

 

Details

(Checklist)

 

 

Deficiencies

 

 

REC

 

Process Eff.

 

Product

Adequacy

 

 

Overall

 

2.0

 

YMP-USGS-QMP-3.16, R0

 

Page 3

 

CDA #1

 

N

 

SAT

 

SAT

 

SAT

 

5.0

 

YMP-USGS-QMP-3.16, R0

 

Pages 1-6

 

N

 

N

 

SAT

 

SAT

 

SAT