U.S. DEPARTMENT OF ENERGY
OFFICE OF CIVILIAN RADIOACTIVE WASTE MANAGEMENT
OFFICE OF QUALITY ASSURANCE
AUDIT REPORT
OF
UNITED STATES GEOLOGICAL SURVEY
AT
DENVER, COLORADO
USGS-ARP-98-01
OCTOBER 28-30, 1997
Prepared by: _____________________________ Date: ___________________
Richard L. Weeks
Audit Team Leader
Office of Quality Assurance
Approved by:_____________________________ Date: ____________________
Donald G. Horton
Director
Office of Quality Assurance
1.0 EXECUTIVE SUMMARY
As a result of Performance Based Quality Assurance (QA) Audit USGS-ARP-98-01, the audit team determined that U.S. Geological Survey (USGS) is satisfactorily implementing the Office of Civilian Radioactive Waste Management (OCRWM) QA Program and process controls for the Scientific Expert Elicitation as applied to the evaluation of seismic hazards in support of the AProbabilistic Seismic Hazards Analysis@ (PSHA).
The activity examined during this audit is in accordance with applicable portions of the U. S. Department of Energy (DOE) OCRWM Quality Assurance Requirements and Description (QARD) document DOE/RW-0333P, Revision 7. The audit team considers that development of the hazards analysis has been both adequate and effective to meet the project=s objectives based on implementation of established procedure controls and plans, adequacy of the process with respect to NUREG-1563 (Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program), and evaluation of available technical data. This evaluation was based on the progress made to date towards the final report scheduled for February 1998.
The audit team identified one deficiency corrected during the audit which is described in
Section 5.6.1. Additionally, 7 recommendations resulted from the assessment of the process adequacy and program documentation which are detailed in Section 6.0 of this report.
2.0 SCOPE
The audit was conducted to evaluate the adequacy and effectiveness of USGS controls for the implementation of the Scientific Expert Elicitation process in the development
of the deliverable for the PSHA. The audit was intended to determine the degree to which the resultant products meet the program requirements and management commitments and expectations, as well as to determine that USGS is implementing the work in accordance with pertinent sections of the QARD. In addition, an evaluation of the process used by USGS was compared to NUREG-1563 to measure the implementation of the NUREG-1563 guidance for the expert elicitation.
The process/activities/end products were evaluated during the audit, in accordance with the approved Audit Plan.
PROCESS/ACTIVITY/END PRODUCT
This audit evaluated the in-process documents generated to support the development of the PSHA for Work Breakdown Structure 1.2.3.2.8.3.6. These documents included level IV deliverables associated with the elicitation process workshops.
The performance-based evaluation of process effectiveness and product acceptability was based upon:
1. Satisfactory implementation of the critical process steps;
2. Use of trained and qualified personnel working effectively;
3. Documentation that substantiates the quality of the product;
4. Acceptable results and adequate end-product; and
5. Effectiveness of corrective action.
Technical Areas
The USGS processes for Scientific Expert Elicitation and its associated end products for the PSHA were evaluated for the following critical process steps:
The audit included a technical evaluation of process effectiveness and/or product acceptability. Details of the technical evaluation are included in Section 5.5.
3.0 AUDIT TEAM AND OBSERVERS
Team Member Responsible Area
Richard L. Weeks, Audit Team Leader Critical Process Steps and Management
Objectives
William J. Glasser, Auditor Critical Process Steps and Process Implementation
Robert P. Hasson, Auditor Critical Process Steps and Process Adequacy
David F. Fenster, Technical Specialist Technical Evaluation of Product
A. K. Ibrahim, NRC Observer
4.0 AUDIT TEAM MEETINGS AND PERSONNEL CONTACTED
The pre-audit meeting was held on October 28, 1997, at the USGS office in Denver, Colorado. Daily debriefing and coordination meetings were held with USGS management and staff, and daily audit team meetings were held to discuss audit status. The audit was concluded with a post-audit meeting held on October 30, 1997, at the USGS office in Denver, Colorado. Personnel contacted during the audit are listed in Attachment 1. The list includes those who attended the pre-audit and post-audit meetings.
5.0 SUMMARY OF RESULTS
5.1 Program Effectiveness
The audit team concluded that, overall, the USGS process controls are effectively being implemented for the areas identified in the scope of this audit.
5.2 Stop Work or Immediate Corrective Action Taken
There were no Stop Work Orders, immediate corrective actions or related additional items resulting from this audit.
5.3 QA Program Implementation of the Elicitation Procedure
The audit assessed the implementation of the USGS Procedure YMP-USGS-QMP-3.16 R0, AScientific Expert Elicitation.@ Requirements for selection and training of personnel, preparation and implementation of a project plan, and implementation of procedure requirements were evaluated. Implementation was evaluated for compliance to QA Program elements:
2.0 Quality Assurance Program
5.0 Implementing Documents
For all audit activities, the audit checklists contain the details of the audit evaluation along with identification of the objective evidence reviewed. The checklists are kept and maintained as QA records.
5.4 Adequacy of the Elicitation Process
The audit team evaluated the process used by USGS to conduct expert elicitation as compared to NUREG-1563. USGS has an established procedure, QMP 3.16, and Project Plan, YMP-USGS-EE-9701, R/0, AProbabilistic Analysis of Fault Displacement and Vibratory Ground Motion and the Development of Seismic Design Bases for Yucca Mountain,@ to perform the PSHA expert elicitation. The procedural controls established were evaluated for adequacy to the NUREG-1563 guidance. Overall, the process controls established to perform the expert elicitation were considered to be adequate. The procedure and project plan controls in place addressed most of the NUREG-1563 guidance. There were some NUREG-1563 criteria not addressed in the documents for areas such as conflict of interest, conducting the elicitation in a private setting conducive to uninterrupted discussion, documenting the rationale for revisions to elicited judgements, and identifying assumptions and unambiguous terms for the subject-matter experts. Although not addressed in the procedural controls, the NUREG-1563 guidance has been considered by USGS in the performance of the expert elicitation. In discussion with USGS personnel, USGS=s intention is not to revise the procedure and project plan since this elicitation is a one-time project. USGS=s plan is to cancel the procedure once the elicitation project is completed. USGS agreed to address these concerns in the PSHA final report explaining what was implemented and what was not and why. Four recommendations (#1-4) were made to address these concerns by the audit team.
The audit team also performed a review of USGS
=s response to the Office of Quality Assurance=s (OQA) comments on USGS Procedure QMP 3.16. OQA issued a letter to USGS, (reference letter D. G. Horton to T. H. Chaney, dated August 8, 1997) stating that the review of USGS response to OQA comments on the Scientific Expert Elicitation procedure have raised concerns that the procedure did not meet the intent of NUREG-1563 and that OQA would follow up with a surveillance to evaluate the procedure to NUREG-1563. Based on the review of the procedure, the project plan, and discussion with USGS=s personnel during the audit, resolution of the questionable responses were agreed upon and will be revised by USGS and resubmitted to OQA for acceptance.
5.5 Technical Evaluation of the Elicitation Product
The technical audit checklist questions were based on technical evaluation that are a necessary part of conducting a probabilistic seismic hazard analysis. The checklist questions and supplementary questions were designed to evaluate how the expert elicitation process evaluated and documented conclusions related to: evaluation of seismic source zones; evaluation of earthquake recurrence and maximum magnitude; evaluation of ground motion/attenuation functions; and, probabilistic seismic hazard calculation for vibratory ground motion.
USGS audit participants were well-prepared and well-organized. All responses to checklist and supplementary questions were satisfactory. Based on these responses and on an examination of the objective evidence, the technical work performed was adequate to meet project objectives. We recognize that the PSHA Expert Elicitation has not been completed and that both the fault displacement evaluation and completion of the activity reports are ongoing efforts.
As a result of the above technical evaluation, two recommendation (#5 and #6) were made.
In conjunction with the technical portion of the audit, an evaluation of the implementation of requirements described in the Project Plan, YMP-USGS-EE-9701,
AProbabilistic Analysis of Fault Displacement and Vibratory Ground Motion and the Development of Seismic Design Bases for Yucca Mountain,@ was performed. It was concluded that the requirements described in the Project Plan were effectively implemented. One recommendation (#7) regarding the security of records is being provided.
5.6 Summary of Deficiencies
5.6.1 Deficiency Corrected During the Audit
During review of the implementation of USGS Procedure YMP-USGS-QMP-3.16, R0, the audit team determined that two USGS staff personnel responsible for the management and implementation of the process had not documented their training to the subject procedure. All other personnel involved in this process had been trained (classroom training) and the training documented on an appropriate training record. Both individuals in question had been in attendance at the classroom training as subject matter experts but had not signed the attendance list. Since both individuals had been trained but the training not documented, the required USGS training record form was completed and reflects both the training date and the date the form was filled out as allowed by the USGS procedure. This deficiency is an isolated case and was satisfactorily corrected during the audit.
5.6.2 Follow up to Deficiency Report YM-97-D-039
During the annual compliance audit of USGS (USGS-ARC-97-15 conducted in April 1997), the subject deficiency was written to identify the untimely approval of the PSHA elicitation process Project Plan and the lack of attestation of qualifications for many of the participants in the process. The audit team determined that the approved project plan has been and is being followed as documented, even prior to the approval date. Training records were complete for all personnel involved in the PSHA process. A memorandum which was required by the Actions to Preclude Recurrence required a minor revision which was completed during the audit. A verification statement will be added to the subject deficiency with a recommendation for closure.
6.0 Recommendations
1. NUREG-1563 discusses the need for subject-matter experts to be willing to publicly disclose all potential conflicts of interest. Although not addressed in the USGS procedure or plan, USGS did provide documentation that was completed by the subject-matter experts regarding conflict of interest statements. A recommendation is made to document USGS implementation of
Aconflict of interest@ in the PSHA final report.
2. NUREG-1563 states that the generalists and normative expert be in attendance for the complete elicitation session with each subject-matter expert in a private setting conducive to uninterrupted discussion. A requirement was not established in the procedure to address this guidance. It is recommended that USGS provide a statement in the PSHA final report as to who was in attendance fulfilling the generalists and normative function and where the elicitation took place that indicate implementation of this guidance.
3. NUREG-1563 guidance states that for all elicited judgements, the rationale for any revisions are required to be well documented. In discussion with USGS personnel, they do not feel this guidance benefits the experts, but actually
Aanchors@ the experts and may possibly cause Abias@ among the experts. A recommendation is made that USGS document their approach regarding this issue in the PSHA final report.
4. NUREG-1563 guidance states that unambiguous definitions and specific terms be provided and any assumptions used in the elicitation be explicitly stated. Although not required by the USGS procedure or project plan, USGS personnel revealed that definitions and terms were standardized throughout the elicitation process and documented in the Ground Motion and Seismic Source activity reports. This was accomplished through consensus with the subject-matter experts throughout the elicitation process. Assumptions made by the experts will be included as part of the expert
=s elicitation summary. A recommendation is made that USGS provide documentation regarding these issues in the PSHA final report.
5. Documentation of expert team decisions as shown on logic trees and described in the draft Seismic Source and Fault Displacement Characterization Project Activity Report, Appendix E, Expert Elicitation Summaries, dated July 15, 1997, appears, in some cases, to require additional detail. For most of the logic tree nodes, the technical basis for each decision is explicit. However, the technical basis is absent or needs to be more explicit at other nodes. We recommend that the management team review the revised draft to be certain that this documentation is complete in the final report.
6. In order to ensure that project files are complete, the management team should contact the expert panelists to be certain that any references, data, interpretations not provided by the Project but generated or used by the experts to support decisions, conclusions or interpretations be provided to the management team. Although it was understood that all information used by an expert team was made available to all other expert teams, it should also be understood that this information should be submitted as a project record to support decisions made and documented in a final activity report.
7.
It is recommended that USGS implement more rigorous controls to ensure the security of the records generated as a result of the PSHA activity. The current system provides only minimum controls for these in-process records in that there is only one location maintaining the records. Based on discussions with personnel responsible for records security, the minimum requirement as described in AP-17.1Q,Section 5.3e is implemented. This requirement states
AKeep records and in-process records in a secured area when not in use (e. g., a locked desk drawer or office).@ Because these records represent a considerable investment, on the part of the Project, and will be retained for a considerable length of time before being submitted to the Records Processing Center, it is suggested that either a 1-hour fire proof storage facility be provided to store the records or dual storage of all records be maintained.
7.0 List of Attachments
Attachment 1 Personnel Contacted During the Audit
Attachment 2 Summary Table of Audit Results
ATTACHMENT 1
Personnel Contacted During the Audit
Name |
Organization/Title |
Pre-audit Meeting |
Contacted During Audit |
Post-audit Meeting |
Tom Chaney |
USGS/EA Chief |
X |
X |
X |
Robert Craig |
USGS/TPO |
X |
X |
X |
Joyce Golos |
USGS/Admin Officer |
X |
|
|
Martha Mustard |
USGS/Hydrologist |
X |
X |
X |
Bruce Parks |
USGS/Team Chief, Seismotectronics |
X |
X |
X |
Richard Quittmeyer |
M&O/WCFS/SPO Staff |
X |
X |
X |
Patricia Sheaffer |
USGS/QA Implemnetation Specialist |
X |
X |
X |
Donna Sinks |
OQA/QA Specialist |
X |
|
|
Carl Stepp |
M&O/WCFS/PSHA Project Director |
X |
X |
|
Ardell Whiteside |
OQA/QA Specialist |
|
|
X |
John Whitney |
USGS/Tetonics Chief |
X |
X |
X |
Ivan Wong |
M&O/WCFS/PSHA Deputy Project Director |
X |
X |
|
Legend:
EA Engineering Assurance
TPO Technical Project Officer
WCFS Woodard Clyde Federal Services
OQA Office of Quality Assurance
NRC Nuclear Regulatory Commission
SPO Site Program Operations
PSHA Probabilistic Seismic Hazards Analysis
ATTACHMENT 2
AUDIT USGS-ARP-98-01 - DETAILED SUMMARY FOR EVALUATION OF EXPERT ELICITATION PROCESS
Activities |
Process steps/MGMT Objectives |
Details (Checklist) |
Deficiencies |
REC |
Process Eff. |
Product Adequacy |
Overall |
Scientific Expert Elicitation for PSHA
|
Process control established for elicitation |
pgs. 1-6 |
N |
#1 |
SAT |
SAT |
SAT |
Definition of Objectives |
pg 7 |
N |
|
SAT |
SAT |
||
Selection and Training of Experts |
pgs 3, 8, 9, 11 |
N |
|
SAT |
SAT |
||
Assemble and Disseminate Basic Information |
pgs 5, 10, 20, 22, 25, 26 |
N |
|
SAT |
SAT |
||
Management of Judgements (elicitation, feedback, and aggregation) |
pgs 12-17, 20-43 |
N |
#2, 3 #5, 6 #7 |
SAT |
SAT |
||
Documentation and Preparation of Report
|
pgs 6, 18,19 |
N |
#4 |
SAT |
SAT |
SUMMARY TABLE OF AUDIT RESULTS
FOR PROCEDURAL COMPLIANCE EVALUATIONS
QA Element |
Document Reviewed |
Details (Checklist) |
Deficiencies |
REC |
Process Eff. |
Product Adequacy |
Overall |
2.0 |
YMP-USGS-QMP-3.16, R0 |
Page 3 |
CDA #1 |
N |
SAT |
SAT |
SAT |
5.0 |
YMP-USGS-QMP-3.16, R0 |
Pages 1-6 |
N |
N |
SAT |
SAT |
SAT |