The Office of Independent Oversight and Performance Assurance (OA) has published the Appraisal Process Protocols to describe the philosophy, scope, and general procedures applicable to all independent oversight appraisal activities. The Office of Safeguards and Security Evaluations (OA-10) prepared this companion volume, the Safeguards and Security Appraisal Process Guide, as part of a continuing effort to enhance the quality and consistency of safeguards and security inspections. When used in conjunction with the OA Appraisal Process Protocols, this Safeguards and Security Appraisal Process Guide provides necessary guidance for conducting safeguards and security inspections; it also offers techniques, formats, and sample documents useful in planning for, conducting, and reporting the results of safeguards and security inspections.

The two process documents, along with OA-10’s topic-specific inspectors guides, provide a comprehensive set of guidance and tools that better enable OA-10 inspectors to evaluate safeguards and security program effectiveness across the Department of Energy (DOE) complex.

Although the process guide is primarily germane to OA-10, it is made available to the field through the DOE’s home page to assist in the conduct of field surveys or self-assessments. A loose-leaf format was selected so that inspectors can remove and copy sections for ready reference.

Definitions

Section 1. Introduction

Mission
About This Guide
Scope of Inspections

Section 2. Approach

Introduction
Inspection Goals and Philosophy
Staff Inspection Roles
Major Inspection Phases
Compliance Versus Performance
Local Representatives
Inspection Standards
Inspector Proficiency
Reporting

Section 3. Planning

Introduction
Goals and Objectives
Preplanning Activities
Team Planning Activities
Topic Team Planning Tasks
Post-Planning Meeting Activities
Continuing Planning Activities

Section 4. Conduct

Introduction
Goals
Scope of the Onsite Inspection
Protection of Classified Information
Relations with Site Headquarters Personnel
Data Collection
Integration
Validation

Section 5. Closure

Introduction
Goals
Data Review
Integration
Analysis of Results
Determining Findings
Ratings
Focus Briefing
Report Preparation
Additional Team Responsibilities
Process Improvement
Outbriefing

Section 6. Follow-up

Introduction
Goals
Headquarters Briefings
Policy Issue Papers
Final Report
Corrective Action Plan Review
Corrective Action Tracking and Follow-up

Inspection Program Terms

(These definitions supplement those found in the OA Appraisal Process Protocols.)

Accepted Risk: The acknowledgement that a protection system may not achieve 100 percent protection against all occurrences, but further improvement in the system is not justified.

Access Authorization or Security Clearance: An administrative determination that an individual is eligible for access to Restricted Data, other classified information, or special nuclear material. The individual may be a DOE employee or an applicant for DOE employment, a consultant, an assignee, another Federal department or agency employee (or other persons designated by the Secretary of Energy), or a DOE contractor or subcontractor employee. Clearances granted by DOE are designated as "Q," "L," "Top Secret," or "Secret."

Access Control: The process of limiting access to the resources of a system only to authorized users, programs, processes, or other systems.

Access Control Measures: Hardware and software features, operating procedures, management procedures, and various combinations of these designed to detect and prevent unauthorized access and to permit authorized access to a system.

Accountability (nuclear material): That part of safeguards which encompasses the measurement systems and records and reports to account for special nuclear material (SNM).

Classified Document: Any document containing classified information; any document containing information the disclosure of which could damage the national security of the U.S. or its allies.

Classified Information: Any information that requires protection against unauthorized disclosure in the interests of the national defense and security or foreign relations of the United States pursuant to U.S. statute or Executive Order. The term includes Restricted Data, Formerly Restricted Data, and National Security Information, each of which has degrees of importance denoted by the classifications Top Secret, Secret, and Confidential.

Classified Interest: Classified documents, information, or material including classified SNM possessed by the Department, a contractor of the Department, a Departmental facility, or any other facility under the Department's jurisdiction.

Classified Materials: Chemical substances, including metals, fabricated or processed items, or machinery and equipment that have been classified by proper authority.

Classified Matter: Classified information, documents, parts, components, or other material.

Competency: The ability to perform a task, including whatever knowledge, skills, and attitudes are needed.

Compromise: Acquisition of classified information by persons not authorized to receive such information.

Consequences: The loss caused as a result of a successful attack on a target. Consequences may include damage to a national program, adverse publicity, etc., as well as direct monetary loss. Non-monetary loss considerations complicate the calculation of risk and introduce an element of subjectivity into risk assessment.

Critical Assets: Those physical and information assets required for the performance of the site mission.

Custodian (nuclear material): Any person having assigned responsibility for the control and accountability of classified matter.

Cyber Security: The protection resulting from all measures designed to prevent deliberate or inadvertent unauthorized disclosure, acquisition, manipulation, modification, or loss of information contained in a computer system, as well as measures designed to prevent denial of authorized use of the system.

Damage Assessment: An estimate of the damage to national security in the event that classified information is compromised or potentially compromised. This estimate is used to determine the potential value of the compromised information to foreign governments and/or hostile organizations.

Design Basis Threat Policy: A policy statement that describes threats that are postulated for the purpose of establishing requirements for safeguards and security significant programs, systems, components, equipment, information, or material.

Detection: The positive assessment that a specific object is the cause of an alarm.

Diversion: The transfer of nuclear material from its authorized use and/or location.

Document: Any record of information regardless of physical form or characteristics, including, but not limited to, the following: (1) all handwritten, printed, or typed matter; (2) all painted, drawn, or engraved matter; (3) all sound, magnetic, electromechanical, or optical recordings; (4) all photographic prints, exposed or developed film, and still or motion pictures; (5) automatic data processing input, memory, program, or output information or records such as punch cards, tapes, memory drums or disks, or visual displays; and (6) all reproductions of the foregoing by any process.

Draft Inspection Reports: Reports, not yet finalized, that contain inspection observations, issues, analyses, and ratings. Draft inspection reports are provided to managers as appropriate to allow timely discharge of their respective duties and responsibilities.

Exercise: A scheduled and planned event that tests the integrated capability and a major portion of the elements of a protection program as specified in that program’s security plans and procedures.

Implementation Plan: A concise description of the approach, resources, and time period planned for implementing orders that require such plans on a sitewide basis. The plan includes a description of the execution of environmental protection, safety, and health responsibilities and authorities by a field organization, and any proposed generic exemptions to parts of such DOE orders.

Level of Protection: The degree of safeguards and security provided to protect Departmental interests.

Line Management: The unbroken linkage of management personnel responsible for an organization’s direction, operations, performance, and effectiveness. In DOE, it is the chain of command that extends from the Secretary to representatives of the Cognizant Secretarial Office (CSO), who set program policy and plans and develop assigned programs; to the field organization managers, who are responsible to the CSO for execution of these programs; and to the contractors and subcontractors who conduct the programs. Line management consists of DOE and contractor personnel organizationally or contractually responsible for work or job tasks, as well as effective security.

National Security: The national defense and foreign relations of the United States.

Operations Security (OPSEC): A program designed to disrupt or defeat the ability of foreign intelligence or other adversaries to exploit sensitive Departmental activities or information and to prevent the unauthorized disclosure of such information.

Personnel Security: The procedures established to ensure that all personnel who have access to any classified information have the required authorities, as well as the appropriate clearance.

Physical Security: (1) The use of locks, guards, badges, alarms, and similar measures to control access; (2) The measures required for the protection of structures housing the system from espionage, theft, or damage by accident, fire, and environmental hazards.

Protection Program: The total program that includes all aspects of the DOE’s activities directed toward protection of national security interests and DOE property. Any adverse impacts on the health and safety of the public resulting from implementation or failure of elements of the protection program are also included.

Protection Program Topic Areas: Subject areas that are used to logically address the many elements of a protection program. From time to time modifications of the scope of certain topic areas may be necessary to accommodate site-specific concerns and programs as well as changes in threats, protection concepts, and technologies.

Risk: The chance that a specific attack against a system vulnerability will lead to loss of an asset.

Risk Analysis (Risk Assessment): An analysis of system assets and vulnerabilities to establish expected loss from certain events based on the estimated probabilities of occurrence of those events.

Sensitive Information: Information that, as determined by competent authority, must be protected because its unauthorized disclosure, alteration, loss, or destruction will cause perceptible damage to some person, organization, or mission.

Sensitive/Unclassified Information: Unclassified but sensitive data requiring protection because of the risk and magnitude of loss or harm that could result from inadvertent or deliberate disclosure.

Technical Security: Includes technical surveillance countermeasures, communications security, and the prevention or suppression of compromising emissions and emanations.

Technical Surveillance: The covert installation of devices or equipment to visually or audibly monitor activities within a target area to acquire information by technical means.

Technical Surveillance Countermeasures: Systematic and effective measures for the detection and nullification of technical surveillance penetrations, technical surveillance hazards, and physical security weaknesses.

Tempest: Short name referring to investigation, study, and control of compromising emanations from telecommunications and automated information systems equipment.

Theft: The removal of government property and/or materials from a DOE or DOE contractor-operator facility without permission or authorization and contrary to law, or the unauthorized removal of SNM.

Threat: A possible event that can, if it occurs, exploit a vulnerability. Threats include both hazards and the triggering of flaws.

Threat Analysis: An analysis of the probability of occurrence and consequences of damaging events.

Threat Statement: A statement of actions and events that can adversely affect the security of a system.

Vulnerability: A weakness in procedures, administrative controls, internal controls, etc., that could be exploited to gain unauthorized access to resources.

The Office of Safeguards and Security Evaluations (OA-10) conducts all independent appraisals of Department of Energy (DOE) safeguards and security programs. These appraisals include inspections, assessments, and special studies that evaluate the effectiveness and implementation of DOE safeguards and security policies and programs across the DOE complex. OA-10 reports the results of these activities to the Secretary, Deputy Secretary, Under Secretary, senior DOE Headquarters and field managers, and others, as appropriate. The results are also used as input to the Annual Report to the President and the Annual Report to the Secretary regarding the status of safeguards and security in the Department.

OA-10 directs its activities toward independently evaluating DOE safeguards and security programs, while line management responsibility for safeguards and security programs and policy is exercised through the program secretarial officers and the Office of Security and Emergency Operations, respectively. OA-10 provides the Department with an independent assessment of the status of safeguards and security programs to complement the views and assessments of the programmatic and policy managers. Further, OA-10 analyzes Department-wide indicators and trends from a broad, Departmental viewpoint, thus providing a unique perspective on safeguards and security concerns. Results are reported to appropriate Headquarters and field element managers.

Appraisals are designed to determine the adequacy of safeguards and security policies and programs, the adequacy of policy and program implementation, and their effectiveness in protecting DOE’s national security interests. OA-10 currently conducts various types of appraisals:

• Comprehensive inspections determine the adequacy of protection programs by examining a broad range of safeguards and security topic areas at a specific location. They are comprehensive in their technical span and in their consideration of all national security interests at the facilities inspected.

• Special inspections include reinspections, unrated reviews, major performance tests, or other inspection activities required on a one-of-a-kind basis. OA-10 conducts special inspections in much the same manner as comprehensive inspections; however, procedures are modified as required. Special inspections (with the exception of major performance tests) are usually more limited in scope than comprehensive inspections, requiring fewer resources and less time spent at the field site. Major performance tests are significant efforts, and are conducted in accordance with the Context and Protocols for Performance Testing of Protective Forces.

• Follow-up Reviews are conducted to determine the status and progress of corrective actions and other activities in response to deficiencies previously identified by OA-10 appraisals. Ratings may be assigned as a result of follow-up reviews.

• Assessments address the effectiveness of protection program elements as implemented across the DOE by analyzing complex-wide protection issues and providing recommendations for improvement. Assessments also may provide an analysis of a specific item of policy as implemented across the DOE complex, rather than an analysis of the broader issues of protection program elements. Conclusions and recommendations (if appropriate) are published, but ratings are not normally assigned.

• Special studies are performed as required to address an area, a concern, or an issue within the safeguards and security program. They also may address areas outside safeguards and security that affect the safeguards and security program. Special studies may focus on the status of a specific program element, the adequacy of selected safeguards and security policies, or the status of specific policy implementation throughout DOE. Special studies contain conclusions and recommendations, but ratings are not normally assigned.

• Special reviews are the responsibility of the Office of Cyber Security and Special Reviews (OA-20); however, OA-10 provides personnel and other resources when necessary to assist in special reviews. Special reviews are conducted at the request of the Secretary or other senior DOE managers who require, on an expedited basis, information regarding the status of a particular program, program element, issue, or Departmental function.

This Safeguards and Security Appraisal Process Guide is a companion publication to the Office of Independent Oversight and Performance Assurance (OA) Appraisal Process Protocols. While the OA Appraisal Process Protocols provide general guidance common to all OA appraisal activities, this OA-10 guide provides additional detail and guidance regarding procedures and methods specific to safeguards and security appraisals conducted by OA-10. Since the intent is to maintain both process documents in a single binder, every effort has been made to avoid unnecessary duplication among the two guides. For that reason, text in this guide sometimes refers to sections or appendices in the OA Appraisal Process Protocols. OA-10 inspectors should maintain familiarity with information in both documents.

OA-10 inspections examine the effectiveness of safeguards and security protection programs in various topical areas, including but not limited to:

• Protection program management
• Personnel security
• Classified matter protection and control
• Physical security systems
• Protective force
• Material control and accountability.

Figure 1 shows these traditional topics and the subtopics and programs included in each.

In addition to examining and rating the status of these topical areas, and providing potential enhancements when appropriate, OA-10 integrates the inspection results from the areas inspected and assigns an overall rating for safeguards and security program effectiveness.

OA-10 achieves the consistency and discipline required to conduct a meaningful and valid inspection program by adhering to and applying established and accepted standards during all phases and aspects of the independent oversight process. The foundations of OA-10’s approach to inspections have been developed over time, through experience, and are frequently reviewed and refined. This section addresses some fundamental aspects of OA-10’s inspection approach and discusses major responsibilities of key participants in the process.

The major goals of the OA oversight process and the underlying philosophy that guides OA’s efforts to achieve those goals are stated in Section 2 of the OA Appraisal Process Protocols. OA-10 accepts and adopts those goals and that philosophy, in their entirety, and applies them to the safeguards and security oversight process.

The Director of Independent Oversight and Performance Assurance provides direction and guidance to the Director, Office of Safeguards and Security Evaluations, relative to scheduling and the overall conduct of inspections and other appraisal activities. Guidance provided includes any senior DOE management concerns, issues that could affect inspections, and matters of concern to external agencies and Congress.

The Director, Office of Safeguards and Security Evaluations, directs and oversees the inspection process. The Director provides all pertinent information to the designated Inspection Chief, assuring that the Inspection Chief tasks the appropriate personnel to serve on the various inspection topic teams. The Director reviews ongoing plans to ensure that they address all identified concerns. During the inspection, the Director monitors progress, provides guidance, participates on the Quality Review Board, and meets with site/facility management as appropriate. After the inspection, the Director may provide briefings to the Secretary, Under Secretary, Deputy Secretary, Lead Program Secretarial Officers, Congressional committees, or other groups who have a legitimate interest in inspection results.

The Director may also choose to assume the role of Inspection Chief; this more commonly occurs during major efforts, such as comprehensive inspections.

The Inspection Chief is responsible for all aspects of an inspection or follow-up review.

The Inspection Chief: 1) manages all phases of the inspection; 2) provides continuity throughout the inspection process; 3) ensures that inspection activities remain properly focused; 4) keeps OA and OA-10 management informed of progress and significant details; and 5) ensures that information is disseminated to appropriate inspection team members. In short, the Inspection Chief is responsible for keeping the inspection process on track.

The Inspection Chief assigns responsibilities for preplanning tasks and assures that all such tasks are accomplished. After reviewing results of the preplanning effort and conferring with OA-10 management, the Inspection Chief proposes a scope and focus for the inspection.

The Inspection Chief is responsible for the conduct of the planning meeting and for ensuring the completion of an Inspection Plan. The Inspection Chief monitors the activities of the topic leads, ensuring that they review appropriate information. The Inspection Chief provides direction to the topic teams to ensure that the inspection focus is understood and that the proposed scope of the inspection is feasible. The Inspection Chief meets frequently with the topic leads during the course of the planning and data collection efforts.

During onsite inspection activities, the Inspection Chief maintains the focus of the inspection, ensures that OA-10 management initiatives are accomplished, establishes priorities, resolves conflicts, communicates information to site management as appropriate, keeps OA-10 management informed, and oversees the activities of the inspection team. After the onsite inspection, the Inspection Chief ensures that the inspection report is finalized, Headquarters briefings are scheduled and conducted, corrective action plans are reviewed, and the After-Action Report and policy issues are completed.

The Deputy Inspection Chief provides support to the Inspection Chief in all phases of the inspection. A primary responsibility of the Deputy Inspection Chief is to assume the duties of the Inspection Chief if the Inspection Chief is absent, thus ensuring continuity of the inspection process. The Deputy Inspection Chief also performs other activities at the direction of the Inspection Chief. These assignments vary from inspection to inspection depending on the Inspection Chief’s needs and the scope of the inspection.

Various logistical and administrative duties are typically assigned to the Deputy Inspection Chief, such as preparing resource lists, coordinating security clearance requirements, arranging hotel accommodations, preparing handouts for topic teams, and coordinating safety plans. Also, the Deputy Inspection Chief prepares correspondence relative to inspection notification, document requests, and distribution of the draft and final reports.

During the conduct phase of the inspection, the Deputy Inspection Chief may act as a topic team member or topic lead. At other times, the Deputy Inspection Chief assists the Inspection Chief in conducting the daily inspection team meetings, in preparing daily reports to management, and in ensuring that report preparation runs smoothly and on schedule. After inspection closure, the Deputy Inspection Chief prepares the After-Action Report.

In cases where inspection scope is limited or narrow, such as some special inspections or follow-up reviews, a Deputy Inspection Chief may not be required. In some inspections, the Deputy Inspection Chief will also perform the functions of the Administrative Support Coordinator.

The Administrative Support Coordinator is the point of contact for all inspection administrative and logistical support. This includes arrangements with the inspected site’s point of contact for office space, telephone service, classified and unclassified storage, reproduction and destruction, fax service, and communication between the site and DOE Headquarters. The Administrative Support Coordinator makes arrangements for computer support and acts as the Computer System Security Officer during the inspection. The Administrative Support Coordinator supervises the inspection team’s administrative staff on site, ensures the availability of necessary support, ensures control and accountability of classified documents, and oversees the preparation of the draft inspection report, final report, and memoranda.

A topic lead is assigned to each topic team participating in an inspection or a follow-up review. The topic lead is responsible for managing the efforts of the topic team and for keeping the Inspection Chief informed of salient topic team activities during the inspection. In particular, the topic lead is the focal point for the topic team and is responsible for coordinating and focusing the activities of the team, ensuring that deliverables are prepared and provided according to the schedule, promoting integration with other topic teams, and acting as spokesperson during meetings and briefings.

OA-10 safeguards and security inspection activities may be characterized by the functional phases into which they are organized: planning, conduct, closure, and follow-up.

The planning phase includes those activities necessary to prepare for all aspects of an inspection. The conduct phase includes the portion of the site visit principally devoted to collecting and validating data. The closure phase involves data integration and analysis, issue identification, rating determination, draft report preparation and quality review, and internal management briefings. The follow-up phase includes comment review and final report preparation, Headquarters briefings, corrective action plan reviews, and corrective action tracking.

Although these phases are identified by the primary activities they encompass, actual inspection activities may overlap significantly. For example, some data are collected during the planning phase, and planning (particularly for performance testing) can extend into the conduct phase. Similarly, analysis begins during data collection and continues throughout the process. Subsequent sections of this guide discuss each of these phases in greater detail.

Figure 3 illustrates the major inspection activities for comprehensive inspections.

DOE safeguards and security policy requires that certain functions be performed and that certain levels of protection be achieved. However, policy does not always specify how those functions or protection levels are to be achieved. Although compliance with DOE policies and procedures is mandatory, failure to comply with one or more specific provisions does not by itself indicate an unsatisfactory program if adequate protection is provided by other means. Conversely, mere compliance with policies and procedures may not produce an effective program—a program may be in compliance, but not actually performing well. OA-10’s inspection approach therefore considers both compliance and performance.

OA-10’s procedure is to include in the inspection report any issues regarding significant cases of non-compliance, while also setting forth mitigating circumstances and providing an analysis of whether program objectives have been met and maintained. Although OA-10 considers mitigating factors when assigning ratings, the facility must take corrective actions to achieve compliance whenever a DOE requirement is not being met.

Mitigating factors may exist for both compliance and performance issues. Deficiencies in program or system performance may be mitigated by the existence of alternative processes or controls, such as:

• Alternative documentation indicating that required functions were performed, factors considered, or decisions made

• Complementary procedures or features that function effectively

• Demonstration, through performance testing, that DOE assets are afforded a level of protection equivalent to that specified by DOE orders.

The cooperation and assistance of DOE field element and facility representatives is essential in order to conduct thorough, efficient, and fair inspections. Local representatives provide detailed site and system knowledge for planning; arrange administrative and logistical support; expedite inspection activities; and identify the local points of contact who participate during data gathering and validation.

Inspections determine the adequacy of safeguards and security programs by comparing capabilities and performance against established standards. The standards applied come from various sources at the national and local levels.

National standards are the basic requirements with which DOE protective programs must comply. They are established by Congress, the DOE, and other executive agencies. DOE policy is promulgated through DOE directives; other national standards are exemplified by applicable public laws, Executive Orders, and other directives.

OA-10 inspectors must be thoroughly familiar with DOE policies and procedures, technically competent and current in their assigned safeguards and security topic areas, and cognizant of OA-10 inspection philosophy and standards. Additionally, they must clearly demonstrate an ability to successfully perform all necessary functions associated with planning, conducting, closing, and following up on OA-10 inspections.

It is essential that individuals selected as OA-10 inspectors have the skills, knowledge, and abilities necessary to provide outstanding performance in the conduct of inspections. The significance of the Independent Oversight role, the protection of national security interests, public safety, and DOE program operations dictates the requirement to ensure the highest degree of proficiency for all Independent Oversight inspectors.

The end product of an inspection is the inspection report. The purpose of the report is to clearly present the results of the inspection, identifying and analyzing the impact of strengths and weaknesses. Although the report has several audiences, it is essentially a management-level portrayal of the status of safeguards and security at the inspected facility or field element. Report preparation is addressed in more detail in Section  5, Closure, and in Appendix A, Comprehensive Inspection Report Format.

The thoroughness and quality of inspection planning significantly affects all other inspection activities. Inspection planning involves gathering and analyzing large amounts of information from many sources, making decisions based on the analysis, and preparing inspection activities based on the decisions. Because there is only a limited amount of time available on site to collect the data necessary to characterize the status of the programs being inspected, planning must focus on determining what program elements to examine and how best to inspect those elements. It also includes identifying support requirements for all phases of the inspection.

This section discusses OA-10’s inspection goals and objectives, preplanning activities, the major planning activities of the topic teams, and the ongoing planning process that continues throughout the inspection. Figure 4 summarizes the major planning events.

The goal of inspection planning is to anticipate and prepare for every action necessary to conduct the highest quality inspection possible with the resources available. At the conclusion of the planning phase, topic teams should be familiar with:

• The character of the program, including size, composition, organization, and mission
• Site management oversight responsibilities
• How personnel responsible for protection programs are trained
• The physical environment in which the program operates
• The specific areas of emphasis for the inspection
• Specific data collection methods to be used and the performance tests to be conducted, if any
• All personnel, administrative, and logistical support requirements necessary for data collection and performance testing
• The documents necessary for conducting a well-planned and effective inspection
• Procedures for communicating with management to gain approval for planned inspection scope and activities
• DOE and facility points of contact for each topic.

Preplanning activities are conducted under the supervision of the Inspection Chief and are aimed at laying the basic groundwork necessary to allow the inspection team to conduct its planning tasks efficiently. The Deputy Inspection Chief, Administrative Support Coordinator, and one or more staff members may be involved in preplanning activities. Specific tasks accomplished during preplanning usually include:

When preplanning activities are complete, detailed inspection planning begins. Detailed planning involves the entire team, begins with the planning meeting, and extends into the conduct phase of the inspection.

This detailed planning activity is normally conducted in the field at the facility being inspected. Although most detailed planning occurs during this formal planning meeting, it is important to remember that planning and adjustments to planned activities continue throughout the inspection. This is especially true of performance test planning, since many details must be planned and the performance tests fine-tuned even as they are occurring.

The Inspection Chief conducts an orientation meeting for all inspection team members at the start of the planning meeting, and the inspected facility may be asked to provide briefings on various aspects of their safeguards and security program. The OA-10 Director (if not acting as Inspection Chief) may address the inspection team. The Inspection Chief holds a meeting with the topic leads each morning during the planning meeting to ascertain progress and to identify any problems or issues that need to be addressed.

The planning meeting is where major decisions are made, the course of the inspection is set, and most inspection details are worked out. Table 1 provides a typical schedule for a planning meeting. During this period the most intensive inspection planning activities take place.

Table 1. Typical Planning Meeting Daily Activities

Planning requirements are numerous and can be very complex. Planning typically involves cooperation between OA-10, the Office of Security and Emergency Operations, responsible DOE program office(s), the DOE field element, and facility contractor personnel.

The planning meeting is designed to allow inspection team personnel to meet, review available facility documentation, interview DOE field element and site personnel, identify areas to be targeted for inspection, and analyze important areas to determine how key interests can be inspected effectively. Planning meeting objectives include planning all data collection activities, developing a schedule covering the conduct and closure phases, and preparing a briefing for OA-10 management on planned topic-specific activities.

A number of specific topic-team planning activities are listed in Figure 5. Activities are neither strictly sequential nor independent of one another. Rather, they represent a series of interrelated efforts, and team members typically work on several tasks at once.

Figure 5. Planning Meeting Activity Checklist

During the planning meeting, the topic teams tour facilities, interview site personnel and review vulnerability assessments, Site Safeguards and Security Plans, and topic-related documents. They also identify site assets, specific threats, and the site mission, including site functions and processes. The topic teams analyze data and further define the focus of inspection activities; identify key personnel to be interviewed during the inspection; determine the types of performance tests to be conducted; and review critical elements and other key information provided by the preplanning team.

• The site. The topic team pays attention to pertinent factors such as size, facility layout, mission, special programs, program changes, or other factors that would affect the program being inspected.

• The program’s composition. Successful planning depends on the identification of all program areas at the site to be visited, and the status of each area. Normally, DOE field elements and facility contractors maintain separate organizations that are potential inspection candidates. Although some screening usually occurs during preplanning, topic teams must ensure that all program areas of each organization are considered. Teams may also recommend new or changed areas of emphasis based on their review.

• The program’s function. After identifying subtopics to be evaluated, the team begins to narrow the focus and concentrate on potential areas of emphasis. It becomes critical that the team understands as much as possible about how each area of the program supports each organization.

Special attention should be given to reviewing program-related issues contained in site security plans, previous inspection reports and policy issues, recent field element survey reports, and data contained in the Safeguards and Security Information Management System. Other documents that may contain pertinent information include OA-10 appraisal reports, Government Accounting Office and Inspector General reports, and changes or policy memoranda promulgated by the Office of Security and Emergency Operations.

Often, the large quantity of available documentation requires team members to divide the documents to screen them for pertinent information. All topic team members should review appropriate sections of all documents. While there are some important documents that all team members may need to review, the topic lead may assign each team member a portion of the remaining documents based upon anticipated data collection assignments. As individuals review their assigned documents, all important information should be shared with all team members.

Document reviews and discussions with other topic teams should provide answers to three important questions:

• What security interests need to be protected?

• What threats are to be protected against?

• How does the program element (topic) function to provide the necessary degree of protection?

The answers to these three questions must be understood before meaningful planning can continue.

Determining the facility’s security interests and the mission of the inspected program is usually straightforward. Documentation provides information regarding the amount, category, and location of classified interests, which can be used to prioritize the various security interests and focus the detailed planning.

Identifying and understanding threats to the inspected program helps establish parameters for a number of data collection activities. Understanding the security interests and threats permits identification of critical program element vulnerabilities and development of data collection activities to characterize the facility program’s effectiveness.

Before deciding how to inspect a particular topic, the topic team must understand the nature of that topic. Examples of information that help define the nature of a topic are location of activities selected for inspection and distances between activities; how activities are organized; how personnel assigned program responsibilities are trained, managed and supervised; and how required elements of any protection program are implemented.

Most of this information can be found in the documents available at the planning meeting. Additionally, facility representatives can usually answer specific questions to provide a clear understanding of their programs.

Questions, issues, and discrepancies often surface during document review; for example, documents provided by the facility may be out of date or inconsistent with one another. The topic team should identify any discrepancies or questions and discuss them with the points of contact at the earliest opportunity during the planning meeting. In many cases, the points of contact can answer questions and explain inconsistencies.

In some cases, team members may identify policy issues or other items that require management attention. For example, they may question the provisions of facility operating instructions or the interpretation of a DOE order as applied at the facility. Such issues should be brought to the attention of the Inspection Chief as early as possible because resolution may require the involvement of the Office of Security and Emergency Operations or Headquarters program offices.

Every effort should be made to resolve identified issues before the data collection visit. It is very important that inspection planning be based on accurate, up-to-date information.

Integration and coordination among topic teams is crucial to the overall inspection process. The results of individual topic team activities provide the information necessary to reach general conclusions regarding the ability of a facility to protect SNM and classified information, and to manage their safeguards and security responsibilities.

There are several major objectives of inter-team coordination. First, topic teams can coordinate efforts so that activities complement each other. For example, if valid conclusions are to be drawn regarding the protection of classified information at a specific facility location, all topic teams must collect data at the same location. It would be of little use to inspect physical security systems at one location, control of classified documents at a different location, and the protective force at yet another location.

A second objective is to ensure that topic teams benefit from the knowledge, experience, and efforts of other topic teams. Ideas from one topic team can often help another topic team focus its efforts more productively. For example, the physical security systems topic team may indicate that the implementation of physical security systems at a particular location appears to result in undue reliance on the protective force. If the protective force topic team finds that Security Police Officers are not performing required checks at the location in question, it would have a significant impact on the ability of the site to protect the affected security interest.

The third objective is to prevent topic teams from interfering with each other. Several topic teams may want to concentrate their activities at the same location. In such cases, coordination of data collection activities, particularly performance tests, avoids undue disruption of the inspected facility and streamlines data collection. For example, problems may arise if the material control and accountability topic team schedules a performance test involving an emergency inventory or transfer of material in the same building and at the same time the classified matter protection and control team plans to inventory classified matter. All topic teams should be aware of what all other topic teams are doing, where they are doing it, and how it will affect their own activities.

An additional integration aspect worthy of particular mention is coordination with the protection program management (PPM) topic team. The nature of the PPM topic mandates integration with all topic teams; management of the protection program does not function in a vacuum. Topic teams may capitalize on this interaction. Information developed by the PPM topic team may affect how the results of inspection activities in other topics are viewed. Similarly, results in other topic areas will have some bearing on how the adequacy of protection program management is viewed.

Once the information gained from document reviews, discussions with facility representatives, and coordination with other topic teams is reviewed, the topic team must decide where to focus its inspection efforts. Topic teams should consider every subtopic for inspection during the planning activity, even though the depth to which a subtopic is inspected may vary.

Since many topics are fairly large and complex, it is not feasible to comprehensively review every facet of the topic. Consequently, the scope of the topic is customized to the resources available and the need to maintain high standards of quality in all inspection activities. The underlying principle is that the quality of the inspection takes precedence over the quantity of areas reviewed.

If, during the planning meeting, serious issues or problems are identified in an area not scheduled for inspection, the topic team may recommend additional inspection activities, which might require adding personnel to the topic team, deleting another subtopic, or scaling back the scope of activities in other subtopics. The Inspection Chief should be consulted as soon as possible regarding the allocation of additional resources, reallocation of existing resources, or rescoping of the topic.

The topic team must clarify the broad focus of their inspection activities. Then, they should discuss issues arising from the data reviewed and begin to make decisions that will narrow the focus of their inspection activities before they begin planning individual data collection.

Some of these decisions may be made in response to the Inspection Chief’s guidance or coordination with other topic teams. However, decisions are generally left to the topic team itself and are based on site-specific conditions and inspection-specific objectives. In the absence of overriding conditions and objectives, topic teams should focus on the most important programmatic elements and devote less time to relatively unimportant areas or areas that have proven to be less problematic in recent inspections. It is important that the selection of elements for review be logical and defensible, and provide for a balanced approach that will accomplish the inspection objectives.

Data collection activities are the essential activities of the inspection process. The data collection methods and techniques that are chosen, and the skill with which they are applied, determine the quality and quantity of information available for evaluation.

Once the topic team has narrowed the inspection focus and finalized the subtopics to be inspected, they must select appropriate data collection methods. Data collection tools and their applications are detailed in the topic-specific inspectors guides. Methods are selected to yield the most accurate, realistic, and useful data for the particular application.

One preferred method of gathering data is to actually watch the responsible individuals perform their assigned tasks, and to review documents pertaining to the program being inspected. Unfortunately, this is not always convenient or possible. Often activities that the inspectors want to observe may not occur while the inspection is in progress. The allocated time, personnel (both site personnel and inspectors), facilities, and other resources may also limit opportunities to observe program functions. OA-10 also relies on performance testing to evaluate programs under controlled conditions.

In addition, data collection priorities must be established. The topic team should attempt to schedule data collection activities for the entire data collection period to avoid wasting collection time if events run smoothly. However, when events do not run smoothly, data collection priorities allow the topic team to delete less important activities and concentrate on gathering essential data required to determine program effectiveness.

Sample size and configuration are important planning points that must be determined for many data collection activities. Since inspectors usually cannot review every document or observe every activity related to their topic, they must examine a sample of the population to form conclusions about the entire population under review.

The sample tested must be large enough to provide a reasonable indication of the entire population under review. For example, examining 10 items from a population of 50,000 would hardly provide a useful result; conversely, examination of all 50,000 items would be clearly impossible, and unnecessary.

Similarly, the sample tested must be representative of the system involved, and the components of the sample must have qualifications or conditions in common. For example, a sample of Top Secret documents could only be representative of documents in a Top Secret document system, and could not be used to draw conclusions about a Secret or sensitive document system.

Planning for each data collection activity should include a determination of how many items will be examined, and how they will be selected.

Population Identification

Before selecting a sample for a particular data collection activity, the appropriate population must be identified. For example, if skills common to all members of the protective force are to be tested, the appropriate population would be the entire protective force. However, if skills required only of special response team members are to be tested, the appropriate population would be limited to special response team members.

Sample Size and Configuration

Sample size normally depends on the size and availability of the population, and the available time. In general, the larger the sample, the better; but sample size must often be limited. Inspectors should strive to test at least the minimum recommended sample size for the population. However, if that is impractical, the need to use a smaller sample should not deter data collection. The topic team must exercise judgment in this regard.

The sample size, once set, may be changed if conditions warrant. For example, if the initial sample yields inconclusive results, the topic team may choose to expand the sample to get a more accurate picture of the status of the subtopic.

The most important factor in sample con-figuration is that the sample be representative of the entire population. The best way to ensure that the sample is representative is to use statistically valid random selection techniques. When statistically valid random samples are not required or possible, samples are selected based on available time and resources, with every attempt to select the best representation of the total population.

Statistically Valid Samples

Statistically valid samples are essential only if the topic team intends to make a formal statistical inference from the results. Validity in such analyses results from strict adherence to specific random sample selection and testing procedures. Additional detailed information may be found in textbooks on statistics.

All data collection tasks should be assigned and scheduled during the planning meeting to ensure the effective use of the time available for onsite data collection. Scheduling provides the basis for logistical planning and is accomplished best during the latter phases of the planning meeting, when the entire topic team is present and inspection priorities are fresh in their minds.

The topic lead is responsible for assigning specific data collection (and data collection planning) tasks to team members. Usually these tasks are assigned by mutual consent, based on the strengths of each team member. Assignments should be recorded in writing. Additional guidelines include:

• Although the skill mixture among team members will vary, assignments should be made to take advantage of each team member’s skills and areas of expertise.

• Workloads should be evenly distributed; inspectors assigned major tasks should be assigned fewer tasks.

• Whenever possible, inspectors should work in pairs during data collection. Although time limitations may require individuals to work alone, two inspectors should be assigned to each task where there is a reasonable potential for disagreement, conflict, or poor performance.

• A person new to the specific subtopic or system should always be paired with an experienced inspector. First-time inspectors should be scheduled to participate in or observe as many data collection activities as possible, instead of being required to complete a specific task from start to finish.

• When inspecting facilities with several quasi-independent systems, it is normally more efficient to form smaller teams of inspectors, with each smaller team reviewing one or more aspects of the system.

• Scheduling should always provide time for daily validation sessions, analysis of data collected, and anticipated inspection team meetings.

A detailed and realistic schedule should be worked out for all inspection activities, but all involved should understand that the schedule may change to accommodate situations encountered during the inspection. The schedule may begin as an outline of data collection activities, locations, and general times (morning, afternoon, evening). Specific times and locations for each event will be worked out during detailed planning by team members.

Additional scheduling considerations are:

• High priority critical and major events (for example, performance tests) should be scheduled first.

• Higher priority tasks should take place early in the data collection period so that the schedule can be slipped if problems arise.

• Travel distances and time of year (weather) should be taken into account.

• The number of hours scheduled should be reasonable.

Each inspector is responsible for planning the details of his or her data collection tasks. Although planning is required for each inspection activity, the planning process will vary greatly, depending on the nature of the activity involved. Typically, detailed planning takes place during the planning meeting, but may continue and culminate during the period between the planning meeting and the return to the site for data collection. However, for some more involved activities, the planning process may continue into the onsite data collection period, especially when safety coordination is required.

Plans must address all items required to accomplish the data collection activity. Some common items that must be determined and arranged before actual data collection activities begin are:

• Location of the activity

• Time of the activity

• Sample size and configuration (people, documents, equipment)

• Equipment needed, and who will provide it

• How the activity (test) will be administered

• Who will evaluate the activity

• What evaluation standards apply

• Questions and answers for written tests

• Questions to be asked during interviews

• What data collection forms need to be developed, if any

• What evaluator checklists are to be used and who will prepare them

• What safety considerations apply and who will coordinate.

Performance tests require detailed planning. Performance tests are addressed in some detail in Section 4 and in more detail in the topic-specific Inspectors Guides.

As many requirements as possible, including likely support requirements, should be identified during the planning meeting so that they can be discussed in person with facility and DOE field element contacts. Support requirements include internal administrative, secretarial, and writing/editing support, as well as external support in providing site access, facilities, points of contact, personnel, and documentation.

Internal support is provided by OA-10 staff or support contractors. It is the topic lead’s responsibility to make the necessary notifications or arrangements for internal support.

External support is provided by the inspected facility or DOE field element. External support requirements must be communicated to the designated points of contact as early as possible, preferably during the planning meeting. Points of contact are responsible for seeing that requested support is provided. External support largely consists of personnel, facilities, and equipment. Figure 6 provides a checklist of items to discuss with the points of contact and includes typical categories of requested support.

Planning and coordination activities often continue between the planning meeting and the return to the site for data collection and closure.

Topic team members refine detailed planning for performance tests and other data collection activities, including revising areas of emphasis, populations, and random samples.

Although inspection planning takes place before the inspection to the maximum extent possible, plans for individual data collection activities may need subsequent revision—perhaps even up until the activities have been completed. The more complex the activity, the greater the chance that plans will have to be revised before or during inspection conduct. Therefore, plans should not be considered as "set in concrete," and topic team members should be prepared to continue the planning process, as required, during the onsite conduct of the inspection.

The conduct phase is the onsite data gathering period, normally comprising the first four to seven days of the inspection data collection visit. Although some data collection occurs during the planning meeting, the bulk of it occurs during the conduct phase. It is a period of intense and varied activity for the entire inspection team and many site personnel involved in the inspection. This stage of the inspection is crucial, because the inspectors collect most of the information they need to determine whether the protection programs meet requirements and are effective.

The goal in conducting the inspection is to accomplish all planned data collection activities in a fair, impartial, professional manner and to validate the technical accuracy of the data collected.

The inspection team’s activities normally begin with a meeting between topic team members and points of contact. This meeting provides the opportunity to:

• Review follow-up items from the planning meeting

• Receive reports from the points of contact regarding support arrangements

• Discuss any issues that may have developed since the planning meeting

• Work out details of the inspection schedule (for example, escorts for inspectors, and points of contact for each activity)

• Identify and discuss any additional actions.

Data collection activities generally follow the plans and schedules developed during the planning meeting. Inspectors normally focus on accomplishing planned activities; however, data collection activities can be adjusted to accommodate changing conditions. For example, inspection results may necessitate reduced or expanded activities in planned areas of emphasis and investigation of areas not originally identified for review. Problems or potential problems that become apparent during the course of the inspection should not be ignored simply because they were not included in formal planning.

Significant changes to planned activities should be discussed with the topic leads and approved by the Inspection Chief before being implemented. All changes should be discussed and coordinated with the points of contact to avoid scheduling conflicts and other potential problems.

Inspectors usually must handle classified documents and sensitive unclassified information during the course of an inspection. This information may be provided by OA-10, screened as part of the inspection process, borrowed from the facility being inspected, or generated by the inspectors. Additionally, most inspectors use classified word processing equipment during the inspection.

Inspectors are required to comply fully with all applicable DOE and local security requirements, especially those concerning classified computers, documents, and discussions. The Inspection Chief will provide for appropriate site-specific guidance and instructions to the team on these matters. All team members must comply with the policy and guidance issued.

The Administrative Support Coordinator is normally in charge of controlling classified matter in the custody of the inspection team. Documents generated by team members must be reviewed for classification by a designated team member who is an Authorized Derivative Classifier.

As discussed in Section 2, the cooperation and assistance of line organization personnel—whether representing DOE Headquarters, the DOE field element, or facility contractor organizations—are crucial in conducting a successful inspection. Inspectors should maintain the highest standards of conduct when dealing with points of contact, supervisors, security managers, and other personnel during the course of inspection activities. Professional conduct and relationships with personnel, points of contact, and trusted agents are covered in more detail in Appendix B of the OA Appraisal Process Protocols.

Data collection is the heart of the inspection process. While some data are collected by way of document reviews and interviews during inspection planning, most data are collected on site during the conduct phase of the inspection. Topic inspection teams may use a variety of methods to gather data. Specific methods vary, depending on the subtopics being inspected, site conditions, and preferences of the topic team members. Inspectors use five basic methods to collect data: document reviews, observations, interviews, knowledge tests, and performance tests. Each of these methods possesses inherent strengths and limitations; inspectors should carefully choose data collection methods and attempt, when possible, to employ complementary methods to ensure complete and accurate data development.

All protection programs rely on detailed documentation to ensure that they are effective and properly administered. The lack of well-developed, comprehensive policies and procedures is often the first indication an inspector receives that the program may be deficient. Therefore, reviewing documentation 1) determines whether written policies and procedures are consistent with DOE requirements; 2) provides a baseline picture of how the program operates; and 3) may reveal weaknesses that need further exploration.

The team may request that certain information be made available at the site, ready for team use at the beginning of the inspection visit. Reviewing documentation continues throughout the data collection phase. Often, inspectors must request additional documents during data gathering to develop a complete picture of facility programs and how they function. Requests for additional documentation should be made to the appropriate point of contact. If difficulties are encountered, the Inspection Chief should make a follow-up request directly to facility management.

Documents of interest are usually 1) policy documents on how the protection programs are supposed to function; and 2) records indicating whether facility programs comply with requirements.

Policy documents normally include, but are not limited to, security plans, policies, and procedural guides.

Records of interest include administrative records, document control records, records indicating completion of required reviews or actions, training records, equipment maintenance/calibration records, and inventory records.

Observations allow inspectors to see how site personnel actually do their jobs, and to evaluate their performance of duties under normal conditions. Such observations provide the best data on whether site personnel follow established procedures, and whether they properly operate any equipment for which they are responsible.

Observations should be made at as many key points in the program as practical. Not all observations need be scheduled inspection activities. Observing personnel at work is an opportunity for adding to data being gathered or helping to validate data already collected.

Although observation of personnel performing their duties would seem to be an ideal inspection tool, it is not necessarily simple:

• The team members must decide how much time they can allocate for observation. Will an hour spent watching a specific task yield an hour’s worth of usable data? In many instances, the answer to such a question will be "no," since not all activities associated with the program being inspected occur on a predictable schedule.

• The presence of an inspector may influence the behavior of the individual being observed, and produce erroneous data. This may be particularly true if the individual’s supervisor or other site representatives are present.

• The results of observation, frequently subjective, may lead to disagreement between the inspection team and site personnel on what was actually observed and may be difficult to validate.

For these reasons, observation as a data collection method is generally confined to rounding out the inspection team’s overall understanding of how routine tasks are carried out, or to evaluate performance in specific areas.

Interviews actually begin during the planning phase, when inspected personnel and points of contact are asked to provide information on certain aspects of the facility’s security program. Interviews also provide an important continuing source of information about the protection programs during the inspection.

Any person associated with the program being inspected is a potential interview candidate. Although interviews are often used to confirm or round out the inspector’s knowledge, they are most effective in determining perceptions and individual understanding of policies, procedures, and duties.

OA-10 uses both formal and informal interview techniques. For formal interviews, topic teams prepare a series of questions based on review of documentation during the planning meeting. The questions are then asked during scheduled interviews with DOE Headquarters, the DOE field element, and facility contractor representatives. Whenever possible, OA-10 Federal staff should be present at management interviews; an OA-10 manager should be present for interviews of senior managers. Interview techniques are discussed in some detail in Appendix A of the OA Appraisal Process Protocols.

Points of contact can usually answer many of the questions during the planning meeting. When a question cannot be answered immediately, site representatives are expected to provide an answer during the interval between the planning meeting and the beginning of onsite data collection, or when the inspection team arrives on site. Informal questions are those that arise from interactions between inspection team members and site personnel. Whether during a scheduled interview or an incidental conversation, inspectors should pay attention to what site personnel say and follow up on subjects of interest.

Since important issues may arise in an unpredictable manner, team members should be cautious about questioning site personnel in the absence of an assigned point of contact. Information elicited when a point of contact is not present may prove impossible to validate. By the same token, inspectors should be wary of attempts by points of contact to coach interviewees or to influence the interview.

The key to successful program implementation is how well personnel know and perform their duties. Job knowledge is normally assessed by interviewing personnel involved in the topic or subtopic during the inspection.

There is a certain body of knowledge, some Department-wide and some site-specific, that people associated with any program must possess. Formal knowledge tests are an effective way to determine whether personnel possess this knowledge. Oral, written, or combined oral-written tests are most often used.

When knowledge tests are given, a representative sample of the appropriate population should be tested. Questions and answers should be carefully validated before the test is administered to ensure that the test is properly constructed to achieve its intended purpose. Inspectors should understand that knowledge tests indicate only whether personnel are knowledgeable in certain areas, not whether they can apply that knowledge or perform related duties.

Performance testing is one of the most valuable data collection methods used during the inspection. In contrast with knowledge testing, performance testing is designed to determine whether personnel have the skills and abilities to perform their duties, whether procedures work, and whether equipment is functional and appropriate. A performance test is a test in which elements of a protection program—personnel, procedures, or equipment—are tested to determine whether they can actually perform or produce what is required.

Virtually any skill, duty, procedure, or item of equipment can be performance-tested. Performance tests may vary in complexity from the simple duplication of a classified document to more complicated and elaborate tests involving adversaries, Engagement Simulation System/Multiple Integrated Laser Engagement System equipment, aircraft, and large numbers of Security Police Officers.

Some tests can be conducted under completely normal conditions, where the subject is unaware of the testing. Other tests must be conducted under artificial conditions, although maximum realism is always a primary consideration.

To promote safety and realism in performance testing, OA-10 has established formal protocols for planning and conducting certain performance tests. These are detailed in the topic-specific Inspectors Guides.

Before any performance test is conducted, all test activities must be coordinated with site representatives. In tests where the subjects are aware that they are participating in a test, all participants should be briefed in detail concerning the actions expected of them; topic team members responsible for conducting the performance test should exercise careful control of all activities during the test; and test results should be validated as soon as possible after the test is completed.

A sample performance test plan format, intended as a convenient guide for describing proposed tests and as a quick reference during the actual conduct of the test, is provided below. The format can be adapted to fit test requirements at varying levels of complexity. The most complex format contains:

• Objective — Identifies the portion of the program the test is to measure and briefly describes what the test is designed to accomplish.

• System Description — Provides a succinct characterization of the system. This helps team members understand system parameters and serves as a quick refresher that can be reviewed immediately before beginning the test.

• Sampling Technique — Explains how the sample to be tested will be selected and handled and serves as a record of these actions for future reference.

• Scenario — Describes how the performance test will be conducted. The scenario may include specific points that must be covered to serve as a reminder to personnel performing the test. Frequently, for less complex performance tests, the system description and sampling technique are discussed together under this heading instead of in separate sections.

• Evaluation Criteria — Provides the applicable references—DOE order, directive, or standard—that will be used to determine whether requirements are met.

• Safety Plan — If the performance test has safety implications, a detailed safety plan is required.

• Additional Controls — In some cases, specialized coordinators, such as a range officer or radiological control personnel, may be required.

This format should not be considered mandatory. In some cases, facility contractors have developed acceptable formats that their personnel are used to, and it may be convenient to use the local format. However, whichever format is used should provide sufficient detail to plan and conduct the test and to serve as a reference and record of what was accomplished.

Major Engagement Simulation System-enhanced performance tests of a facility’s tactical response capabilities are not normally conducted during comprehensive inspections. The extensive planning, coordination, and resource requirements associated with such tests, and their potential impact upon other data collection activities preclude their inclusion during the normal course of a comprehensive inspection visit. When indicated, such major performance tests will normally be conducted as a special inspection, using the protocols established in the Context and Protocols for Performance Testing of Protective Forces.

Other Methods

While the five basic methods of data collection described above will satisfy most inspection data collection needs, team members may use any legitimate method to most effectively collect needed information. Other methods, such as the use of surveys or questionnaires for example, may be used when appropriate.

Data Collection Forms

Collected data must be recorded in a standard manner so that it can be properly analyzed and archived. On a daily basis, inspectors are required to record pertinent data on the OA-10 Data Collection Form. The form accommodates a discussion of collected data, including its importance, its impact on the program being evaluated, and necessary follow-up activities. It can be modified as more data is collected or its impact changes. A copy of the Data Collection Form format is provided in Appendix B. An electronic version of the form is available from the Inspection Chief. All Data Collection Forms must be completed in an electronic file format, and will be archived by OA-10 for future reference.

In addition to internal topic team use, completed Data Collection Forms are to be turned in to the Inspection Chief daily, and will be used to develop the daily briefing for DOE field element and facility managers. Completed forms must be reviewed for classification, as appropriate.

Information sharing among topic team members and between topic teams is imperative. Information collected by one team member may have a direct impact on a line of investigation being pursued by another. Information collected by one topic team may become more significant when combined with information collected by another topic team. It is absolutely essential for the PPM topic team to be aware of the data being collected by other topic teams. Consequently, a conscious and deliberate effort at timely information integration is a necessity. Integration is conducted both formally and informally. Informal integration is expected on a daily basis between topic team members and between topic teams. Inspectors are expected to make a deliberate effort to share information they have collected with other team members who may find it useful, and to seek out needed information from team members who may have it. More formal integration is achieved through team meetings called by the Inspection Chief and by reading the folding containing all Data Collection Forms.

Validation is the process inspectors use to verify the accuracy of the information they have obtained during data collection activities. It is the most critical element of the onsite inspection. Validation is a continuous process to ensure that:

• All data collected by the inspectors are factually correct and can be used to evaluate the effectiveness of the program.

• Points of contact and site management are aware of the data that have been collected. They must either acknowledge its accuracy, provide correct information, request that further data be collected, or provide mitigating information. Representatives of the Cognizant Secretarial Office (CSO), Lead Program Secretarial Office (LPSO), DOE field element, and facility contractor may participate in validations.

Information to be validated should be presented as thoroughly, accurately, and concisely as possible. The purpose of validation is to ensure that points of contact agree with the accuracy of the information collected and understand its potential implications and impacts.

Inspectors are required to validate inspection results with their points of contact on a daily basis. The exact method of validation depends on the topic team, the points of contact, and the schedule of events. Even if the points of contact accompany the inspectors on every inspection activity and validate observations on the spot, a daily validation meeting is required. Usually, a short meeting is held at the end of the day to validate the day’s events. At times, particularly if activities extend late into the evening, the daily validation meeting may be held the following morning. However, any such delay should be discussed with the Inspection Chief since this could cause facility management to receive reports of potential problems before the points of contact know the issues involved.

Major Deficiency Identification

When serious or potentially serious deficiencies are identified during an inspection, they are to brought to the attention of the Inspection Chief and the appropriate DOE field element and facility personnel (usually managers) as soon as possible. Once enough data is collected to be reasonably sure that a significant or potentially significant deficiency exists—particularly a rating-impacting deficiency—it is to be identified and formally communicated to site managers in sufficient detail to ensure that it is fully understood. This formal communication is achieved through the use of the Issue Form, a copy of which is provided in Appendix B (electronic copies may be obtained from the Inspection Chief). The topic team is responsible for completing the form, which is submitted to and signed by the Inspection Chief, who forwards it to the DOE field element/facility. The responsible field element or facility organization is required to respond to the issue paper in writing, and may also request a meeting to discuss or further clarify the deficiency and its potential impact.

Deficiencies identified in this manner may or may not ultimately result in formal findings, depending on the individual circumstances. The Inspection Chief will communicate significant deficiencies to the OA-10 Director, who will, when appropriate, inform the OA Director. As necessary, the OA Director will inform senior Departmental managers.

A summary validation is held after data collection activities are completed. Ideally, the summary validation is conducted at the working level and attended by members of the topic team and points of contact for the program; however, summary validations are often attended by senior managers and representatives of interested Headquarters organizations. At the summary validation all significant information, including items validated previously, are validated again. This is the final validation activity before inspection report preparation. Although actual (or even potential) ratings should not be discussed, the validation process should ensure that the issues and their impacts are fully understood so that there will be no surprises in the report.

After data collection, the data must be assimilated, compiled, and analyzed in order to report the results. The inspection closure process usually takes place during the last week of the inspection visit and includes a number of tasks to ensure that all pertinent information is accurate, reported in a standardized format, and appropriate for the intended audience.

This section discusses the tasks involved in inspection closure, including data review, analysis of results, determination of findings, assignment of ratings, and integration with other topics. These tasks form the basis for the inspection report. Other closure tasks include preparing briefing materials, reporting policy issues, and accomplishing various administrative actions.

The goals of inspection closure are to:

• Identify and clearly report the inspection results, including both strengths and weaknesses

• Determine the individual and cumulative impact of inspection results on the ability of the protection program to accomplish its mission requirements

• Assign rating(s) that accurately reflect the actual performance of the program(s) (for inspections where ratings are assigned)

• Report inspection results to local management

• Produce a report that clearly and objectively represents the current status of protection programs, including an assessment of mission performance

• Brief Headquarters management and other appropriate parties

• Complete all routine and special tasks that may be assigned by the Inspection Chief.

Data review consists of sorting out and logically grouping all validated data collected for each topic and subtopic during any phase of the inspection. Although the topic teams are generally aware of most of this data, not all team members will be familiar with all data collected. Consequently, the topic teams must review all pertinent data to develop a comprehensive picture of how effectively the protection program meets requirements.

Topic teams generally arrange the collected data according to positive or negative features to aid in clearly identifying strengths, weaknesses, and positive or negative trends. Proper organization and thorough review of all inspection data are essential to completing the analysis and preparing the report.

Data gathered and developed by one topic team often affect other topics being inspected. To take this interdependency into account, topic teams continue their integration activities until all pertinent information has been shared. This integration normally consists of a discussion of inspection results among topic teams regarding how information developed by one team influences the adequacy of the performance observed in another topic area.

Each topic team should consider information obtained through integration, along with its own data, during data analysis. When necessary, the inspector who observed the data to be integrated may prepare draft input for use by another topic team.

The continuous process of analyzing collected information culminates during the closure phase, when all data are critically reviewed—a review that results in conclusions regarding the effectiveness of the evaluated program. A discussion of the analysis process is contained in Section 5 of the OA Appraisal Process Protocols.

Each topic team is responsible for determining which inspection results are designated as findings; findings usually identify aspects of the program that do not meet the intent of DOE policy. Although any program element or system not in compliance with DOE policy or not meeting DOE performance standards may be identified as a finding, topic teams are expected to exercise judgment. Minor and non-systemic items are omitted.

Findings are presented in a manner that identifies both the specific problem and the reference (DOE order requirement). If findings address specific aspects of a standard, the topic team should determine whether the potential findings should be "rolled up" and reported as a single finding. This "rollup" may be appropriate if the single finding statement can clearly and completely convey the problems. Findings should always be worded to express the specific nature of the deficiency, clearly indicate whether the deficiency is localized or indicative of a trend, and clearly identify the organization responsible for the deficiency. Typically, the impact is presented immediately after each finding and includes compensatory measures, mitigating factors and current and planned corrective actions.

The rating system OA-10 uses to characterize the status of safeguards and security programs at DOE facilities is summarized in Section 5 of the OA Appraisal Process Protocols.

OA-10 assigns ratings based on a thorough analysis of inspection results and their implications. The OA-10 inspectors are responsible for assigning ratings; however, internal OA-10 protocols require the inspectors to defend the validity of the ratings with the inspection manager. In turn, the manager presents the validations to the Director of Independent Oversight and Performance Assurance. This layered "check and balance" concept of operation assures the highest degree of confidence that the ratings are fair and objective.

For comprehensive inspections, OA-10 normally assigns ratings at two levels: topic level and overall safeguards and security program level. However, when special inspections or major performance tests evaluate subtopical or cross-topical elements of a program, ratings may be assigned to indicate the performance level of the specific program elements evaluated. This rating system provides the necessary flexibility to allow use of the most appropriate rating structure for each individual situation encountered.

In rating topics (or subtopics), the analysis (and rating) is applied to the standard at the topic level. Inspection results in each subtopic are evaluated and analyzed. Deficiencies in one subtopic do not necessarily determine the topic rating. For example, deficiencies in a protective force training program do not necessarily lower the topic rating; if no other significant deficiencies exist, and if the protective force can adequately accomplish its mission (that is, meet the requirements of the topic standard), the topic area could be rated Satisfactory.

For the purposes of rating the "bottom-line" effectiveness of a safeguards and security program, OA-10 analyzes and integrates the results of all topic areas to determine how well the program accomplishes its basic purposes. This analysis cuts across topics and recognizes the interrelationships of the various programs and functions represented by the topics; that is, the functions represented by each topic do not stand alone, but are all interrelated parts of a single safeguards and security program. This integration and analysis process is complex, and cannot follow a strict formula of "averaging" topic area ratings. Weaknesses in one or more topics may not significantly affect the overall rating if those weaknesses are mitigated by strengths in other topics and do not significantly affect the protection level. On the other hand, a catastrophic weakness in one topic, or pervasive weaknesses throughout several topics, can affect the overall program rating if the soundness of the overall system is degraded.

At times, particularly during special inspections, including major performance tests, OA-10 inspects and rates performance on issues, functions, or program elements that do not fit into the discrete boxes defined by topics. In such cases, OA-10 has the latitude to clearly define the scope and purpose of the inspection, and to rate performance associated with the specific issue inspected. The same terminology and definitions are used.

The inspection report is the formal product of the inspection process and is intended for dissemination to appropriate managers at DOE Headquarters, field elements, and facility contractors. It is the only published record of the activities and results of an inspection, and the information in the reports is used as input to the Annual Report to the President and the Annual Report to the Secretary, both describing the status of safeguards and security in the Department. The report should reflect a balanced view of program strengths and weaknesses.

Each topic team is responsible for writing the report appendix documenting the inspection of its topic area. The information contained in an appendix includes a discussion of the current status of the topic, inspection results, conclusions drawn from those results, a topic rating, and identified opportunities for improvement.

Typically, one individual is assigned to write the main body of the report, which, drawing on information provided in the topical appendices, summarizes the results of the inspection, analyzes and draws conclusions about the effectiveness of the evaluated safeguards and security program, and indicates an overall program rating. Appendix C of the OA Appraisal Process Protocols provides guidance for this process.

A typical report format for a comprehensive inspection is provided in Appendix A. Additional site-specific report format guidance, if different from that provided in Appendix A, is provided by the Inspection Chief before preparing the report.

The onsite report-writing process follows a fairly standard sequence, from initial to final draft.

The Quality Review Board normally consists of several managers and senior personnel from OA-10 and its support contractors. They review each draft report appendix to ensure that it is readable and logical, and that it contains adequate, balanced information to support conclusions and ratings. The Quality Review Board may require topic teams to revise portions of their appendices. Each draft appendix must be reviewed and accepted by the Quality Review Board before the report-writing process continues and the appendix is included in the draft report.

OA Director’s Review

Following acceptance by the Quality Review Board, report sections are typically submitted to the OA Director for review and approval. Since the OA Director is usually not on site when this occurs, report sections are often faxed to Headquarters for review. If this review indicates needed changes, the appropriate topic team coordinates with administrative support personnel to make the changes.

DOE Field Element Comments

When each appendix is reviewed and accepted by the Quality Review Board and approved by the OA Director, it is sent to the inspected DOE field element (operations office) for review. The field element staff and, at their discretion, facility contractor staff are allowed a brief period (typically less than one day) to review the draft report for factual accuracy. They may provide written comments concerning any portion of the report that they believe to be in error.

If the CSO has representatives on site, they may also be provided the opportunity to participate in the field element review.

Final Draft

Upon receiving DOE field element comments, the topic teams recommend any appropriate changes to the draft report; the Inspection Chief authorizes any changes merited by field element comments. When all such changes have been made, the final draft of the report is prepared and is normally provided to the manager of the DOE field element at or immediately following the inspection outbriefing.

Topic teams must accomplish several additional inspection-related activities, as required, during the inspection closure process:

• Preparing policy issue papers if the topic team encountered any issues that should be brought to the attention of DOE Head-quarters elements (typically the Office of Security and Emergency Operations, but potentially CSOs and/or LPSOs)

• Preparing a one-page executive summary of the inspection results for the Secretary

• Preparing inspection data to be maintained by OA-10 at Headquarters for future reference and possible use in preparing the Annual Report to the President and the Annual Report to the Secretary

• Returning all site items (for example, documents, access credentials, dosimeters, and special clothing).

Formats and other specific information associated with preparing policy issues are provided in Appendix B.

OA-10 continuously strives to improve its processes and increase the effectiveness of its oversight activities. Immediate feedback from inspection team members provides important input to the improvement process. Information is solicited from team members regarding possible improvements to any aspect of the inspection process. Typically this is accomplished in a roundtable discussion conducted after the initial draft report sections have been completed but before significant numbers of team members have been released to leave the site. The Inspection Chief determines the time and agenda for the roundtable discussion and assigns a staff member (often the Deputy Inspection Chief) to chair the discussion and record significant results. While the roundtable discussion is the common method for soliciting process improvement information, the Inspection Chief may determine that priorities require other, perhaps less formal, methods to be used to collect this data. The imperative is that all team members have the opportunity to provide input, and that all input is recorded and provided to managers for consideration.

OA-10 typically provides an outbriefing to managers of inspected organizations before departing a site. The outbriefing is normally scheduled for the morning of the last day on site, but may be scheduled differently if necessary to accommodate the availability of critical managers. When necessary, the briefing is scheduled for a later date.

The briefing is usually given by the OA-10 Director; the OA Director often attends, particularly following comprehensive inspections. The OA-10 Director usually designates the topic leads and other team members who will be required to attend. The manager of the DOE element being inspected generally determines the inspectees who will attend. Typically, managers and primary safeguards and security staff from the DOE field element and the facility contractor will attend, along with any present representatives of the CSO/LPSOs and the Office of Security and Emergency Operations.

Depending upon the nature and results of an inspection, it may be necessary to brief the Secretary, Deputy Secretary, and/or Under Secretary on significant issues. The (one-page) executive summary of inspection results is submitted to the Secretary (Deputy Secretary, Under Secretary as appropriate); if requested, the senior official will also be briefed, normally by the OA and/or OA-10 Director(s). Other senior Headquarters managers may attend at the discretion of the senior official being briefed. OA-10 will be prepared to brief the significant inspection results and their implications, but the specific focus of the briefing may be prescribed by the official being briefed.

As soon as practical after OA-10 has received and reviewed the combined CSO/DOE field element comments to the final draft report, the OA-10 Director will coordinate, through the OA Director, with the CSO and DOE field element to schedule a briefing for the DOE Security Council. Whenever possible, the briefing will be conducted at the next regularly scheduled Security Council meeting. OA-10 will brief the inspection results.

The CSO and DOE field element have ten working days from their receipt of the final draft report to provide written consolidated comments to OA-10. Upon receipt, the OA-10 staff will review the comments and determine the appropriate responses. One or more topic team members will review the comments for their report section. When necessary, topic team members not located in the Washington area may be contacted by telephone or fax. If comments are received during a subsequent appraisal activity at another site, comments may be reviewed at that location by available team members.

The Deputy Secretary established protocols to guide the Department’s response to OA appraisal reports in a memorandum issued on August 31, 1999. Those protocols are provided in Appendix D of the OA Appraisal Process Protocols. They include requirements for various stages of corrective action plans and for tracking corrective actions and closing findings. OA-10 is assigned some responsibility under those protocols.

The CSO and the DOE field element have ten working days from receipt of the final draft report to prepare and provide to OA-10 a preliminary corrective action plan to address immediate and initial planned responses to all findings in the OA-10 final draft report. As soon as practical, but within ten days of receipt, OA-10 will provide the CSO and DOE field element appropriate informal comments regarding the adequacy of the proposed corrective actions in correcting the identified deficiencies.

Within 30 working days of receiving the final draft report, the CSO and DOE field element will provide OA-10 with an interim corrective action plan addressing, in detail, ongoing and planned corrective actions for each deficiency identified in the final draft report. OA-10 will review and comment on the interim corrective action plan within 15 days of receipt.

Within 30 working days of their receipt of the final report, the CSO and DOE field element will issue a final corrective action plan. Final corrective action plans should address, in detail, all completed, ongoing, and long-term actions associated with each finding in the report.

This section describes how an OA-10 comprehensive inspection report is prepared and provides guidance to inspectors, editors, and typists to ensure that the reports are clear and concise and cover essential points for the intended audiences. Questions on report preparation arising during an inspection should be directed to the Inspection Chief. Authors should not deviate from the established format without specific authorization.

In some ways, the process and guidance for preparing a report for other types of inspections (e.g., special inspections or follow-up reviews) is similar and the guidance below may be useful; however, the report outlines may vary for other types of inspections. The Inspection Chief will provide guidance as necessary for other types of inspections.

OA-10 comprehensive inspection reports are written for different readers with varying needs and levels of familiarity with the OA-10 inspection process and the specific facility being inspected. For example, experienced program officials need only a general description of the inspection topic to permit a quick understanding of a particular subject or operation. Inspected facility personnel need even less descriptive material, since they are presumed to be familiar with the complexities of their facility. Other readers might not be totally familiar with specific operations of the inspected site and may need more information to understand OA-10 issues and analyses. The report is generally written for readers who are knowledgeable in safeguards and security, but who may not be familiar with the inspected site. The report should reach a balance, not burdening readers with too much detail but giving them enough information to enable them to comprehend the scope of the inspection and the resulting issues.

The comprehensive inspection report is composed of two principal sections: the summary report, or "front end," that is intended for management-level reading and which provides an overall analysis of the effectiveness of the safeguards and security program; and the appendices, which are intended for those who seek a more detailed account of specific topical inspection activities and issues. Attachments are not necessary in most reports; however, experience has shown that there are times when they are essential to the analysis of an issue. In instances where attachments are deemed critical, the responsible OA-10 manager will decide on the placement and numbering system to be used.

OA-10 inspection reports consist of two parts: the report itself, or "front end," and the appendices. The front end is written by designated individual(s) selected before the inspection. The appendices are written by the topic teams who conduct the topic area inspections. The standard outline for the report is provided below. Specific guidelines for writing the report and appendices, including annotated outlines and examples, are provided in following subsections of this appendix.

The summary report (that is, Sections 1.0 through 4.0) consists of an introduction followed by a discussion of the performance and effectiveness of the various program elements as determined by the inspection activities. The report is a management-level overview integrating the results of the topical inspections and summarizing the "big picture" of a facility's program effectiveness. It contains conclusions and a rating pertaining to the facility’s overall safeguards and security program effectiveness.

Example Summary Report

The following is a generic example of a "front end" (summary report) of a longer appraisal report in which the details of appraisal results are contained in individual appendices. This example is provided to illustrate an application of the guidance provided in the previous pages; it should not be considered to be a rigid template that must be copied in all cases. It is not necessary to copy the language of the example when writing actual reports. While the general format and flow of information should be used in all cases, the individual circumstances of each appraisal effort will dictate the specific length and content of the summary report. For example, some of the specific information that may be addressed in the introduction will depend upon such circumstances as mission or contract changes, past problems, recent initiatives, and so forth. The length and complexity of the discussion in the results section will depend upon the results of the topical appraisals. Consequently, the specific circumstances associated with each appraisal should be considered, and appropriate judgment should be exercised in applying the guidance and principles provided above.

APPENDIX A – Supplemental information is administrative in nature and primarily identifies the individuals who managed and conducted the inspection. It identifies managers, members of the Quality Review Board, inspectors, and administrative support personnel who participated in the inspection. This appendix is typically prepared by the Deputy Inspection Chief, but it may be assigned to any other member of the inspection team.

APPENDIX B – "Findings" is a chart that consolidates all findings identified in the topical appendices. It is typically prepared by the administrative support staff.

APPENDIX C through I (topical appendices)

The topical appendices are written by the respective topic teams. An annotated outline of the topical appendix is provided on the following pages, followed by examples of typical Introduction and Conclusion sections. The Status and Results sections of the topical appendices contain finding statements. Finding statements are inserted at the appropriate point in the section – usually immediately following the discussion of the problem leading to the finding. Finding statements are preceded by a finding statement designator and are followed (in parenthesis) by the appropriate reference that applies to the finding. Finding statements are bolded, and the finding statement designators are formatted as follows:

OR1999-Y12-CS-1

Field 1(OR): Use HQ if a finding against a headquarters element; use the DOE field element symbol (e.g., OR, AL, RL, SR, etc) if a finding against a DOE field element or facility contractor.

Field 2 (1999): Use the four-digit designator for the year in which the inspection occurred.

Field 3 (Y12): Use the symbol for the facility to which the finding applies (e.g., Y12, LANL, LLNL, ORNL, etc.)

Field 4 (CS): Use the symbol for the topic area to which the finding applies (e.g., CS, PS, PSS, PPM, MCA, CMPC, or PF)

Example Appendix Sections

(U) The primary focus of the inspection of the ZZ Laboratory protective force centered on the protective force’s ability to perform the routine and emergency duties associated with the protection of special nuclear material. A secondary focus involved the force’s performance of duties associated with the protection of classified matter. Although performance of duties was emphasized, the areas of protective force management, training, and equipment and facilities were also examined. While data collection activities included interviews, document reviews, observation of routine activities, and several limited-scope performance tests, a significant aspect of data collection consisted of a series of large-scale MILES-enhanced performance tests.

(U) The January 1998 OA inspection of the ZZ Laboratory rated the protective force as satisfactory, although noting some deficiencies in basic skills and vehicle reliability. The May 1999 XX Operations Office’s security survey rated the ZZ protective force as marginal, noting deficiencies in response plans, firearms qualifications, and tactical response skills.

(U) In general, the physical security systems at ZZ Laboratory are primarily maintained in a manner that effectively balances the protection of SNM with the construction process. With the exception of the coaxial cable reliability problems, all previously identified issues have been or are being corrected. The formal process controlling changes to the alarm systems necessitated by construction assures that unauthorized changes are not implemented and that all responsible parties are notified of changes. Although some physical security system equipment is aging or lacking full support, the operations and maintenance methods in place should be sufficient to ensure satisfactory performance for the remaining required operational life of the equipment.

POLICY ISSUE FORMAT

Subject

CLASSIFICATION