Lesson Info:

• Lesson Number: 1183
• Lesson Date: 1989-10-25
• Submitting Organization: KSC
• Submitted by: Thomas Utsman/ Eric Raynor

Subject:

Description of Driving Event:

The Orbiter Processing Facility (OPF) Bay 2 Firex Deluge Zone 3 Viking arm valve AV1.3-2 was leaking, and on September 21, 1989, Problem Report (PR) number PV-6-140247 was initiated to replace defective parts. During system leak check operations following valve repair, water flow from zone 3 (directly over the Orbiter) occurred in OPF Bay 2. The time of first flow was established as 10:28 AM on September 24, 1989, as recorded by OPF Water Pump House instrumentation due to diesel pump startup. The Zone 3 water flow over the Orbiter was at a reduced rate, with only a partial nozzle discharge pattern.

Shuttle Processing Contract (SPC) Water Systems technicians were cycling the manual arming and firing valves while leak checking the system. Only a few minutes had passed from the time they cracked open (2 turns) the system manual supply valve until they heard about the water flow problem in Bay 2. The SPC Lead Water Systems technician ran into the OPF to see what was wrong and noted the deluge flow. He returned to the manual activation station behind the OPF and found that two SPC ground support equipment (GSE) technicians had opened the manual arming and firing valve to Zones 1, 2, 3, 4, and 5. (It is presumed Zone 3 did not flow additional water and had been deactivated by this time.) The Lead Water Systems technician then closed all the open manual arming and firing valves, as well as the manual supply isolation valves to stop all Bay 2 water flow.

The scenario that lead to the two GSE technicians being at the deluge system manual flow control valve station is as follows:

After the flow from Zone 3 occurred, the SPC OPF Bay 1 Site Division Manager asked a GSE technician to find someone who could turn the system off. This GSE technician, accompanied by a co-worker, proceeded to the manual activation station behind Bay 2. Upon arrival, they proceeded to position the manual arming and firing valves to Zones 1, 2, 3, 4, and 5 to the "on" position, thinking they were turning the firex system off. This turned on deluge system Zones 1, 2, 4, and 5 (Zone 3 had already been deactivated) to the "full on" mode. The Lead Water Systems technician then returned to the manual valve panels, proceeded to turn the manual arming and firing valves to the "off" position, and then helped his technicians close the manual supply isolation valves to Zones 1 through 5, which stopped all water flow.

Lesson(s) Learned:

The primary cause of the mishap was failure to follow the procedural instructions in PR number PV-6-140247, in which: (1) facility water technicians added a flow test on the Zone 3 system after valve diaphragm repair, and (2) in not fully closing or in improperly reopening the manual riser isolation valve, water was allowed to flow into Zone 3 of the OPF Bay 2 water deluge system.

Contributing causes included the following:

1. The OPF GSE technicians, who are members of the "Contingency Team," had not been adequately trained in the operation of the OPF water deluge system. This led to the OPF GSE "Contingency Team" technicians activating Zones 1, 2, 3, 4, and 5 of the Bay 2 deluge system, while thinking they were turning the system off.
2. A thorough pre-task briefing (PTB) was not held for the water deluge system valve repair task, which could have prevented deviation from the repair procedure by the facility water technicians.
3. The labeling on the remote manual arming and firing valve activation station of the OPF water deluge system was misinterpreted. This contributed to the OPF GSE technicians activating Zones 1, 2, 3, 4, and 5 of the Bay 2 deluge system when they believed they were turning the system off.
4. The remote manual arming and firing valves are in the "on" position when perpendicular to the water lines, which is not the industry norm for hydraulic systems. This contributed to the OPF GSE technicians believing they were closing the valves by positioning them perpendicular to the water line when, in fact, they were opening the valves. This human factor problem was not addressed in the System Assurance Analysis for the Water System at the OPF, SAA09FY09-007, Revision B, dated July 1987.
5. The manual arming and firing valves behind the OPF are easily accessed by simply opening the individual valve box doors, which destroys the paper integrity seals. Essentially, there was no physical barrier to the valves.
6. Even though redundancy exists for the Viking valves in the deluge system, these valves are open when no dome pressure is applied. Failures which remove dome pressure could result in the flow of water.
7. There are significant configuration differences between the OPF Bay 1 and 2 water deluge systems. The OPF Bay 1 configuration does not include the manual riser isolation valve. Engineering Order E084-79KO5423, which installs the manual riser isolation valve for both bay configurations, has not yet been accomplished for Bay 1. This should be implemented for Bay 1 after OV-103 rollout for STS-33 and prior to another Orbiter occupying Bay 1. There are only 6 boxes for the manual arming and firing valves in the Bay 1 configuration, with the arming and firing valves for a particular zone in the same box. The Bay 2 configuration has 12 boxes, one for each valve. Signs on the boxes are not uniform between OPF Bays 1 and 2.
8. The NASA Test Director (NTD) evacuated OPF Bay 2 via a public address announcement. He could not use the area warning system because OPF Bay 1 would have also been evacuated, which was not required.
9. As a result of the inadvertent activation of the firex deluge system, Orbiter GSE was damaged (i.e., the Mechanical Control Interface Unit for the Remote Manipulating Arm and the Cargo Verification Unit). No redundancy is available for the Mechanical Control Interface Unit.
10. The OPF Bay 2 evacuation during this incident was in accordance with established procedures. Security control of Bay 2 was properly maintained. All personnel and emergency service actions were professional and properly carried out.

Recommendation(s):

1. Emphasize to all individuals involved in the work task that proper procedure execution is mandatory.
2. Access to the OPF deluge system Zones 1 through 6 should be positively controlled (e.g., a lock and key arrangement). Personnel with access should be fully trained and certified in the operation of the system.
3. A policy should be developed outlining which operations/ activities require PTBS. This policy should be rigorously implemented when developed.
4. As part of the reassessment of the use of this system, all control panels should be properly labeled. A positive position for open versus closed should be readily visible on all valves.
5. The manual arming and firing valves should be oriented to the industry norm and provided with specific open and closed position markings.
6. Access to OPF deluge system Zones 1 through 6 should be positively controlled (e.g., a lock and key arrangement). Personnel with access should be fully trained and certified in the operation of the system.
7. The policy of free access to the water deluge control panel key should be reviewed.
8. Evaluate the risk of inadvertent OPF deluge system water flow with the current system design.
9. The water deluge system valve repair task was not observed by Engineering, which could have prevented the deviation to written procedure by the facility water technicians.
10. When malfunctions on critical facility/GSE systems occur which could affect flight hardware, a clear policy is required that outlines what level of management approval is needed to initiate repairs.
11. The OPF area warning system should be split between Bay 1 and Bay 2, enabling the NTD to only evacuate one OPF bay when the emergency does not warrant evacuating both bays.
12. Non-redundant GSE should be identified, and workarounds developed if this GSE is destroyed or incapacitated for an extended period of time.

Evidence of Recurrence Control Effectiveness:

Documents Related to Lesson:

Mission Directorate(s):

• Space Operations
• Exploration Systems

Additional Key Phrase(s):

• Emergency Preparedness
• Facilities
• Fire Protection
• Ground Equipment
• Ground Operations
• Hardware
• Human Factors
• Logistics
• Mishap Reporting

Additional Info:

Approval Info:

• Approval Date: 2002-04-17
• Approval Name: Gena Baker
• Approval Organization: KSC
• Approval Phone Number: 321-867-4261