Lesson Info:

• Lesson Number: 1198
• Lesson Date: 1990-07-26
• Submitting Organization: KSC
• Submitted by: Robert Luken/ Eric Raynor

Subject:

Description of Driving Event:

On July 26, 1990, 14:06 EDT, a Payload Hazardous Servicing Facility (PHSF) fire alarm indication was received at the Protective Services Control Center (PSCC) in the Launch Control Center (LCC). Fire services responded and found no alarm indications at the facility. Initial troubleshooting by EG&G Fire Alarm (F/A) technicians indicated the trouble was on the communications pair between the PHSF and the PSCC.

At 17:48 EDT, a communications technician was dispatched to the PHSF, to support troubleshooting of the problem. The technician arrived at the PHSF Control Building and checked the signal pair at Frame 102. Indications were that the problem was at the Service Building end of the line. He then proceeded to the Service Building and, upon entering, heard an audible alarm. Reporting the alarm, via radio, to the trouble desk at the Communications Distribution & Switching Center (CD&SC), he proceeded to the other side of the facility to find someone to ask about the alarm.

Meanwhile, the security guard having reported the audible alarm to the PSCC, via radio, performed a perimeter sweep of the building and encountered the LSOC technician. After some discussion the technician resumed his troubleshooting and the guard continued the sweep.

Fire Services responded to the facility and found a Fire Alarm Indication for Zone 19, indicating the Water Deluge System had been activated. Checking showed that the ARM valves had been activated, which caused the Fire Alarm indication. However the ACTIVATE valves were still closed and no water was flowing.

The Gamma Ray Observatory spacecraft was located in Zone 1 of the Service Building highbay. The Water Deluge System for Zone 2 had been armed. Water flow from Zone 2 would not have caused direct impingment on the spacecraft but could have caused extensive water damage requiring retesting and repair.

Analysis of the system uncovered several design and implementation deficiencies that could have contributed to the close call, but no positive evidence was found to explain the anomaly.

Possible causes:

1. Manual activation of pushbutton by an unknown party.
2. Inadvertent short of terminals by the LSOC technician.
3. Intermittent short in cable plant.

Lesson(s) Learned:

Communication System:

1. Frame Control: Each communication system mainframe has attached to it a large status indicator. These indicators are used to control access to the frames and are remotely operated by communications personnel. However, only those frames presently used to support Shuttle operations or those in facilities directly supporting Shuttle operations are controlled. Frames, in facilities like PHSF, are kept in the "OPEN" state, allowing uncontrolled access for any type of work or troubleshooting.
2. Critical Functions: Critical functions that are routed through comm. frames are not identified or protected, except for Fire Alarm indications. F/A pairs utilize red & white jumper wires and have plastic terminal covers to prevent accidental shorts. All other pairs utilize black & white jumper wires. Jumper wires are used to make connections between the vertical columns of terminals on one side of the frame to the horizontal rows of terminals on the opposite side.
3. Documentation: There were no formal troubleshooting procedures being followed, nor was there any documentation of the troubleshooting steps being performed. Troubleshooting was performed using radio, telephone, and handsets. The only traceability was from radio channel recording; telephone and handset communication are not recorded. There seemed to be no distinction in trouble-shooting methods employed for critical vs. non-critical circuits.

1. Drawings: The system schematics of the Water Deluge System at the PHSF did not reflect that the control lines between the Control Building (M7-1357) and the Service Building (M7-1354) were routed through comm. system Frames. They incorrectly showed direct dedicated lines between the buildings
2. Separation: The "ARM & ACTIVATE" values are manually operated from three control panels, two located in the Service Building (M7-1354) and one located in the Control Building (M7-1357). These panels contain momentary pushbuttons for opening and closing each value (redundant pair). When activated, a pushbutton provides a current path between two control wires. This causes a pair of control relays to activate and latch themselves in the "on" state. Any connection between the two control wires will have the same effect as operating a pushbutton. The Water Deluge System at the PHSF is implemented such that the control wires are in a common pair and appear on physically adjacent terminals in at least six places.
3. Routing: The control lines for the "ARM & ACTIVATE" valves are routed through Terminal Distributors and Frames along with lines from other systems. There is no access control on this equipment; it is accessible for work or troubleshooting by people with no knowledge of the presence of criticality of the Water Deluge circuitry.
4. Monitoring: The Water Deluge control logic is designed and implemented such that there is no monitoring or traceability of which panel and/or pushbutton was actuated. Therefore it is impossible to tell if the system was actuated by a malfunction of by a switch actuation.
5. Other Facilities: There are at least 15 other facilities with the Water Deluge Systems utilizing remote control panels. There is a high probability that these systems are implemented that same as the PHSF system.

Recommendation(s):

Communication System:

1. Frame Control: The existing Frames Status Indicators should be utilized for Frame access control in all facilities that have critical or hazardous systems in operation. Frame Access control should be coordinated with Facility Management and potentially affected systems should be safed prior to Frame access.
2. Critical Functions: All critical or hazardous functions routed through the Comm. system should be appropriately identified and protected.
3. Documentation: Formal documentation of troubleshooting procedures should be required when work involves critical or hazardous systems or is accomplished in the vicinity of them.

1. Documentation: Systems drawings should be updated and audited to verify accuracy and completeness.
2. Control lines: Control functions which require an OPEN/CLOSED path between two wires should be designed and implemented such that exposed terminals are non-adjacent and are adequately protected. If possible separate wires should be utilized, not a twisted pair.
3. Routing: control lines for "ARM & ACTIVATE" values and other critical functions should not be routed through uncontrolled TDs and Frames along with other miscellaneous systems
4. Monitoring: Control logic should be designed such that traceability is provided to indicate how the system was activated. (i.e. which pushbutton).
5. Verification: Examine all Water Deluge Systems and identify those with conditions similar to the PHSF system so that appropriate action can be taken.
6. Oversight: Establish a fire protection systems committee consisting of representatives from Codes SI, CM, TE, DE, RT & CCAFS to provide oversight and integration of the contractors involved.

Evidence of Recurrence Control Effectiveness:

Documents Related to Lesson:

Mission Directorate(s):

• Space Operations
• Exploration Systems

Additional Key Phrase(s):

• Communication Systems
• Computers
• Configuration Management
• Emergency Preparedness
• Facilities
• Fire Protection
• Flight Equipment
• Flight Operations
• Ground Equipment
• Ground Operations
• Industrial Operations
• Mishap Reporting
• Payloads

Additional Info:

Approval Info:

• Approval Date: 2002-05-06
• Approval Name: Gena Baker
• Approval Organization: KSC
• Approval Phone Number: 321-867-4261