WARP Hardware

A WARP anomaly occurred on August 25, 2004 (Day 238) that caused the loss of 8 DCEs and 1 R/T pass (no engineering data loss).

The anomaly occurred during an Autonomous Sciencecraft Experiment (ASE) long duration test, and since ASE was hosted on the WARP (and controlling EO-1), the first indication of the problem was that no telemetry was received during the Svalbard Ground Station (SGS) pass on the above date at 09:38z. Below is an overview of that day's events:

At 238-09:38z: SGS pass – No telemetry during which a SatWatch alert was triggered.

At 238-11:00z: "Blind" TRDS pass – S/C health was OK, but the WARP was not generating any telemetry packets. Power Output Module #2 (OM-2) showed that the WARP should be powered on.

At 238-12:05z: "Blind" TDRS pass – Sent WARP "No-op" and Memory Dump commands with no response.

At 238-12:55z: SGS Pass (scheduled pass, but we had to perform a "Blind" acquisition since WARP and ASE were not responding) – Dumped VR/DS-1. Attempted to send /WARPRESET with no response. Sent /WARPPWR OFF with expected response on Power Page LOS w/S-Band transponder on.

At 238-14:32z: SGS Pass – Dumped VR/DS-0. Uplinked Automatic Time Sequence (ATS) load with S-Band passes only. Switched ATS buffers.

Telemetry playback analysis showed that the last packet generated by the WARP RSN (packet 03D) was at 238-07:01:38z and that Output Module #2 (OM-2) current dropped by about 1 amp compared to normal. No WARP activities were scheduled to occur at or near that time. At this point the prime suspect for the anomaly was the Solid State Power Controller (SSPC) and not ASE. EO-1 had a similar SSPC problem in September 2001 that caused a power cutoff to the Attitude Control Electronics (ACE).

At 238-17:47z: SGS Pass – The Engineering Team determined that the best course was to cut power to the WARP at the SSPC and attempt to reset. The reset was performed at this time and the WARP returned to standard operations.

At 238-18:43z: "Blind" TDRS pass – Final WARP configurations (Memory Scrub and EDAC log in overwrite mode). Cleared 1773 WARP and 1773 WARP/RSN Bus errors. Switched ATS buffers.

At 238-21:19:33z: Uplinked ATS load to resume normal imaging operations at the next available scene.

This anomaly was fully explained as traceable to a transient Single Event Upset failure of the WARP SSPC and was similar to the Attitude Control Electronic (ACE) latchup that occurred in September 2001. The WARP was powered down and restarted. This power reset action cleared the problem. This type of problem occurred so infrequently that no standard recovery operating procedure was needed.