Ch6-3

Computers in Spaceflight: The NASA Experience
- Chapter Six -
- Distributed Computing On Board Voyager and Galileo -

Galileo - True distributed computing in space


[186] Project Galileo began at JPL in the late 1970s with the objective of developing an orbiter and probe for further exploration of Jupiter. Galileo will proceed toward Jupiter, launching a probe 5 months before arrival. Plans are for the probe to enter the Jovian atmosphere at a relatively low angle, using an aeroshell braking system for entry followed by a parachute system for final braking and descent. Due to the nature of the entry, an antenna large enough to send data directly to earth cannot be carried on the probe. Instead, it has to relay the data to the orbiter, which will fly a parallel path thousands of kilometers above. At the end of the probe mission, expected to last 60 to 75 minutes until the probe is crushed by atmospheric pressure, the orbiter will execute an insertion burn. For the next 2 years, the orbiter will fly by the four Galilean satellites (hence, the mission name), using gravity assists to change its path after each encounter.

Great demands will be placed on Galileo's on-board computer systems, because of both the nature of the mission and the design of the spacecraft itself. First, Galileo is a one-shot mission. Interplanetary probes have been mostly launched in pairs for the obvious.....
[187]

FIGURE 6-3. The Unified Data System: precursor to Galileo's Command and Data Subsystem. (JPL Photo 360-630)

....reason that a full backup then exists. Such dual launches are generally cost effective, as a second spacecraft can be obtained for 15% of the price of the first one⁷⁶. Budget constraints forced NASA to buy just one Galileo, so there was tremendous pressure to construct a highly reliable spacecraft. Additional pressure has been on the project because of changes in the launch date and booster rocket. Originally scheduled for a 1982 launch, delays in the Shuttle program and other factors caused rescheduling to 1984, then 1985 and, finally, 1986, when the grounding of the Shuttle fleet forced an indefinite postponement. At first, the Air Force's Inertial Upper Stage rocket was chosen for the booster. Later, the new "wide body" Centaur got the job. Centaur upper stages have flown on Atlas and Titan III boosters since the 1960s. The new "fat Centaur" would carry 50% more fuel than the earlier version. Other changes were made to adapt it to the Shuttle cargo bay. One JPL engineer said that it is "like Abe Lincoln's axe. The head broke and they replaced it and the handle broke and they put on a new one, but it's still Abe Lincoln's axe"⁷⁷. However, NASA canceled the Shuttle version of the new Centaur in the spring of 1986 due to safety considerations, leaving Galileo without a ride to Jupiter. [188] By early 1987, NASA decided to go back to using the Inertial Upper Stage, but it has significantly lower lifting capability than the Centaur. As a result, the flight path has to be changed to include a Venus flyby and two Earth flybys to gain velocity by gravity assistance. Unfortunately, the total flight time to Jupiter will nearly triple to about seven years.

Spacecraft design also caused problems for the computer designers. All previous JPL probes have been three-axis inertial to gain advantages such as easing communication and providing a stable scan platform for imaging. Galileo has a fixed attitude area, called the "despun section," and also a "spun section" that rotates three times a minute. Fields and particles experimenters required a spin to help them differentiate local fields from external fields. Aside from the obvious increase in the order of magnitude of the attitude control problem on a dual-spin spacecraft, communication between the two parts is hampered by the need to transmit serially across a rotary transformer. Four hundred milliseconds are required to send a message between the spun and despun sections⁷⁸. To overcome this time penalty and provide more real-time control, a fully distributed system of computers is mandated.




FIGURE 6-4. A model of the Galileo spacecraft with the probe visible at the bottom center. (JPL photo 230-222A)

[189] Finally, computer system complexity is further increased by the number of science experiments on board and the fact that they are largely computer controlled as well. Eight of the nine instruments have microprocessors for control and data handling⁷⁹. These have to communicate with the Command and Data System, itself containing six microprocessors. Attitude and Articulation Control has dual computers, and the probe also contains a dual microprocessor system. In all, Galileo contains 19 microprocessors with about 320K of semiconductor random access memory and 41K of read-only memory⁸⁰. No unmanned spacecraft launched to date can approach Galileo in the power and size of its on-board computer network.

JPL took great care in the selection of the computer systems for Galileo. Procurement of the systems for the Command and Data handling equipment and the attitude control equipment proceeded separately but in a somewhat coordinated fashion. In the case of the Command system, a 1977 study examined using the existing Voyager computers, the National Standard Spacecraft Computer (NSSC-1), and some form of a microprocessor distributed system like the Unified Data System (UDS), then a research program at JPL aimed at complex long-duration space missions⁸¹. Although a lot of pressure was exerted on Galileo's builders to choose either the existing equipment or the NSSC-1, cost factors favored the UDS as the basis for the Command and Data Subsystem⁸². Similarly, the attitude control group was pressured to use the NSSC-1, but the desire for floating point and greater power defeated that idea. Since the star tracker is in the spun section and thus moving, complex coordinate transformations must be calculated, and the NSSC-1 was not up to it⁸³.

With new computers needed for both major controlling subsystems, JPL carefully explored memory requirements and software development prospects. Prototype programs were written in HAL/S and FORTRAN for the command computer and the attitude control computer⁸⁴. Ideas for the content of the programs came from Voyager experience and the executive written for the NSSC-1. The project office originally specified that HAL would be used for programming all flight software. When irreducible inefficiencies appeared in the compiler bought for the command and data computers, HAL was abandoned for that system and replaced with "structured macros"⁸⁵. HAL was retained for the other computer system. Although most microprocessors in the scientific experiments are coded in assembler, one is programmed in FORTH, so high-level languages finally appeared on unmanned spacecraft. The project office set a limit of 75% memory usage at launch (later raised to 85% for the attitude control computer only) and 85% load at Jupiter insertion. Officials hoped to avoid the tight memory problems associated with earlier missions and even asked the Shuttle software office for advice in setting these limits⁸⁶.

[190] JPL considered software crucial to the success of the overall Galileo mission. As of 1985, Neil Ausman of JPL was in charge of all software, both flight and ground support, and he reported directly to project manager John Casani. Patricia Molko, also of the project office, wrote a standards document for software development. At one point, Howard W. Tindall, first introduced in the chapter on the Apollo computer systems, was brought in to serve as a consultant. He found that in some ways JPL was "going through the same problems that we did when we developed our original large programs"⁸⁷. However, in some respects Galileo is more complex than Apollo because of the number of intercommunicating computer systems. Thus, size is not the only factor contributing to the difficulty of writing the software.

The Galileo probe mission is handled by NASA Ames Research Center, and the entry probe was assembled by Hughes Corporation. Even though it contains a dual microprocessor system, its function is primarily confined to sequencing, and its architecture is similar to systems already described on Mariner, Viking, and Voyager.


Galileo's Command and Data Subsystem Origins: The UDS

STAR was JPL's foray into ultrareliable computer research in the 1960s; the UDS was its 1970s counterpart. David Rennels, who had been instrumental in the STAR program, led in developing the system. Assisting him on the hardware side was Borge Riis-Vestergaard, a visiting scientist, and Vance C. Tyree. Frederick Lesh and Paul Lecoq did the software. One reason the UDS project started was the desire to develop a new architecture for flight computers that would reduce life cycle costs⁸⁸. Another impetus came from 1973 studies of distributed systems done in support of Voyager and by the Air Force⁸⁹. Distribution of functions among several computers on Voyager has been shown to be a natural result of requirements. The Air Force study found that avionics tasks are better handled by partitioning and using dedicated computers for specific functions. Since microprocessors became commercially available at about that time, they were recommended for use in such distributed systems.

Rennels' UDS project explored the difficulties in tying multiple computers together in a flexible manner. He defined an architecture using two levels of computers. Individual computers and associated memories at one level were called "high-level modules (HLM)." These computers controlled system-wide functions such as the data buses and fault detection. Other computers were located at specific subsystems and were called "terminal modules." They controlled one functional area such as engineering instruments or attitude and articulation. Each module had its own processor and memory.

[191]
FIGURE 6-5. Galileo and its booster being deployed from an orbiting shuttle. (P25722AC)

Communication between modules was accomplished on a bus that carried data from one memory to another. By using direct memory access for all intercommunication, processor resources other than for transfer commands were unaffected. HLMs did not have I/O capability other than to the terminal modules via the bus. All input and output to spacecraft systems and the ground was handled by the terminal modules.

Some influence from the STAR project can be noted in that Rennels kept critical functions highly redundant and simple. Reconfiguration after failures reflected STAR concepts⁹⁰. In order to avoid a potential single-point source of failure, there was no central bus controller in the UDS. Each of the HLMs controlled a separate bus, but [192] only one bus was needed to support all processors because any HLM could transfer data between any two memories⁹¹. The breadboard built for the UDS project had three HLMs and three terminal modules, so it had three buses as well⁹². Failure of a HLM caused its functions to be accepted by the remaining ones. Reliability is obtained by such reallocation of functions to resources, making this a highly fault- tolerant system.

One advantage of distributed systems is that interfaces can be simpler than in a system using a central computer. Each local computer is responsible for its own timing and control. Only on and off commands and data transfers need be made between machines⁹³. System-wide synchronization is accomplished by providing all processors with a real-time interrupt signal. By using a cyclic interrupt, the complexity attendant with priority interrupt systems is avoided⁹⁴. Every 2.5 milliseconds, a signal is sent to all components. Basically, every processor has to be finished with its current processes before the next interrupt occurs⁹⁵. Data transfers and scheduling of tasks can be timed using the periodic interrupt.

Key advantages of the UDS concept are that expanded requirements can be handled by adding terminal modules, software can be highly specialized and distributed, and fault tolerance is very high. Availability of flight-capable microprocessors in the mid-1970s made it possible for JPL to seriously consider the UDS as a competitor with NSSC-1 and the old Voyager equipment. Its flexibility and potential as a permanent architecture for space flight helped its case.

NASA chose RCA's 1802 microprocessor for the Galileo implementation of the UDS. A CMOS-type device, it was "nobody's favorite choice"⁹⁶ but at the time (c. 1977) was considered to be the only microprocessor suitable for spaceflight. Recall the use of CMOS components in the processors and memory in the Voyager Flight Data System. Similar advantages accrue with the use of CMOS microprocessors: low power requirements (30 milliwatts) and tolerance of a wide range of voltages⁹⁷. However, some disadvantages had to be dealt with. CMOS chips are especially susceptible to damage from electrostatic discharges⁹⁸. RCA 1802s are slow, with a 5-microsecond cycle time and an average of two cycles per instruction. In contrast, the discrete component Voyager CCS had a 1.37-microsecond cycle, making it faster for functions that did not require multiple cycles⁹⁹. Speed has been a major constraint to the software development^l00. Carryover of the direct memory access cycle to Galileo from the Voyager Flight Data System alleviates this problem somewhat¹⁰¹. In general, making the six microprocessors "come together" has been much more difficult than originally expected¹⁰². Software is the most important component in achieving the success of the CCS.


[193]

FIGURE 6-6. Circuit boards for the Galileo Command and Data Subsystem. (JPL Photo 360-1756)

Box 6-4: Evolution of the Command and Data Subsystem; Galileo's Command and Data Subsystem adapted UDS technology fairly directly. In 1978 designs, the Subsystem was shown as consisting of three HLMs and four low-level modules (LLMs), which were the realization of the tensional modules. Three buses were also present, each controlled by a HLM. Functionally there was one HLM dedicated to stored sequence control, one for real-time control, and the third as a spare. LLMs in that configuration handled sequencing, telemetry. Status polling, and other subfunctions. Eventually one high-level processor was eliminated, but three buses remain, though one of them is used for test equipment only and will not function in flight¹⁰³. Software architecture is now much different than the UDS.; [194] One major difference from the UDS concept is the way the processors in the Subsystem are separated into redundant strings. Whereas reconfiguration after a failure was done by combining any of the remaining processors into a new control string in the UDS, on Galileo two basically identical strings are configured from the start, one backing up the other much like the backup processors on Viking and Voyager. Each HLM has 32K of memory and a bus controller associated with it. A LLM with a processor and 16K of memory for engineering control is connected to the string, along with a data bulk memory (DBUM) of 8K and a bulk memory (BUM) of 16K. These components are in the spun section of the spacecraft. Another LLM with 16K of memory is in the despun section, connected to the probe and the launch vehicle, among other functions^104. This configuration of one HLM and two LLMs, a BUM and a DBUM is repeated in string B. Therefore, the total system consists of six microprocessors with 176K of semiconductor memory. About 12K of HLM memory is write protected and used for programs¹⁰⁵. BUMs are used for auxiliary storage and buffering in that all new sets of commands are directly inserted into them from the ground and then redistributed by the software in the HLMs. Data memories serve as buffers for incoming science instrument data, again with direct memory access¹⁰⁶. Commands and data are transferred on the buses using packets with three-word headers. Headers contain the code numbers of the source and recipient, the starting address in memory of the message, and fillers for timing. More than one address can be specified for a message, but usually there is only one recipient¹⁰⁷. Data transfer is coordinated by the real-time interrupt. Odd-numbered real-time intervals are used for input; even numbered intervals for output.; As in previous missions, several operating modes are available for the Command and Data Subsystem. During cruise and other noncritical mission phases, one string is up and running and the other is in a quiet state. Otherwise both can be commanding in one of several ways. Dual string mode means that the strings are executing code concurrently and both send commands. Parallel mode is used for time-critical operations needing closer synchronization. Tandem mode is used during maneuvers. If a failure is detected in one string, the other halts the activity¹⁰⁸.


Developing the Command and Data Subsystem Software

[194] Development of the software for this Subsystem consumed more time and labor than any previous unmanned spacecraft. Dr. John Zipse, the Cog Engineer for the Subsystem software, had an average....
[195]


Figure 6-7. Memory modules for the Galileo Command and Data Subsystem. (JPL Photo 360-1704)
.....of 12 full-time software developers working under him at peak periods¹⁰⁹. They shared five terminals hooked to an IBM System 370/158 computer on which the assembler and functional commands resided. Originally, HAL/S was specified as the programming language for the Subsystem¹¹⁰. A prototype compiler for the RCA 1802s was not successful, and HAL was dropped in favor of "structured macros"¹¹¹. Called "functional commands" in the software documentation, they have names such as IF, ELSE, DO, ASSIGN, and others very similar to the statements of a high-level programming language¹¹². These functional commands make up the "Virtual Machine Language" in which most of the software was written. Each command causes the execution of a prepared block of 1802 assembly code, much like a subroutine call. Project documents recognize three layers of language associated with the Subsystem: Level A is the hardware external to the 1802s that may provide input and receive output, Level B is the 1802 assembler, and Level C is the Virtual Machine Language¹¹³.


Recognizing the complexity of the software, JPL instituted ever.....

[196]

FIGURE 6-8. The software architecture of the Galileo Command and Data Subsystem High Level Modules. (From JPL, 625-340-006000)

....more stringent development requirements. Preparation for the development process began with a "Galileo

Software Thinking Group" which met in 1977-1978¹¹⁴. Programmers were ordered to keep software modules smaller than 150 assembly language statements and were reminded that simplicity was the highest priority¹¹⁵.


: [197] Box 6-5: Command and Data Subsystem Software Architecture; Galileo's Command and Data software is considered a "hierarchical software architecture" and is divided into two sets of processes, with further divisions within them. Foreground processes are executed at each real-time interrupt, or every 1/15th second. They are a self test, a clock, and bus control. Background processes begin each 2/3 second, and are much more complex. Functions done in the background have been divided into six virtual machines," a term of many meanings. In this case, three virtual machines are considered "privileged": the administration machine, the contingency action program machine, and the fault processing machine. These three machines consist of software that is always resident in the Galileo computers and is kept in the 12K of write-protected memory^116. They are called privileged because the nonprivileged machines can be canceled if they do not complete processing before the end of the 2/3-second cycle, whereas the privileged machines can never be canceled. Nonprivileged machines include an immediate action program machine, a delayed action program machine, and a stored sequence program machine. So the nonprivileged machines are reserved for commands and sequence control, whereas the privileged machines are for executive and fault detection and correction. Nonprivileged software is to be updated about once a week in flight¹¹⁷. Originally, the immediate action programs were considered privileged, but with the addition of a contingency machine they were moved to nonprivileged status¹¹⁸. Software developers imagine that a "wall" exists between the privileged and nonprivileged machines. They consider that the nonprivileged software is more error prone because it is constantly changing, whereas the privileged software should have had a thorough exercise over several years in testing before the flight.; Execution of the virtual machine software is related to the 2/3-second interrupt. In each cycle the software goes through each virtual machine pending program stack and executes what is waiting. Many programs can be running in each virtual machine in each cycle, up to 10 in the administration and fault protection machines, for example. As mentioned above, the privileged machines always get to clear their pending programs, whereas the nonprivileged machines do what they can until the time is up.; Software described so far is resident in the HLMs. LLMs have specialized software for their particular functions, such as monitoring engineering instrumentation and talking to the launch vehicle. Data from those tasks needed by the virtual machines are passed on the buses during the 1/15-second interrupt.


[198] Design of the software was done in a JPL-developed Software Design and Development Language that used statements similar to those in high-level languages¹¹⁹. Even with excellent documentation and tools, such as a hardware-based simulator for software validation ¹²⁰, it takes (according to Zipse) a new programmer three months to be effective. Until Galileo has flown, no final evaluation can be made of the virtual machine architecture. Yet, future spacecraft requiring expandability and a high degree of flexibility could probably gain from using such an architecture as a complement to the UDS type hardware structure.


Galileo Attitude and Articulation Control Computer System

In terms of tasks, the Attitude and Articulation Control System has less to do than the Command and Data Subsystem, but it must perform its jobs faster and with more critical tolerances. During the discussion of the Voyager computers, it became clear that the attitude control system needed a fast-cycle, real-time software architecture running on a high-speed computer. Galileo's control requirements are much greater than Voyager's; the dual spin problem and more complex imaging equipment indicated from the beginning a need for a completely new computer system. The computer was to provide starbased attitude determination and control an inertially referenced, target-body-tracking scan platform¹²¹. Speed was the primary criterion for the new processor¹²².

Kenneth Holmes, in charge of looking for the Galileo control computer¹²³, and the other engineers ran old Voyager attitude control programs on several processors. One of those processors soon proved itself superior: Itek's 2900 series¹²⁴. Itek, now a-division of Litton Industries, built a computer known as the ATAC, or Advanced Technology Airborne Computer. Using 2900 series processors, each with 4-bit words, Itek assembled a 16-bit, low-power, flying minicomputer roughly equal in power to a Digital Equipment Corporation PDP-11/23. Navy aircraft use this computer, although its specific applications are classified¹²⁵. ATAC's basic cycle time is 250 nanoseconds, or more than five times faster than the Voyager computer's cycle. However, the memory cannot cycle faster than 2 microseconds, so operations rates average 143,000 cycles per second¹²⁶. Floating-point capability is a plus, and since it handles eight interrupts using microcode, there is no software overhead for real-time operation¹²⁷. Another good feature is that its 16 registers are general purpose; none are dedicated as accumulators, program counters, address registers, and so on. Therefore, multiprocessing is made much easier. Further advantages to the computer are that special.....

[199]

FIGURE 6-9. The central processor and input/output circuits of the Galileo Attitude and Articulation Control Subsystem. (JPL 230-1128bc)

....instructions can be added by the user for specific applications. Four instructions added by the Galileo project saved over 1,500 words of code in the flight program¹²⁸.

The ATAC came with a considerable amount of software support. Target compilers were available for FORTRAN, BASIC, and HAL/S¹²⁹. Galileo project management wanted a higher order language used in the coding, so HAL was adopted for the attitude control system. Unlike the Command and Data Subsystem, HAL was successfully adapted to the ATAC. Compilers developed by Intermetrics for the ATAC are about 12% speed inefficient, but 36% memory inefficient compared with assembler¹³⁰. Apparently this was within acceptable limits, and the flight applications code is written in HAL, with the operating system in assembler. Edward H. Kopf, Jr. said, "We love HAL/S; we could never do it without HAL/S," even though he referred to it facetiously as "flight PL/I." Given the complexity of the resultant software, he was probably right that a high-level language was critical to success.


Attitude Control Electronics Software Organization

Implementing HAL/S on the ATAC involved creating a special operating system. Ted Kopf wrote GRACOS, or the Galileo....

[200]

FIGURE 6-10. Memory modules of the Galileo attitude control computers. (JPL 230-11216c)
.....Real-time Attitude Control Operating System, to accomplish that implementation¹³⁵. GRACOS begins the operation of the software by scheduling a HAL/S module called STARTUP. Within STARTUP are statements that set up the concurrent processes necessary to do the attitude control and articulation tasks (a version of STARTUP is reproduced in the HAL/S appendix as an example of the language). Up to 17 concurrent processes may be running under GRACOS, with at least 10 up all the time¹³⁶. STARTUP is given as much time as necessary to complete, and then the established processes begin at the next 1/15-second real-time interrupt¹³⁷. Kopf wrote GRACOS to be mission independent. He avoided constraining the timing of rate groups and other things that would have been too specific¹³⁸. Interestingly, if the fault-handling routine has to come in, it restores the registers of the failed HAL/S module where the fault occurred and tries again, very similar to the "roll back and try again" scheme in STAR¹³⁹.

Sanford M. Krasner, the Software Cog Engineer for the Attitude Control Electronics, reported directly to Brian T. Larman, the Spacecraft Flight Software System Engineer. Early in the project, five people were full time on the software development, several more than on Voyager¹⁴⁰. The coding process was speeded up when HAL was used as the software design language¹⁴¹. To make development easier, Voyager structure was adopted whenever possible¹⁴²

[201] Box 6-6: Configuration of the Attitude Control Electronics; Galileo's Attitude and Articulation Control Subsystem necessarily has parts in both the spun and despun sections of the spacecraft. Most components are in the spun section, including the two redundant processors and 64K of memory. Called the ACE, for Attitude Control Electronics, its despun partner is the DEUCE, for Despun Section Electronics, a long way to go for an acronym. Communication between the ACE and DEUCE is across the rotary transformer. Since the transformer is not considered fully reliable, input or output to the DEUCE is not complete until an ACKNOWLEDGE interrupt reaches the ACE¹³¹. CMOS-type memories similar to those used in the Voyager flight Data System are in the ACE. Sixty-six 1K chips are needed, two of which are actually permanent read-only memory. Those two contain the Memory Loss Recovery Routine written in ATAC assembler¹³². A 5-volt keep-alive current directly from the generators is constantly fed to the memories, but in the case of a destructive transient the Recovery Routine can restart the software after it has been repaired or replaced.; Planning for the system included careful memory sizing. Based on actual Voyager programs and extrapolations from them to handle the new requirements, a 1978 study thought 10K of memory to be sufficient. Using a policy of 100% margin and 75% limit at launch, 32K was eventually bought¹³³. As noted above, a waiver to 85% full at launch was given. Ground computers can reprogram the ACE in flight by sending code to the off-line memory through the Command and Data Subsystem. As with its own LLMs and as pioneered in the UDS, the command computers can directly access the ACE memory. Commands can be placed in the active memory where they are "discovered" by the ACE software¹³⁴.



Ironically, the advanced nature of the new attitude control system and its control computer made it more vulnerable to space conditions than its predecessors. A potential disaster was averted when the "single event upset" was discovered and dealt with before launch.


The Single Event Upset Problem


Space environments are much harsher to electronics than the surface of the earth. Since circuitry essentially consists of hardware that moves electrons, creates and destroys magnetic fields, and emits waves of electromagnetic radiation, fields and particles of the types loose in space can affect the operation of electronic equipment. Ironically, the miniaturization of components has made electronics more sensitive to interference. One possibility that concerns computer [202] designers is the effect of highly charged particle impacts on memory cells. If a particle has sufficient energy to change the information stored in a bit, it can affect the software in a potentially disastrous way. Such a spurious change is called a single event upset (SEU). Sufficient numbers of particles can cause so many bits to change states that the software fails. Since primary and backup memories are equally vulnerable, simple redundancy is not a solution. Error-correcting codes that test for random bit flips exist but require storage and processing time not always available on a spacecraft. Additionally, bits in the processor can be affected during execution, so the problem is not limited to memory.

Galileo's processors and memories were chosen in 1977. Voyager had not yet reached Jupiter, so hardware decisions were based on 1973-1974 Pioneer data¹⁴³. Nothing was known about SEU vulnerability, so no space for error detection and correcting codes and no provision for special shielding was made. Some incorrect imaging commands sent by sequencers in the Pioneers were later tagged as SEUs. Voyager's clocks were slowed by Jovian radiation so that the computers were forced out of synchronization occasionally¹⁴⁴. By 1980-1981, the nature of the SEU problem became apparent. Sulfur ions from Jupiter's volcanic moon, Io, were being whipped up to high energy by the Jovian gravity. In 1982, Galileo Project Chief Engineer B. Gentry Lee was assigned the job of determining how bad the SEU problem could be and finding a solution. Lee arranged for cyclotron tests at the University of California's Berkeley campus in which computer and other electronic parts were submitted to bombardment by high-speed particles. Results indicated that the 2901 chips used in the attitude control computers were highly SEU sensitive, with 20% 50% of hits causing probable software failures¹⁴⁵. RCA 1802s used in the Command and Data Subsystem were actually much less sensitive, being of older, and thus less dense, technology.

Attitude control engineer Kopf commented, "It is not worth flying the mission if you cannot get rid of the SEU problem." Failures were most likely at the most critical part of the probe mission when the orbiter is very near Jupiter. In order to avoid possible further delays in an already much postponed mission, Lee searched for solutions along two tracks. One solution would use a radiation-hardened processor built by Trecor called the RHEC-Rad Hardened Emulating Computer-1750A. Even though it is an emulator capable of imitating the 2901, a new retargeted HAL compiler would be needed. The cost of this solution would be $20 million. Lee's second solution was to contract with Sandia National Laboratories to custom make radiation-hardened 2901s. No software needed be changed, just new ICs were necessary, and they cost $5 million. Due to cost considerations and the inherent attraction of retaining the already created and largely validated software, the Sandia solution was chosen¹⁴⁶. As a footnote, [203] it is interesting that if the Galileo had launched on time, a sufficient understanding of the SEU problem would not yet have been available, and a doomed spacecraft carrying an unknown time bomb would have been traveling toward an unfriendly Jupiter waiting to hurl ion thunderbolts at it.