Concerns Over High Performance
Computer Exporters' Ability to Review End-Users in the PRC
Prompted the Requirement for Prior Notification
The January 1996 revisions to the Export Administration Regulations
governing HPCs made several other important changes. Most importantly,
they made exporters responsible for determining whether an export
license is required, based on the MTOPS level of the computer,
and for screening end users and end uses for military or proliferation
concerns.134
Thus, U.S. companies that wish to export HPCs are now authorized
to determine their own eligibility for a license exception.135
Prior to this change, only U.S. HPC exports to Japan were
allowed without an individual license. At that time, a violation
of the Export Administration Regulations could be identified
by an export of an HPC that occurred without a license.
Since the change, in order to prove a violation of the regulations,
the Commerce Department must demonstrate that an exporter improperly
used the Composite Theoretical Performance license exception
and knew or had reason to know that the intended end user would
be engaged in military or proliferation activities.136
Also, the revised Export Administration Regulations required
that exporters keep records and report to the Commerce Department
on exports of computers with performance levels at or above 2,000
MTOPS. In addition to existing record-keeping requirements, the
regulations added requirements for the date of the shipment,
the name and address of the end user and of each intermediate
consignee, and the end use of each exported computer. Although
these records have been reported to the Commerce Department on
a quarterly basis for the past two years, some companies have
reported inconsistent and incomplete data for resellers or distributors
as end users.137
Since U.S. HPCs obtained by countries of proliferation concern
could be used in weapons-related activities, the Congress enacted
a provision in the Fiscal Year 1998 National Defense Authorization
Act138 that required exporters to notify the Commerce Department
of all proposed HPC sales over 2,000 MTOPS to Tier 3 countries.
The Act gives the U.S. Government an opportunity to assess these
exports within 10 days and determine the need for a license.
Following such notification, the Departments of Commerce, State,
Defense, and Energy, and the U.S. Arms Control and Disarmament
Agency, can review a proposed HPC sale and object to its proceeding
without an export license. The Commerce Department announced
regulations implementing the law on February 3, 1998.139
A November 1998
Defense Department study, however, identified potential problems
with the 10-day notification procedure. The study noted that
the Defense Department provides comments on export notices referred
to it regarding those end users for which the Defense Department
has information. The study also noted that:
The operating assumption is that, if there is no information
on the end-user, then the end-user is assumed to be legitimate.
This is probably true in most cases; however, there is no means
to verify that high performance computers are not making their
way to end-users of concern to the United States.140
Furthermore, the Defense Department study expressed concern
that foreign buyers might circumvent current Export Administration
Regulations provisions requiring attestation to the buyer's knowledge
that the export will have no military or proliferation end user
or end use.141 By designating a company in the United States
to act on its behalf, the foreign company could have its U.S.
designee submit the HPC notification to the Commerce Department;
the U.S. designee and not the foreign buyer would then be responsible
for all compliance with notification procedures.142 The U.S.
designee would be responsible only for shipping the item and
would not take title of the item.143
Under the Export Administration Regulations, the U.S. designee
could complete the notification to its knowledge, which might
be useless if the U.S. designee is in fact ignorant of the actual
end use. The Defense Department study noted the obvious problems
with this system.
The study also observed that the 10-day notification period
was insufficient to ensure that U.S. designees and foreign buyers
are providing accurate and complete information.144
Finally, the Defense Department study warned that foreign
buyers of U.S. computer technology might circumvent the notification
procedure by notifying the Commerce Department that they are
purchasing a system that is not above the 7,000 MTOPS threshold,
but later upgrading the system with processors that are below
the 2,000 MTOPS level. There would be no requirement to notify
the Commerce Department of the acquisition of the lower than
2,000 MTOPS upgrades to the previously-notified system.145
The
U.S. Government Has Conducted Only One End-Use Check for High
Performance Computers in the PRC
The Fiscal 1998 National Defense Authorization Act now requires
the Commerce Department to perform post-shipment verifications
on all HPC exports of HPCs to Tier 3 countries with performance
levels over 2,000 MTOPS.146
Post-shipment verifications are important for detecting and
deterring physical diversions of HPCs, but they do not always
verify the end use of HPCs.147
The PRC traditionally has not allowed the United States to
conduct post-shipment verifications, based on claims of national
sovereignty, despite U.S. Government efforts since the early
1980s.148 This obduracy has had little consequence for the PRC,
since HPC exports have continued to be approved and, in fact,
have increased in recent years.
In June 1998, the PRC agreed with the United States to cooperate
and allow post-shipment verifications for all exports, including
HPCs.149 PRC conditions on the implementation of post-shipment
verifications for HPCs, however, render the agreement useless.150
Specifically:
· The PRC
considers requests from the U.S. Commerce Department to verify
the actual end-use of a U.S. HPC to be non-binding
· The PRC
insists that any end-use verification, if it agrees to one, be
conducted by one of its own ministries, not by U.S. representatives
· The PRC
takes the view that U.S. Embassy and Consulate commercial service
personnel may not attend an end-use verification, unless they
are invited by the PRC
· The PRC
argues scheduling of any end-use verification - or indeed, whether
to permit it at all - is at the PRC's discretion
· The PRC
will not permit any end-use verification of a U.S. HPC at any
time after the first six months of the computer's arrival in
the PRC
The Select Committee has reviewed the terms of the U.S.-PRC
agreement and found them wholly inadequate. The Clinton administration
has, however, advised the Select Committee that the PRC would
object to making the terms of the agreement public. As a result,
the Clinton administration has determined that no further description
of the agreement may be included in this report.
According to Iain S. Baird, Deputy Assistant Secretary of
Commerce for Export Administration within the Bureau of Export
Administration, post-shipment verifications are conducted by
the PRC's Ministry of Foreign Trade and Economic Cooperation
for U.S. computers having over 2,000 MTOPS that are exported
to the PRC. He says such verifications are done in the presence
of the U.S. commercial attaché.151
Commerce reported on November 17, 1998, that no post-shipment
verifications would be performed on HPCs that were exported to
the PRC from November 18, 1997 through June 25, 1998 because
the PRC/U.S. agreement applies only prospectively from June 26.
Since June 26, the
Commerce Department reported, only one post-shipment verification
has been completed and one was pending as of November 12, 1998.
Commerce also stated that "Post shipment verifications were
not done on most of the others [HPCs] because the transactions
do not conform to our arrangement with the PRC for end use checks."152
Thus, post-shipment verifications will not be done on any
HPCs exported to the PRC prior to the agreement, nor on any HPCs
shipped that are exported in the future under the Composite Theoretical
Performance license exception (that is, those between 2,000 and
7,000 MTOPS) to civilian end users.
According to Commerce Department Under Secretary for Export
Enforcement William Reinsch, a pending regulatory change will
instruct HPC exporters to seek end-use certificates from the
PRC Government. Where PRC end-use certificates are obtained,
this regulation purportedly would allow more post-shipment verifications
to be requested consistent with the PRC-U.S. agreement.153
Reinsch stated that the PRC has indicated that it would be
willing to issue end-use certificates. However, the PRC office
in question reportedly has a staff of five, which would severely
limit the number of post-shipment verifications it could implement.154
According to a September 1998 report from the General Accounting
Office, U.S. Government officials agreed that the manner in which
post-shipment verifications for computers traditionally have
been conducted has limited their value because they establish
only the physical presence of an HPC, not its actual use. In
any event, according to national weapons laboratory officials
within the Energy Department, it is easy to conceal how a computer
is being used.155
Even when U.S. Government officials perform the post-shipment
verification, the verifying officials have received no specific
computer training and are capable of doing little more than verifying
the computer's location. It is possible to verify an HPC's use
by reviewing internal computer data, but this is costly and intrusive,
and requires sophisticated computer analysis.156
The General Accounting Office report also noted that the U.S.
Government makes limited efforts to monitor exporter and end-user
compliance with explicit conditions that are often attached to
HPC export licenses for sensitive end users. The U.S. Government
relies largely on the HPC exporters to monitor end use, and may
require them or the end users to safeguard the exports by limiting
access to the computers or inspecting computer logs and outputs.157
The end user may also be required to agree to on-site inspections,
even on short notice, by the U.S. Government or exporter. These
inspections would include review of the programs and software
that are being used on the computer, or remote electronic monitoring
of the computer.158
Commerce officials stated to GAO that they may have reviewed
computer logs in the past, but do not do so anymore, and that
they have not conducted any short-notice visits. They also acknowledged
that they currently do not do any remote monitoring of HPC use
anywhere and that, ultimately, monitoring compliance with safeguards
plans and their conditions is the HPC exporter's responsibility.159
Some
U.S. High Performance Computer Exports
to the PRC Have Violated U.S. Restrictions
During the 1990s, there have been several cases of export
control violations involving computer technology shipments to
the PRC. One ongoing case concerns the diversion of a Sun Microsystems
HPC from Hong Kong to the PRC.160
On December 26, 1996, a Hong Kong reseller for Sun Microsystems,
Automated Systems Ltd., sold an HPC to the PRC Scientific Institute,
a technical institute under the Chinese Academy of Sciences -
a State laboratory specializing in parallel and distributed processing.
At some point after the sale but before delivery, the computer
was sold to Changsha Science and Technology Institute in Changsha,
Hunan Province. The machine was delivered directly to that Institute
in March 1997.161
Automated Systems of Hong Kong claimed to Sun officials in
June 1997 that it had understood that the Changsha Institute
was "an educational institute in Wuhan Province providing
technological studies under the Ministry of Education."
The end use there, according to Automated Systems, was to be
for "education and research studies in the college and sometimes
for application development for outside projects." Sun was
recommended to contact the end user, the Changsha Institute,
for more specific end-use information.162
The HPC sale came to the attention of the Deputy Assistant
Secretary for Export Enforcement, Frank Deliberti. He queried
the U.S. Embassy in Beijing about the Changsha Institute. Deliberti
gave the information he obtained to Sun Microsystems, which then
initiated efforts to have its computer returned.163
During the same period, the Foreign Commercial Officer at
the U.S. Embassy in Beijing consulted his contacts at the PRC's
Ministry of Foreign Trade and Economic Cooperation. The Ministry
denied that the Changsha Institute was affiliated with the PRC
military.164
Subsequently, the
Ministry called the FCO to inform him that the actual buyer of
the computer was an entity called the Yuanwang Corporation, and
that Sun Microsystems had been aware of this corporation's PRC
military ties. Reportedly, Yuanwang is an entity of the Commission
on Science, Technology, and Industry for National Defense (COSTIND).
So far as the PRC's Ministry of Foreign Trade and Economic Cooperation
reportedly could determine, the end-use statements that had been
provided to Sun through Automated Systems of Hong Kong were totally
fictitious. The Changsha Science and Technology Institute, according
to the Ministry, did not exist.165
The official position of the Ministry of Foreign Trade and
Economic Cooperation was that the PRC Government would not help
to obtain the return of the computer. The role of the PRC Government,
the Ministry asserted, had been merely to help two private parties
rectify a misunderstanding. In any event, the computer was returned
to the United States on November 6, 1997.166 The Commerce Department
investigation reportedly is continuing.167
A number of other violations of U.S. laws and regulations
concerning computers exported to the PRC have been investigated
by the Commerce Department:
New World Transtechnology
On December 20, 1996, New World Transtechnology of Galveston,
Texas, pled guilty to charges that it violated the export control
laws and engaged in false statements by illegally exporting controlled
computers to a nuclear equipment factory in the PRC in August
1992. The company was also charged with attempting to illegally
export an additional computer to the PRC through Hong Kong in
October 1992. The company was sentenced to pay a $10,000 criminal
fine and a $600 special assessment fee.168
Compaq Computer Corporation
On April 18, 1997, the Commerce Department imposed a $55,000
civil penalty on Compaq Computer Corporation of Houston, Texas,
for alleged violations of the Export Administration Regulations.
The Commerce Department alleged that, on three separate occasions
between September 17, 1992 and June 11, 1993, Compaq exported
computer equipment from the United States to several countries,
including the PRC, without obtaining required export licenses.
Compaq agreed to pay the civil penalty to settle the allegations.169
Digital Creations
On June 12, 1997, Digital Creations Corporation of Closter, New
Jersey, was sentenced to pay an $800,000 criminal fine for violating
the Export Administration Act and Regulations in connection with
exports of computers to the PRC. Digital had previously pled
guilty in December 1994 to charges that it had violated the Export
Administration Regulations by illegally exporting a Digital Equipment
Corporation computer to the PRC without obtaining the required
export license.170
Lansing Technologies Corporation
On June 17, 1997, Lansing Technologies Corporation, of Flushing,
New York, pled guilty to charges that it violated the Export
Administration Regulations in 1992 by exporting a Digital Equipment
Corporation computer vector processor and a data acquisition
control system to the PRC without obtaining the required export
licenses from the Commerce Department.171
Other serious violations of HPC export control laws and regulations
have occurred in recent years, but these concerned Russia. On
July 31, 1998, for example, the Department of Justice announced
that IBM East Europe/Asia Ltd. entered a guilty plea. IBM received
the maximum allowable fine of $8.5 million for 17 counts of violating
U.S. export laws through the sale of HPCs to a Russian nuclear
weapons laboratory known as Arzamus-16. In another example, an
ongoing U.S. Government investigation of Silicon Graphics Incorporated/Convex
is examining whether a violation of law occurred in a sale of
HPCs to another Russian nuclear weapons laboratory, Chelyabinsk-70.172
High Performance Computers
at U.S. National Weapons Laboratories
Are Targets for PRC Espionage
No other place in the world exceeds the computational power
found within the U.S. national weapons laboratories. For this
reason, both the computational power and the data it can generate
have been the focus of the PRC's and other countries' intelligence
collection efforts.
The desire for access to this computing power and data, in
turn, is one of the reasons so many foreign nationals want to
visit the laboratories.
According to David Nokes, the network administrator at Los
Alamos National Laboratory, all operating systems have vulnerabilities
that can be exploited by a knowledgeable, valid user.173 Nokes
also says that there are a few solutions to issues of HPC network
security. These include:
· Allowing
only U.S. students to use the networks
· Limiting
physical access to high performance computer networks at universities
· Enhancing
physical security and security education at universities174
U.S.
National Weapons Laboratories Have Failed
to Obtain Required Export Licenses for
Foreign High Performance Computer Use
When foreign nationals use the U.S. national weapons laboratories'
HPCs, their activities should generally be considered "deemed
exports." The "deemed export" rule [15 CFR 734.2
(b) (ii)] covers those situations in which an export-controlled
technology or software-source code information is released to
a visiting foreign national, for which a license would have been
required. In such situations, an "export" is "deemed"
to have occurred.
The Select Committee is concerned that HPC system managers
in the U.S. national weapons laboratories lack an essential understanding
of the deemed export rule. This lack of understanding was substantiated
by interviews with representatives from the Department of Commerce
who had no recollection of ever having seen an application for
a deemed export from any of the U.S. national weapons laboratories.
When PRC nationals visit and use the HPCs at a U.S. national
weapons laboratory, their access should be limited to the same
computing capabilities to which the PRC itself is restricted,
especially for military uses.175 The Select Committee discovered,
however, that the laboratories do not even measure the computational
power of their HPCs in MTOPS. Moreover, many of the laboratories
have difficulty in converting to MTOPS from the units they use
to measure the power of an HPC.
The Department of
Commerce could not recall a laboratory ever having sought guidance
on how to compute an HPC's MTOPS rating. Significantly, the
Select Committee discovered that a rather modest HPC (by Department
of Energy standards) in a U.S. National Laboratory used by foreign
nationals had a substantially higher MTOPS rating than the controlled
threshold. No licenses, however, had ever been obtained.
The "deemed export" rule also applies in those instances
in which a PRC national or entity accesses an HPC remotely via
the Internet.
In the absence of an effective audit system, which monitors
the codes being run by the PRC user, the U.S. national weapons
laboratories cannot verify that they are in compliance with the
law, or that PLA or PRC intelligence is not using the HPCs for
the design or testing of nuclear or other weapons.
PRC
Students Have U.S. Citizen-Like Access To High Performance Computers
at the National Weapons Laboratories
The U.S. national weapons laboratories rely upon nuclear weapons
test simulation software and computers provided by the Accelerated
Strategic Computer Initiative (ASCI). Five major U.S. universities
support ASCI through the Academic Strategic Alliances Program
(ASAP).
As a result, hundreds of research students and staff at these
universities have access to the HPCs used by the national weapons
laboratories for U.S. nuclear weapons research and testing. As
many as 50 percent of these research students and staff are foreign
nationals, some of whom may have foreign intelligence affiliations.
Holders of Immigration and Naturalization Service "green
cards" - PRC nationals who have declared their intent to
remain permanently in the U.S. - are treated as U.S. citizens
for export control purposes. They are then given U.S. citizen-like
HPC access, free to return to the PRC once their objectives are
fulfilled.
In November 1998, the Secretary of Energy issued an Action
Plan that includes a task force to review HPC usage by foreign
nationals and provide a report to the Secretary within six months.
The Department of Energy is currently preparing an implementation
plan to address counterintelligence issues identified in a July
1998 report, entitled "Mapping the Future of the Department
of Energy's Counterintelligence Program," including HPC
usage by foreign nationals.
Many Types of Computer Technology
Have Been Made Available to the PRC
That Could Facilitate Running Programs
Of National Security Importance
One of the bases for the 1996 increase in export control thresholds
was that individual PCs were widely available on the open market
in the United States, but not able to be exported to the potentially
huge PRC market.176 What was an HPC in 1993 (those capable of
195 or more MTOPS) was no longer even considered necessary to
control for weapons proliferation concerns.177
By 1997, PCs and workstations assembled in the PRC captured
approximately 60 percent of the PRC's domestic market.178 All
of these locally-assembled computers used imported parts - over
70 percent contained United States-produced Pentium microprocessors.179
Three of the largest manufacturers in the PRC were affiliates
of IBM, Hewlett Packard, and Compaq, with a combined market share
of approximately 21 percent.180 A large share (but probably not
more than 20 percent) of the PC assembly in the PRC was done
by small, independent assembly shops.181
The largest individual producer of PCs and workstations in
the PRC is the Legend enterprise, a spin-off of the Chinese Academy
of Sciences.182 This domestic computer assembly industry dovetails
well with Beijing's overall plans for economic modernization.
Beijing reportedly desires an independent PRC source of most
high-technology items to avoid reliance on foreign providers
for these goods.
To participate more
fully in the PRC market, United States firms have been pressured
by the PRC government to relinquish technological advantage for
short-term market opportunities. The PRC requires that foreign
firms be granted access to the PRC market only in exchange for
transferring technology that would enable the state-run enterprises
to eventually capture the home market and begin to compete internationally.
However, the PRC's strategy of coercing technology from foreign
firms has not enabled state-run industries to close the technology
gap with more developed nations. In the context of establishing
domestic production of computers for sale in the PRC, this PRC
"technology coercion" policy appears to have worked.183
The PRC now has a growing industrial base of small computer assemblers.
For the most part, these companies are not State-run. The technology
that was "coerced" from U.S. computer manufacturers
as a cost of entering the PRC market apparently better serves
the expansion needs of small, relatively independent enterprises
and not the intended needs of central planners in Beijing.
90 percent of PRC consumers of PCs and workstations are business,
government, and educational entities, with individual purchases
accounting for only 10 percent of the PRC's PC market.184 To
illustrate the size of the individual purchaser segment of the
PRC's market, it is estimated that only 5 million individuals
out of the PRC's 1.2 billion have the expendable funds required
to purchase a low-end PC in the PRC.185
Despite the limited number of individual purchasers, the actual
size of the PRC PC and workstation market was 2.18 million units
in 1996; 3 million units in 1997; and 4.5 million units in 1998.
It is anticipated the PRC PC and workstation market will grow
at the rate of 1.5 million to 2 million units per year through
the year 2000. According to figures provided by the Asia Technology
Information Project, an independent research foundation, non-PRC
manufacturers of PCs and workstations, including U.S. manufacturers,
could expect to partake of a portion of the almost 2 million
units expected to be imported for sale in the PRC in 1998.186
The PRC Has a Limited Capability
to Produce High Performance Computers
The PRC has demonstrated the capability to produce an HPC
using U.S.-origin microprocessors over the current threshold
of 7,000 MTOPS. The PRC "unveiled" a 10,000 MTOPS HPC
- the Galaxy III - in 1997 based on Western microprocessors.
But PRC HPC application software lags farther behind world
levels than its HPC systems. Also, despite the existence of a
few PRC-produced HPCs based on Western components, the PRC cannot
cost-effectively mass-produce HPCs currently. There really is
no domestic HPC industry in the PRC today.
While it is difficult to ascertain the full measure of HPC
resources that have been made available to the PRC from all sources,
available data indicates that U.S. HPCs dominate the market in
the PRC.187
Although the PRC has a large market for workstations and high-end
servers, there is a smaller market for parallel computers which
is entirely dominated by non-PRC companies such as IBM, Silicon
Graphics/Cray, and the Japanese NEC. However, there continues
to be significant market resistance to Japanese HPC products
in Asia, especially as U.S. products are beginning to have significant
market penetration.188
U.S. High Performance Computer
Exports
To the PRC Are Increasing Dramatically
A review of Commerce Department information regarding the
total of HPC license applications that were received for the
time frame January 1, 1992 to September 23, 1997, revealed the
following:
· Only one
HPC export license to Hong Kong (with a value of $300,000) was
rejected
· 100 HPC
export licenses to the PRC (with a total value of $11,831,140)
were rejected by Commerce
· 37 HPC
export licenses to Hong Kong (with a total value of $55,879,177)
were approved
· 23 HPC
export licenses to the PRC for HPCs within the 2,000 to 7,000
MTOPS range (with a total value of $28,067,626) were approved
· Two of
the 23 HPC export licenses to the PRC for HPCs within the 11,000
to 12,800 MTOPS range (with a total value of $2,550,000) were
approved in 1998189
The approximate total value of the HPCs exported, of whatever
description, to both Hong Kong and the PRC, for the six-year
period ending September 23, 1997, was only $86 million.190
The nine-month period between January 1998 and September 1998,
however, saw U.S. exporters notify the Commerce Department of
their intention to export 434 HPCs (in the 2,000 to 7,000 MTOPS
range) to the PRC (total value $96,882,799).191 Nine times the
number of HPCs were exported in one-ninth the time.192
During approximately the same time frame (calendar year 1998)
it is estimated that 9,680,000 individual PCs and workstations
were sold in the PRC. The market share that U.S. exporters could
reasonably expect to benefit from was approximately 3,872,000
units, worth approximately $1.8 billion.193
Apparently, the proximate cause of U.S. computer manufacturers
aggressively lobbying for the raising and maintaining of export
thresholds above the PC level was to capture this $1.8 billion
per year market share.
The United States dominates the PRC's HPC market, but U.S.
exports clearly do not dominate the PRC's personal computer and
workstation market.194 The difference between the 460-unit, $100
million HPC market described above, stretched over a six-year
period, and the yearly 3.8 million-unit PC and workstation market,
with a value of $1.8 billion, is dramatic.
The performance levels of U.S. HPCs reported to be exported
to the PRC over the past year continued to be predominantly in
lower-end machines, as shown in the following table. For example,
77 percent of U.S. HPCs (a total of 388 machines) have performance
levels below 4,000 MTOPS.
The PRC Is Obtaining Software
From U.S. and Domestic Sources
In June 1997, it was estimated that 96 percent of software
programs sold in the PRC were pirated versions of commercially
available U.S. programs. These programs were designed for use
on PCs and workstations, and are not considered useful for the
very sophisticated programming done on HPCs.
Some major U.S. software producers have begun contracting
with PRC programming firms. These PRC software firms are comprised
of recently-graduated PRC university students. They are attempting
to write programs in Chinese to capitalize on a huge domestic
market.196
Two factors mitigate against the success of the PRC developing
its domestic programming industry.
The first factor is that street-level "software pirates"
sell dozens of U.S. computer programs at a time on one CD-ROM
for a small fee (reportedly $20). In other words, one can meet
most or all of one's programming needs in the PRC for a nominal
fee. It is anticipated that it will be difficult, if not impossible,
for a domestic software industry to recoup the start up costs
associated with just one software program, let alone the dozens
needed to compete with the street level dealers.
The second factor is that these pirated U.S.-produced, English
language programs are more mature, widespread, and robust than
PRC programs.197 It is axiomatic that any new product will have
"bugs in the system." It is considered unlikely that
new, unproven, and possibly weak software programs will effectively
compete with cheap, proven, and robust software that is widely
available at such nominal fees. It is conceivable that the PRC
will abandon instituting a domestic programming industry altogether.198
Potential Methods of
Improving End-Use Verification
According to a 1996 RAND study, there are non-intrusive and
intrusive approaches to assessing the manner in which a buyer
is actually applying dual-use technologies. Among the non-intrusive
methods are:
· Memoranda
of understanding and agreements
· National
technical means of verification
· Limitations
designed into the transferred technologies
· Transparency
measures
Among the intrusive methods are:
· Inspections
· Tagging199
Tagging
Tagging is achieved by attaching an active system to the item
that is to be exported, rather than just a passive tag for identification
during an inspection. The active system would both monitor the
object tagged and communicate that information back to the United
States. The RAND study noted that in practice, this means the
objects to be tagged must be physically large systems, such as
a machine-tool cell, or a major component of some larger system,
such as a turbine engine in a helicopter.200
According to the RAND study, the tag should be capable of
at least communicating information about the item's physical
location. Some sensors may provide other kinds of information,
as well. The information could be communicated to a satellite
or over a data link. Early versions of such devices were already
in use in 1996 to monitor nuclear materials and technologies.201
These "smart" tags exploit the potential of several
technologies, according to the RAND study. They combine encryption,
the Global Positioning System, and emerging global wireless communications
systems, such as Iridium or Orbcomm. These technologies would
allow the tags to report back on the status and location of the
tagged object. In principle, such tags could report the position
of an object at any given time in order to verify limitations
on their location. Such tags could also report on the activities
of a "smart" system to which they are attached. For
example, a machine-tool cell could report whether the machine
had been used to make parts resembling aircraft components.202
Such tags could have many applications in a cooperative regime.
Their application and use in a prohibited environment would be
more difficult and consequential.203
The RAND study cautioned that all sellers of a particular
technology must participate in the tagging and that this would
probably also require cooperation of the buyers. Otherwise, buyers
would gravitate to untagged items, if they were available. Attempts
to conceal system location or deviate from a pattern of cooperation
would be considered evidence of a potential failure of performance
by the buyer. The study concluded that tagging may become an
important oversight method for controlling technology transfers,
but that it should never become the sole means of oversight.204
Technical
Safeguards
In 1994 several types of technical safeguards were in advanced
development in the United States. The technologies required for
these safeguards were expected to enter testing within the next
two years. They included:
· Controlled-execution
UNIX - a modified computer operating system that could run
only certain pre-approved programs; likely to be most useful
for computers sold to facilities such as weather-forecasting
centers, oil companies, automobile manufacturers, and banks
· "Black
box" monitoring hardware - inexpensive, secure, long-term
audit recording devices, possibly based on write-once optical
storage units that could be embedded in mass-produced workstations;
analogous to the black box flight-data recorders that are installed
in aircraft and used for post-crash accident analysis
· "Meltdown"
software - modified operating system programs designed to
require updating by the manufacturer at fixed times; if not updated,
the computer refuses to run
· Automated
auditing tools - pattern-recognition or rule-based software;
would assist monitoring agencies to more effectively inspect
huge collections of data from system activity logs and detect
the (presumably few) incidents worth detailed analysis
Although these technical safeguards seem feasible, none had
been proved to be inexpensive, sensitive enough to detect most
illegal activity, and difficult to circumvent by determined adversaries.
The auditing tools under development showed great promise, however.
Authorities were pessimistic about the likelihood that technical
high-performance computer safeguards would be widely adopted
and able to succeed in the near future.
Other
Possibilities
Officials of the Mitre Corporation made several suggestions
to strengthen U.S. national security in the context of HPC export
controls. These included:
· Improving
and enforcing end-use and end-user verification
· Controlling
embedded HPC systems that are useful in military applications
· Monitoring
or precluding the expansion capability of computer hardware
· Marketing
aggressively all generic computing capabilities, such as scanning,
to the PRC to maximize profits and to keep the PRC market-dependent
on the United States
· Focusing
on control of any hardware, software, tools, and services that
uniquely support PRC military applications that are strategic
in nature or could facilitate the tactical turning point in a
conflict205
Chapter 3
Technical Afterward
CHANGING HIGH PERFORMANCE
COMPUTER TECHNOLOGY
IS MAKING EXPORT
CONTROL MORE DIFFICULT
ew designs in HPCs and systems of computers, as well as availability
of more advanced and less costly processors, software, and peripheral
equipment, is rendering the challenge of applying export controls
to HPCs more difficult.
For certain types of computer designs, the ability to add
processors or boards could increase the machine's performance
beyond authorized levels. In addition, advances in computer processor
communications technology have facilitated the clustering of
personal computers and workstations into effective parallel computers.
The usefulness of clustered computers is application-dependent.
Some U.S. Government and computer industry experts have concluded
that for many problems, networks of workstations could not compete
with appropriately designed high performance computers.206 Most
traditional HPCs achieve far greater efficiency than parallel
machines, due to their use of custom-made components.
Foreign access to high performance computers through networks
is possible because of inadequate security measures.
Vector Architectures
Vector architecture relies on custom-designed processors to
move a complex problem through computer processing units in sequential
stages. This type of machine is designed to handle arithmetic
operations efficiently on elements of arrays, called vectors.207
Vector systems are especially useful in high-performance scientific
computing.208 Vector systems, also called "pipeline"
architectures, work like an assembly line. They work best with
many similar tasks that can be broken down into steps.
The memory interface in vector machines is custom-made, and
subject to export controls.
Vector machines
are useful for cryptography, modeling fluids, and in the design
of weapons. In particular, vector systems are suited to problems
in which data at one point influence other variables in the problem,
a common situation in national security applications.209
It is more straightforward for a programmer to use a vector
system than a system comprised of parallel processors (discussed
below), since it is easier to obtain maximum performance with
one or a few high-power processors than with a collection of
many lower capability processors.210
Since one of the main concerns with any HPC system is the
rate of speed with which data can be retrieved from memory, another
advantage is that a vector machine has a very fast memory.211
Still further advantages of vector systems are that they feature
high memory bandwidth and low memory latency - that is, very
large amounts of data can travel to and from memory very efficiently.
A related advantage is that vector systems have the ability to
seek multiple memory locations at the same time. This translates
into very fast computational speed.
A disadvantage of a vector machine is that vector system software
is not really portable. It cannot be readily transported to other
vector machines.212
The main disadvantage of vector systems, however, is their
high cost. Significant improvements in software and hardware
allow the purchase of a parallel processing system for $40,000,
as opposed to $1 million for a comparable vector computer.213
At the Defense Department's High Performance Computer Management
Office, vector systems are being phased out in favor of parallel
processing systems. Out of a total of 40 HPCs in the High Performance
Computer Management Office inventory, fewer than 10 are now vector
systems.214
Parallel Processing: The Connection of Computers Into a Powerful
Central Resource
A parallel processing computer is a collection of processors
that are connected through a communications network.215 The type
of processor, the network configuration, and the operating system
that coordinates the activities distinguish parallel processing
systems.
Many national security applications involve problems that
can be separated into independent variables, and it is for these
types of problems that parallel processing is best suited.216
The fastest parallel machines are all based on commodity processors
- that is, processors that are commercially available on the
market.217 This approach has been applied to virtually every
area of theoretical and applied physics.218
Massively Parallel Processors
A massively parallel processor is a collection of computers,
or central processing units, linked together.219 Each computer
that is part of the whole massively parallel processor has its
own memory, input/output system, and central processing unit.220
Massively parallel processors now use commodity processors, and
can utilize commodity interconnects to communicate between the
individual computers that make up the system.221 Some massively
parallel processors use custom-made, very fast interconnect switches
that are not commodities and are subject to export control.222
An advantage of a massively parallel processor is that an
unlimited quantity of processors can be incorporated into the
design of the machine. In a massively parallel processor, the
more processors, the greater the computing speed of the machine.223
Because each processor is equipped with its own memory, massively
parallel processors have much more memory than traditional supercomputers.
The extra memory, in turn, suits these machines to data-intensive
applications, such as imaging or comparing observational data
with the predictions of models.224
A disadvantage of
massively parallel processors is that memory latency is a bigger
problem because the processors have to share the available memory.
Another disadvantage is that each one of the computers that is
part of the system has to be instructed what to do individually.225
This phenomenon requires specialized, extremely proficient programmers
to create efficient communications between the individual computers.
The commercial availability of inexpensive, powerful microprocessors
has given massively parallel processors a boost in their competition
with vector machines for the supercomputer market. IBM, for example,
more than doubled the number of its computers in the Top 500
list (discussed below) between November 1997 and June 1998 by
introducing the SP2, which strings together up to 512 of the
company's RSI6000 workstation microprocessors.226
If optimum speed is desired, this massively parallel configuration
is the best of all HPC designs.227 The fastest high performance
computer now available is the ASCI Blue Pacific.228 That machine
is part of the Department of Energy's Accelerated Strategic Computing
Initiative (ASCI) program and is located at Lawrence Livermore
National Laboratory. Developed in conjunction with IBM, it is
a 5,856-processor machine, boasting a top speed of 3.8 teraflops229
(Tflops) with 2.6 terabytes (Tbytes) of memory.230 In the next
phase of the ASCI initiative, IBM will deliver a 10-Tflops machine
to the Department of Energy in mid-2000.231
Symmetrical Multiprocessor Systems
Symmetrical multiprocessor systems use multiple commodity
central processing units (CPUs) that are tightly coupled via
shared memory. The number of processors can be as low as two
and as many as about 128.232
Symmetrical multiprocessor systems treat their multiple CPUs
as one very fast CPU.233 The CPUs in a symmetrical multiprocessor
system are arranged on a single motherboard and share the same
memory, input/output devices, operating system, and communications
path.
Although symmetrical multiprocessor systems use multiple CPUs,
they still perform sequential processing,234 and allow multiple
concurrent processes to be executed in parallel within different
processors.235
An advantage of symmetrical multiprocessor systems is that
the programming required to control the CPUs is simplified because
of the sharing of common components.236
Another major advantage is cost. A Silicon Graphics symmetrical
multiprocessor system, for example, with 18 microprocessors,
each rated at 300 megaflops (MFLOPS)237 or more, and a peak speed
of more than 5 gigaflops (GFLOPS), costs about $1 million, whereas
a Cray C90 costs about $30 million.238
Even though the Silicon Graphics machine is about a third
as fast as the Cray machine, it is still very popular with consumers
of these types of machines. The University of Illinois Supercomputing
Center reportedly likes the price, flexibility, and future promise
of symmetrical multiprocessor systems so much that it plans to
use them exclusively within two years. Its older Crays were "cut
up for scrap" at the beginning of this year, and its massively
parallel computers will be phased out by 1997.239
One disadvantage
of a symmetrical multiprocessor system is that all the CPUs on
a single board share the resources of that board. This sharing
limits the number of CPUs that can be placed on a single board.240
Although the programming model that a symmetrical multiprocessor
system provides has proved to be user-friendly, the programmer
must exercise care to produce efficient and correct parallel
programs. To limit latency in individual jobs, most software
requires enhancement - for example, employing special programming
techniques to prevent components of the computer program from
competing for system resources - thereby increasing inefficiency.
For this reason, symmetrical multiprocessor systems are not
good platforms for high-performance real-time applications.241
In a symmetrical multiprocessor system design, as is true
with a massively parallel processor system, the number of CPUs
determines how fast a machine potentially will operate. This
fact causes a problem for export controls because it is possible
to add CPUs to the boards of a symmetrical multiprocessor system,
or boards to a massively parallel processor system, and push
the machine over export control thresholds after the original
export-licensed purchase.242
Clusters of Commercial Off-the-Shelf
Computers and Networks
Recent advances in the process of computer-to-computer communication,
or networking, allow computers to be linked together, or "clustered."
Networking has allowed the clustering of personal computers and
workstations into well-balanced effective parallel computers,
with much higher computing capabilities than any one of the clustered
computers.243
Four thresholds have been crossed in connecting commercial-off-the-shelf
components to create parallel computers:
· Using commercial-off-the-shelf
components to create parallel computers is simple because
of the ease of hardware configuration and the availability of
all necessary system software from market vendors
· It is versatile
because a wide range of possible network designs with excellent
communication characteristics and scalability to large sizes
is now available
· Clustered
systems performance has now matured to the point that network
communication speed is within 50 percent of that in vendor-assembled
parallel computers244
· Commercial-off-the-shelf
clusters are now affordable
According to officials at the Lawrence Livermore National
Laboratory, networking represents only a 10 percent additional
cost over the cost of the computing hardware for large systems.
Thus, up to approximately 50,000 MTOPS, the computing capability
available to any country today is limited only by the amount
of money that is available to be spent on commercial-off-the-shelf
networking.245
A typical commercial-off-the-shelf networking technology contains
five essential elements. They are all inexpensive and widely
available. The three hardware elements are switches (approximate
cost: $2,000), cables (approximate cost: $100), and interface
cards (approximate cost: $1,500). The two software elements are
low-level network drivers for common operating systems, and industry
standard communication libraries. The hardware and software technology
necessary to successfully cluster commercial-off-the-shelf CPUs
into effective parallel computers is well developed and disseminated
in open, international collaborations worldwide.246
The concept of clustering commercial-off-the-shelf computers
has been a subject of open academic study for over a decade.
Today, the Beowulf Consortium acts as a focal point for information
on clustering technology and has links to many projects. One
Beowulf project is the Avalon computer at Los Alamos National
Laboratory. Avalon can operate at 37,905 MTOPS247 and was built
in four days in April 1998 entirely from commodity personal computer
technology (70 DEC Alpha CPUs) for $150,000.
Although commercial-off-the-shelf networking technology has
only recently become effective, it has been adopted rapidly.
There currently are at least seven competing high-performance
network technologies (over 100 megabytes per second or higher):
Myrinet, HIPPI, FiberChannel, Gigabit Ethernet, SCI, ATM, and
VIA. One network vendor reported over 150 installations in the
United States and 17 foreign countries including Australia, Brazil,
Canada, the Netherlands, England, France, India, Israel, Italy,
Japan, the Republic of Korea, and the PRC.248
Gigabit Ethernet is of particular interest because it is being
developed by a cooperative, worldwide industry effort called
the Gigabit Ethernet Alliance. 74 companies have pledged to develop
products for the open standard - that is, the source software
is available openly to software developers. Foreign companies
are alliance members and also participate as members of the steering
committee and the certification process for compliance. Gigabit
Ethernet is projected to be a $3 billion market by the year 2000,
which at today's prices translates into approximately 300,000
network switches per year.249
On October 15, 1997,
a group of experts met to discuss computer performance metrics
for export control purposes. The computer and high-tech industries
were represented by Hewlett-Packard, Silicon Graphics/Cray Research,
IBM, Digital Equipment Corporation, Intel, Sun Microsystems,
the Center for Computing Sciences, the Institute for Defense
Analyses, and Centerpoint Ventures. The U.S. Government was represented
by the National Institute of Standards and Technology, the Naval
Research Laboratory, the Defense Advanced Research Projects Agency,
the National Security Agency, Lawrence Livermore National Laboratory,
the Defense Technology Security Administration, and the Department
of Commerce Bureau of Export Administration.250
The consensus of the discussion was that commercial-off-the-shelf
networking is not so significant a threat to replace HPCs as
might at first appear to be the case:
Networks of workstations using [commercial-off-the-shelf]
networking technology differ from supercomputers. Some problems
will run easily and effectively on such networks, while other
classes of problems important to national security concerns will
not run effectively without a major software redesign effort.
For many problems no amount of software redesign will allow networks
of workstations to compete with appropriately designed high performance
computers.
Even if a "rogue state" assembled such a large
network of workstations by legitimately acquiring large numbers
of commodity processors, the actual effort to produce the software
necessary to realize the full potential of such an aggregate
system would take several years. During this time,
the state of the art of computational technology would have increased
by approximately an order of magnitude.
After considerable discussion, most of the participants
were in
agreement that there was a fundamental difference between a system
designed by a single vendor that was built as an aggregate of
many commodity processors and included the software to enable
these processors to cooperatively work on solving single problems
of national concern, and a large collection of commodity processors
not subject to export
control that are externally networked together.251
According to one expert, many universities have clustered
systems, as they are easy to establish. For $70,000, a 12-node
system with two Pentium II processors at 300 megahertz (MHz)
each would produce a system with 7,200 GFLOPS.. However, the
system must be properly structured to perform well, and performance
will vary depending on the application, the programmer's ability,
and the connection of the machines. An integrated system from
Silicon Graphics/Cray will achieve between 10-20 percent of peak
performance at best.252
An example of a powerful commercial-off-the-shelf network
can be found at the Illinois Supercomputing Center. Four eight-processor
and two 16-processor machines from Silicon Graphics are connected
in a cluster with a peak speed of nearly 20 GFLOPS.253
According to one expert, it does not require any special expertise
to network workstations using commercial-off-the-shelf technology.
The software engineering techniques are being taught to undergraduates
as part of standard courses in advanced computing, but anyone
with programming knowledge should be able to create a network
as well.254
The parallel supercomputers of today have peak speeds of over
100 billion floating point operations per second (100 GFLOPS).
This is roughly 100 times the peak speed of a Cray YMP class
machine, which was the standard for high-performance computing
of just five years ago.255
However, it is difficult to achieve a high percentage of this
peak performance on a parallel machine.
Whereas a tuned
code running on a Cray might reach 80-90 percent of peak speed,
codes running on parallel computers typically execute at only
10-20 percent of peak.256 There are two reasons for this:
· The first
is that Cray-class computers incorporate extremely expensive,
custom-designed processors with vector-processing hardware.
These processors are designed to stream large amounts of data
through a highly efficient calculational pipeline. Codes that
have been tuned to take advantage of this hardware ("vectorized"
codes) tend to run at high percentages of peak speed.257
Parallel machines, on the other hand, are generally built
from much simpler building blocks. For example, they may use
the same processors that are used in stand-alone computer workstations.
Individually, these processors are not nearly so sophisticated
or so efficient as the vector processors. Thus, it is not possible
to achieve so high a percentage of peak speed.258
Some parallel machines contain custom processors (TMC CM-5
vector units) or custom modifications of off-the-shelf processors
(Cray T-3D modified DEC alpha chips). Even in those cases, however,
the percent of peak achievable on a single node is still on the
order of 50 percent or less. In parallel computer design, there
is constant tension between the need to use commodity parts as
the computational building blocks in order to achieve economies
of scale, and the desire to achieve ever-higher percentages of
peak performance through the implementation of custom hardware.259
· The second
reason that parallel computers run at lower percentages of peak
speeds than vector supercomputers is communications overhead.
On parallel computers, the extraordinary peak speeds of 100 GFLOPS
or more are achieved by linking hundreds or even thousands of
processors with a fast communications network.
Virtually all parallel computers today are "distributed
memory" computers. This means that the random access memory
(RAM) is spread though the machine, typically 32 megabytes at
each node. When a calculation is performed on a parallel machine,
access is frequently needed to pieces of data on different nodes.
It may be possible to overlap this communication with another
computation in a different part of the program in order not to
delay the entire program while waiting for the communication,
but this is not always the case. Since the timing clock continues
while the communication is taking place, even though no calculational
work is being performed, the measured performance of the code
goes down and a lower percentage of peak performance is recorded.260
Domain Decomposition
"Domain decomposition" involves partitioning the
data to be processed by a parallel program across the machine's
processors.261
In distributed memory architectures, each processor has direct
access only to the portion of main memory that is physically
located on its node. In order to access other memory on the machine,
it must communicate with the node on which that memory is located
and send explicit requests to that node for data.262 Figuring
out the optimal domain decomposition for a problem is one of
the most basic and important tasks in parallel computing, since
it determines the balance between communication and computation
in a program and, ultimately, how fast that program will run.263
Memory access constitutes an inherent bottleneck in shared-memory
systems.264
Highly Parallel Technology
Microprocessor-based supercomputing has brought about a major
change in accessibility and affordability. Massively parallel
processors continue to account for more than half of all installed
supercomputers worldwide, but there is a move toward shared memory,
including the use of more symmetrical multiprocessor systems
and of distributed-shared memory. There is also a tendency to
promote scalability through the clustering of shared memory machines
because of the increased efficiency of message passing this offers.
The task of data parallel programming has been helped by standardization
efforts such as Message Passing Interface and High-Performance
Fortran.265
Highly parallel technology is becoming popular for the following
reasons. First, affordable parallel systems now out-perform the
best conventional supercomputers. Cost is, of course, a strong
factor, and the performance per dollar of parallel systems is
particularly favorable.266 The reliability of these systems has
greatly improved. Both third-party scientific and engineering
applications, as well as business applications, are now appearing.
Thus, commercial customers, not just research labs, are acquiring
parallel systems.267
Twice a year the "Top 500 list," a compendium of
the 500 most powerful computer systems, is published.268 On the
previous page is an example of the numbers and types of systems
in the biannual list of the top 500 fastest computers. As this
chart points out, massively parallel processors and symmetrical
multiprocessor systems are on the rise, while vector systems
are losing ground.269
Microprocessor Technology
While vector and massively parallel computers have been contending
for the supercomputing market, an important new factor has become
the availability of extremely powerful commodity microprocessors,
the mass-produced chips at the heart of computer workstations.
Ten years ago, workstation microprocessors were far slower
than the processors in supercomputers. The fastest microprocessor
in 1988, for example, was rated at one million floating point
operations per second (MFLOPS) while Cray's processors were rated
at 200 MFLOPS.270 A floating-point operation is the equivalent
of multiplying two 15-digit numbers. Today, Cray's processors
have improved by a factor of ten, to two gigaflops in the brand-new
T90; but the fastest microprocessor runs at 600 MFLOPS, an improvement
by a factor of 600.
Commercial off-the-shelf microprocessor power is available
for a fraction of the cost of a traditional vector processor.
Unlike vector processors, which consist of complex collections
of chips and are only fabricated by the hundreds each year, commercial
off-the-shelf microprocessors are designed for mass production
based on two decades of experience making integrated circuits.
Research and development costs for each commercial off-the-shelf
microprocessor are spread over hundreds of thousands of chips.271
Microprocessors, also known as CPUs, are integrated circuits.
They can be divided into broad categories of logic family technologies.
The selection of a certain logic technology in the design of
an integrated circuit is made after determining an application
and weighing the advantages of each type of logic family. Among
these are:
· Emitter-Coupled
Logic (ECL) is used for circuits that will operate in a high-speed
environment, as it offers the fastest switching speeds of all
logic families; it is the first type HPC chip. ECL, however,
is power-hungry, requires complex cooling techniques, and is
expensive.272
· Complementary
Metal-Oxide Semiconductor Logic (CMOS) is relatively inexpensive,
compact and requires small amounts of power. CMOS off-the-shelf
is the standard PC or workstation chip; proprietary CMOS is custom-built,
specially designed for the particular HPC and incompatible with
PCs and workstations.
Realizing the differences between logic technologies gives
a perspective to understanding where CPU technology is headed,
and the reasons that the market is driving one technology faster
than another. As the following chart illustrates, commercial
off-the-shelf, inexpensive CPUs are coming to dominate the high
performance computing world.273
Interconnect Technology
In multiprocessor systems, actual performance is strongly
influenced by the quality of the "interconnect" that
moves data among processors and memory subsystems.274
Traditionally, interconnects could be grouped into two categories:
proprietary high-performance interconnects that were used within
the products of individual vendors, and industry standard interconnects
that were more readily available on the market, such as local
area networks.275 The two categories featured different capabilities,
measured in bandwidth and latency.
Recently, a new class of interconnect has emerged: clustering
interconnects. These offer much higher bandwidth and lower latency
than local area networks. Their shortcomings are comparable to
proprietary high-performance interconnects, including lower bandwidth,
higher latency, and greater performance degradation in large
configurations or immature system software environments.276
Message Passing Interface
Message Passing Interface (MPI) is a program containing a
set of sub-routines that provide a method of communication that
enables various components of a parallel computer system to act
in concert. The communications protocol that MPI uses is the
same utilized by the Internet. According to Dr. Jeff Hollingsworth
of the University of Maryland Computer Science Department, an
example of how each of the different software applications interact
with the hardware would be as follows:277
Application (Code)
MPI
TCP/IP
Linux
Windows NT (Operating system)
Hardware
Some software, says Hollingsworth, is sold in a version that
is compatible with MPI. One example is automobile crash simulation
software. This software, which is essentially code to simulate
a physical system in three dimensions, is adaptable to other
scientific applications such as fluid dynamics, according to
Hollingsworth.278
Hollingsworth states that software that is not already "MPI
ready" can be modified into code that can be run in an MPI,
or parallel, environment. Modifying this software to enable it
to run in an MPI environment can be very difficult, or quite
easy, says Hollingsworth, depending on "data decomposition."
279
The ease of converting software that is not "MPI ready"
into an "MPI ready" version is dependent on the expertise
of the software engineers and scientists working on the problem.
For a single application and a single computer program, the level
of expertise required to convert a computer program in this way
is attainable in graduate level, and some undergraduate level,
college courses, according to Hollingsworth.280
It has not been possible to determine which, if any, commercially
available software is both MPI ready and applicable to defense-related
scientific work.
|