COMPUTING RESOURCES

To provide state-of-the-art capabilities in scalable parallel computing, NERSC received delivery of the 512-processor Cray T3E-900 on July 14. The NERSC T3E-900 features the largest I/O system built to date and is a first-of-its-kind configuration. The fully configured machine offers 1.5 terabytes (TB) of disk storage, a read/write capability of 800 megabytes (MB) per second, and 128 gigabytes (GB) of memory. NERSC also has acquired a T3E-600 to provide a development platform for NERSC clients and to support Berkeley Lab's computing needs. The successful checkpointing/restart of the T3Es is described above.

There are four systems in NERSC's Cray J90se cluster: Killeen is the interactive machine, while Franklin, Seymour, and Bhaskara perform batch processing. Franklin and Seymour now have queues that can accept jobs up to 1.6 GB, while Bhaskara has a queue limit of 4 GB, which represents half the physical memory on the machine. These expanded queues allow the J90 cluster to accept larger batch jobs. In a configuration that NERSC calls "SuperHome," the cluster machines mount Killeen's home directories via NFS so that user executables and files are available to each machine.

Members of NERSC's Operations and Systems groups ensure that the center's supercomputing facilities are up and running at optimum availability, reliability, and efficiency. Clayton Bagwell of the Operations Group and Jackie Scoggins of Systems are part of the team that has helped NERSC consistently provide clients with ever-improving service.

Requirements-based batch scheduling on the T3Es and J90s is managed by the Network Queuing Environment (NQE). With its intelligent scheduler, NQE automatically assigns the required processing resources to each computing task while balancing the workload among the processors. Clients specify their requirements (memory, number of processors, disk space, and processor time) for each batch job. The NQE scheduler examines each request and determines when and where the job is run, based on system load information from NQE execution servers.

To keep pace with improvements in processing capacity and capability, we also upgraded our storage systems and bandwidth. We installed uninterruptible power and increased the UniTree disk cache ninefold from 94 GB to 846 GB. We expanded the archival storage bandwidth from 27 MB/s to a potential 252 MB/s and expanded capacity from 33.6 TB to 106 TB (uncompressed). We added a second storage control processor and disks to store UniTree/ HPSS databases. Finally, we acquired six IBM processors to use as mover machines for HPSS and to upgrade the local-area network within the machine room, increasing the bandwidth.

HPSS (High Performance Storage System) is the next-generation storage technology selected by NERSC after completing a market survey of storage systems and evaluating the options. The primary objective of HPSS is to move very large data objects between high performance computers, workstation clusters, and storage libraries at speeds many times faster than is possible with today's software systems. We set up an HPSS test environment in late 1997, and over the next year we will convert the UniTree and CFS environments to HPSS.

In order to store and retrieve data gathered and used by researchers, NERSC operates two high-performance data storage systems. Together, these systems can hold 50,000 tapes, each storing from one to 20 gigabytes of compressed data. Not only have Wayne Hurlbert and Nancy Meyer helped keep NERSC's storage systems running at top efficiency, but they've also tracked down system design problems which could affect all users of such systems.

A new remote visualization server -- an SGI Onyx 2 -- has also been installed to enable remote clients to create high-quality visualizations of data. The Onyx is integrated with the computing systems to provide a consistent environment for maintaining visualization software across NERSC systems.

Nationwide and international high-speed access to NERSC is provided by the Energy Sciences Network (ESnet), which is operated around the clock by the NERSC Operations staff.

The PDSF (Particle Detector Simulation Facility), a networked, distributed, production computing environment, supports the detector simulation, software development, and data analysis needs of large-scale high-energy physics and nuclear science investigations. The PDSF has brought a new client community to NERSC, with research teams from six experiments currently using the facility.

As part of NERSC's Advanced Systems Development program, we are collaborating with Sun Microsystems in the testing, evaluation, and development of a prototype single-system-image cluster of four symmetric multiprocessor systems. The cluster is designed to permit the systems, which have a total of 32 processors, to be deployed, managed, and viewed as a single operating system.

Implementing all of these new systems presented many challenges. For example, during the long and rigorous testing of the Cray T3E, NERSC and SGI/Cray representatives worked closely together to identify and resolve problems and make improvements. The end result was that NERSC clients had to wait a little longer, but were rewarded with a truly scalable production MPP machine. The solutions generated by the NERSC-Cray collaboration benefited other centers with T3Es as well.

Next Page
Back to Table of Contents