Section site map: News

• News Home
• Media Press Room
• User/System News

Feature Stories

The Next Big Thing

by Karen Green, NCSA

With $53 million from the National Science Foundation, NCSA, SDSC, Argonne, and Caltech will create the largest, most comprehensive infrastructure ever deployed for scientific research.

In the near future, scientists will have access from their desktop workstations to the world's fastest computers, data archives from around the world, and sophisticated software and visualization tools. Without leaving the office or lab, researchers will be able to collaborate in real time with colleagues thousands of miles away, manipulating computer simulations and mining insights from the petabytes of data collected daily by scientific instruments.

That's the vision of the TeraGrid, a project that will include the fastest unclassified supercomputers, a wealth of scientific applications and visualization environments, and toolkits for grid computing linked through the world's fastest network into an integrated information infrastructure.

"The TeraGrid is about a vision of the future that will radically transform the way we interact and the way we work," predicts Dan Reed, director of NCSA and the Alliance and co-principal investigator of the DTF project. "Unprecedented amounts of data are being generated by new observatories and sensors, and groups of scientists are conducting new simulations of increasingly complex phenomena. This new age of science requires a sustainable national infrastructure that can bring together new tools, powerful computers, and the best minds in the country. This is the infrastructure that will allow us to solve the most pressing scientific problems of our time."

In August NSF awarded funds to build and deploy the TeraGrid to a partnership of four sites: NCSA, the San Diego Supercomputer Center (SDSC) at the University of California at San Diego, Argonne National Laboratory (Argonne), and the California Institute of Technology (Caltech). To build the facility the partnership expects to work primarily with IBM, Intel Corporation, and Qwest Communications along with Myricom, Sun Microsystems, and Oracle Corporation.

Linux cluster power
The bulk of the TeraGrid's computational power—an 8-teraflop IBM Linux cluster powered by the next generation of Intel® Itanium™ processors, code named McKinley—will be based at NCSA. This TeraGrid system will build upon the two existing clusters of more than 1,300 Itanium and IA-32 processors already deployed at the center and will include 240 terabytes of secondary storage.

NCSA's 8-teraflop system will be part of a 13.6-teraflop Linux cluster system distributed across the four sites—the fastest Linux cluster ever deployed and the fastest supercomputer in a nonclassified setting. The distributed system will consist primarily of clustered IBM servers based on the Intel Itanium family of processors interconnected with Myricom's Myrinet. The system will be capable of managing and storing more than 450 terabytes of data.

To operate as a single distributed facility, the clusters will be linked via a dedicated optical network that will initially operate at 40 gigabits per second and later be upgraded to between 50 and 80 gigabits per second. This TeraGrid network, developed in partnership with Qwest, will transport data 16 times faster than the fastest research networks now in operation. It will connect to Abilene, the high-performance network that links more than 180 research institutions across the country; STAR TAP, an interconnect point in Chicago that provides access to and from international research networks; and CENIC's CalREN-2, an advanced high-speed network that connects institutions in California. In Illinois the I-WIRE optical network will provide the TeraGrid with network capacity and will give Argonne and NCSA additional bandwidth for related network-research initiatives.

The TeraGrid network could revolutionize collaborative science, according to Argonne's Charlie Catlett, architect of the network, simply by giving researchers the bandwidth they need to do large-scale data analyses across geographic locations. "Many of the things scientists have wanted to do over the past several decades have been left untried simply because moving the data around can take days or weeks. This network will reduce that time to minutes or hours, opening up entirely new possibilities," says Catlett.

A needs-driven project
According to Fran Berman, co-principal investigator on the TeraGrid project and director of SDSC and NPACI, development of the TeraGrid will be driven by the needs of society. "We are in the decade of data, and analysis and synthesis of that data are an important part of enabling scientific advances." She said use of the TeraGrid and its terascale computing system will help scientists develop better drugs for cancer treatment, allow further in-depth study of the human genome and brain, and enable scientists to analyze weather data so quickly they will be able to create real-time forecasts that can predict down to the kilometer where a tornado or severe storm is likely to strike.

The Globus toolkit, a set of services and software libraries that supports grid computing and grid applications, will be an integral part of the TeraGrid. Developed by Argonne and the Information Sciences Institute at the University of Southern California, Globus includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. More than 100 institutions will contribute additional applications to the TeraGrid.

"Recent breakthroughs in chemistry and the life sciences have presented us with an even greater demand for advanced computation," says Argonne's Rick Stevens, TeraGrid project leader and the Alliance's chief computational architect. "If we are to achieve the performance necessary to support these new applications, we must develop capabilities to harness the collective power of not only dozens of supercomputers but thousands of individual PCs. The TeraGrid will provide critical insight into building such systems, while immediately enabling new classes of science."

Complementing NCSA's 8-teraflop cluster will be a data-intensive IBM Linux cluster based on Intel Itanium family processors (McKinley). This cluster will be located at SDSC and have a peak performance of just over 4 teraflops as well as 225 terabytes of network disk storage. In addition, SDSC will deploy a next-generation Sun Microsystems high-end server, which will provide a gateway to grid-distributed data for data-oriented applications.

Argonne will lead the effort to deploy advanced distributed computing software, high-resolution rendering and remote visualization capabilities, and networks. This effort will require a 1-teraflop IBM Linux cluster with parallel visualization hardware.

Caltech will focus on providing online access to very large scientific data collections and will facilitate access to those data by connecting data-intensive applications to components of the TeraGrid. Caltech will deploy a 0.4-teraflop cluster and an IA-32 cluster that will manage 86 terabytes of online storage.

But beyond the teraflops of computing power, the data storage facilities, and the ultrafast network, the TeraGrid represents a leap forward in creating a comprehensive computational, data management, and networking infrastructure of unprecedented scale and capability. It connects scientists and engineers as a cyber community with distributed scientific instruments, terascale and petascale computing facilities, multiple-petabyte data archives, and gigabit (and soon terabit) networks.

"The TeraGrid is a transforming idea in which reference to 'place' becomes obsolete," says Reed. "It will be a far more powerful, more flexible, and more inclusive scientific tool than any single supercomputing system could ever be, and it will give us the power and the resources we need to head into a new age of scientific discovery."

Top