A century’s worth of reconstructed weather data will provide a better baseline for climate change studies
Project: The 20th Century Reanalysis Project
PI: Gil Compo, University of Colorado/ CIRES/Climate Diagnostics Center and NOAA Earth System Research Lab
Senior investigators: Jeffrey Whitaker, NOAA Earth System Research Lab; Prashant Sardeshmukh, University of Colorado/CIRES
Funding: INCITE, CIRES, NOAA
The distinction between climate and weather was expressed most succinctly by science fiction writer Robert A. Heinlein: “Climate is what you expect; weather is what you get.” But as global warming produces more noticeable changes on a planetary scale, how do we even know what to expect in a particular region?
Climate change studies are increasingly focused on understanding and predicting regional changes of daily weather statistics. But to predict the next century’s statistical trends with confidence, researchers have to demonstrate that their forecasting tools can successfully recreate the conditions of the past century. That requires a detailed set of historical atmospheric circulation data — not just monthly averages, but statistics for at least every six hours, so that phenomena like severe storms can be analyzed.
Although there is scant atmospheric data from weather balloons and none from satellites for the first half of the 20th century, there is an enormous amount of observational data collected at the Earth’s surface by a variety of sources, from meteorologists and military personnel to volunteer observers and ships’ crews. Until recently, this two-dimensional data was widely available only on hand-drawn weather maps (Figure 1). Despite many errors, these maps are indispensable to researchers, and extensive efforts are being made to put these maps into a digital format and make them available on the Web.
Now, using the latest data integration and atmospheric modeling tools and a 2007 INCITE award of 2 million supercomputing hours at NERSC, scientists from the NOAA Earth System Research Lab and the Cooperative Institute for Research in Environmental Sciences (CIRES) are building the first complete database of three-dimensional global weather maps of the 20th century.
Called the 20th Century Reanalysis Project, the new dataset will double the number of years for which a complete record of three-dimensional atmospheric climate data is available, extending the usable digital dataset from 1948 back to 1892. The team expects to complete the dataset within two years, including observations currently being digitized around the world. The final maps will depict weather conditions every six hours from the Earth’s surface to the level of the jet stream 1 (about 11 km or 36,000 ft high), and will allow researchers to compare the patterns, magnitudes, means, and extremes of recent and projected climate changes with past changes.
“We expect the reanalysis of a century’s worth of data will enable climate researchers to better address issues such as the range of natural variability of extreme events including floods, droughts, hurricanes, extratropical cyclones, and cold waves,” said principal investigator Gil Compo of CIRES. Other team members are Jeff Whitaker of the NOAA Earth System Research Lab and Prashant Sardeshmukh, also of CIRES, a joint institute of NOAA and the University of Colorado.
“Climate change may alter a region’s weather and its dominant weather patterns,” Compo said. “We need to know if we can understand and simulate the variations in weather and weather patterns over the past 100 years to have confidence in our projections of changes in the future. The alternative — to wait for another 50 years of observations — is less appealing.”
From two to three dimensions
Compo, Whitaker, and Sardeshmukh have discovered that using only surface air pressure data, it is possible to recreate a snapshot of other variables, such as winds and temperatures, throughout the troposphere, from the ground or sea level to the jet stream.1 This discovery makes it possible to extend two- dimensional weather maps into three dimensions. “This was a bit unexpected,” Compo said, “but it means that we can use the surface pressure measurements to get a very good picture of the weather back to the 19th century.”
The computer code used to combine the data and reconstruct the third dimension has two components. The forecast model is the atmospheric component of the Climate Forecast System, which is used by the National Weather Service’s National Centers for Environmental Prediction (NCEP) to make operational climate forecasts. The data assimilation component is the Ensemble Kalman Filter.
Data assimilation is the process by which raw data such as temperature and atmospheric pressure observations are incorporated into the physics-based equations that make up numerical weather models. This process provides the initial values used in the equations to predict how atmospheric conditions will evolve. Data assimilation takes place in a series of analysis cycles. In each analysis cycle, observational data is combined with the forecast results from the mathematical model to produce the best estimate of the current state of the system, balancing the uncertainty in the data and in the forecast. The model then advances several hours, and the results become the forecast for the next analysis cycle.
The Ensemble Kalman Filter is one of the most sophisticated tools available for data assimilation. Generically, a Kalman filter is a recursive algorithm that estimates the state of a dynamic system from a series of incomplete and noisy measurements. Kalman filters are used in a wide range of engineering applications, from radar to computer vision to aircraft and spacecraft navigation. Perhaps the most commonly used type of Kalman filter is the phase-locked loop, which enables radios, video equipment, and other communications devices to recover a signal from a noisy communication channel. Kalman filtering has only recently been applied to weather and climate applications, but the initial results have been so good that the Meteorological Service of Canada has incorporated it into their forecasting code. The 20th Century Reanalysis Project uses the Ensemble Kalman Filter to remove errors in the observations and to fill in the blanks where information is missing, creating a complete weather map of the troposphere.
Rather than making a single estimate of atmospheric conditions at each time step, the Ensemble Kalman Filter reduces the uncertainty by covering a wide range — it produces 56 estimated weather maps (the “ensemble”), each slightly different from the others. The mean of the ensemble is the best estimate, and the variance within the ensemble indicates the degree of uncertainty, with less variance indicating higher certainty. The filter blends the forecasts with the observations, giving more weight to the observations when they are high quality, or to the forecasts when the observations are noisy. The NCEP forecasting system then takes the blended 56 weather maps and runs them forward six hours to produce the next forecast. Processing one month of global weather data takes about a day of computing, with each map running on its own processor. The Kalman filter is flexible enough to change continuously, adapting to the location and number of observations as well as meteorological conditions, thus enabling the model to correct itself in each analysis cycle.
“What we have shown is that the map for the entire troposphere is very good, even though we have only used the surface pressure observations,” Compo said. He estimates that the error for the 3D weather maps will be comparable to the error of modern two- to three-day weather forecasts.
The reanalysis team ran their code on all four of NERSC’s large-scale computing systems — Bassi, Jacquard, Seaborg, and Franklin — and switched to a higher-resolution algorithm when they moved to Franklin. “We got fabulous support from the consultants,” Compo said, “especially Helen He and David Turner, on porting code, debugging, disk quota increases, using the HPSS, and special software requests.” They parallelized virtually their entire workflow on the Franklin architecture via job bundling, writing compute-node shell scripts, and using MPI sub-communicators to increase the concurrency of the analysis code.
Filling in and correcting the historical record
With the 2007 INCITE allocation, the researchers reconstructed weather maps for the years 1918 to 1949. In 2008, they plan to extend the dataset back to 1892 and forward to 2007, spanning the 20th century. In the future, they hope to run the model at higher resolution on more powerful computers, and perhaps extend the global dataset back to 1850.
One of the first results of the INCITE award is that more historical data are being made available to the international research community. This project will provide climate modelers with surface pressure observations never before released from Australia, Canada, Croatia, the United States, Hong Kong, Italy, Spain, and 11 West African nations. When the researchers see gaps in the data, they contact the country’s weather service for more information, and the prospect of contributing to a global database has motivated some countries to increase the quality and quantity of their observational data.
The team also aims to reduce inconsistencies in the atmospheric climate record, which stem from differences in how and where atmospheric conditions are observed. Until the 1940s, for example, weather and climate observations were mainly taken from the Earth’s surface. Later, weather balloons were added. Since the 1970s, extensive satellite observations have become the norm. Discrepancies in data resulting from these different observing platforms have caused otherwise similar climate datasets to perform poorly in determining the variability of storm tracks or of tropical and Antarctic climate trends. In some cases, flawed datasets have produced spurious long-term trends.
The new 3D atmospheric dataset will provide missing information about the conditions in which early-century extreme climate events occurred, such as the Dust Bowl of the 1930s and the Arctic warming of the 1920s to 1940s. It will also help to explain climate variations that may have misinformed early-century policy decisions, such as the prolonged wet period in central North America that led to overestimates of expected future precipitation and over-allocation of water resources in the Colorado River basin.
But the most important use of weather data from the past will be the validation of climate model simulations and projections into the future. “This dataset will provide an important validation check on the climate models being used to make 21st century climate projections in the recently released Fourth Assessment Report of the Intergovernmental Panel on Climate Change,” Compo said. “Our dataset will also help improve the climate models that will contribute to the IPCC’s Fifth Assessment Report.”
