Summary of ARM Carbon Data Products Project

October 22, 2004

Raymond A. McCord, William W. Hargrove, Craig C. Brandt, and Henriette I. Jager

Environmental Sciences Divsion

Oak Ridge National Laboratory

Background

Funds were established in FY 2000 for facilitation of carbon cycle research within the Atmospheric Research Measurement (ARM) Program. Two primary activities were identified for the "facilitation" of carbon cycle research:

Add new measurements at the ARM sites that explicitly measured carbon cycle attributes. This activity is lead by Margaret Torn (LBNL) and is documented at: http://esd.lbl.gov/ARMCarbon/
Extract measurements from the existing data collection at the ARM Archive and package them in ways that would be directly usable by carbon cycle researchers. This activity was proposed because the ARM data collection involves millions of data files and 100's of measurements. This activity was lead by Raymond McCord (ORNL). "Facilitation" was expected as a result of preparing data products that were specifically tailored and ready to use in carbon cycle research.

The efforts to develop and use data products for carbon cycle research from existing ARM data are summarized in this report.

Summary of activities

This section provides a summary of activities that were conducted during the project. The project's activities can be described by the following outline:

Identify "matches" between ARM measurements and carbon information "needs"
Process data to meet those "needs"
Demonstrate the usefulness of these processed data
Make contacts with potential data users

More details are provided on each of these topics in the sections below.

Identification of "matches" between ARM measurements and carbon data "needs"

The primary use of ARM measurements (ongoing atmospheric and meteorological observations) for carbon cycle research is with carbon cycle models. The ARM measurements can be used either for input for the environmental conditions of the model (driver files) or for comparisons between measured and modeled (estimated) parameters. A review of the primary carbon cycle models was conducted and we determined the following products could be extracted from the ARM data collection:

Driver (forcing) files based on basic meteorology and radiation measurements
Comparison (validation) data sets (measured vs. modeled) for soil water and temperature.

Brief consideration was also given to measurements of latent and sensible heat and eddy covariance (energy and carbon flux). However, the usefulness of these measurements contained uncertainty that was not resolved before the project ended. The latent and sensible heat needed some bias corrections that were incompletely implemented by ARM. The eddy covariance instruments were completely replaced during the duration of this project and previously collected data discarded.

Overview of data processing

The processing required for the data products included concatenation, filtering, aggregation, QA screening, and gap filling. Each of these processes are summarized below. Additional details about data processing can be found at "Processing ARM Data For Carbon Modeling" (see this section of this reference).

Concatenation, filtering, and aggregation.

The measurements collected by ARM are structured into data streams composed of daily files and aggregated by source instrument (or instrument group). The information needed for the ARM Carbon data products was distributed across several ARM data streams, intermingled with unneeded measurements (for carbon purposes), and distributed through 10,000's of daily data files. The first data processing step for these data products was to request the ARM data files from the Archive, concatenate the daily data files into yearly segments, and remove unneeded data fields.

The ARM measurements are also collected at differing time intervals. After the data files were reorganized, the data with higher temporal resolution were aggregated into useful time intervals (usually hourly or daily). The aggregations include the computation of appropriate statistics (mean, total, minimum, etc.).

Quality filtering

The ARM measurements were also filtered for quality problems based on the following logic:

Range testing - measurements that exceed QC limits were removed from the processing.
Known data quality problems - The ARM Program maintains Data Quality Reports (DQRs) that describe abnormal instrument operations and known problems with the measurements. The data values that had been previously indicated by DQRs to be unusable were removed from the processing. These DQRs are recorded in database independent from the data files and are provided in a report to data users along with the requested data files . We processed the DQR information to identify ARM measurements that should be excluded from further processing. Measurements designated as unusable in a DQR were excluded.
Illogical values - For example, soil water values were compared with precipitation measurements; Dry soil values after rainfall events were removed.

Gap filling

The data products used as "driver files" for carbon cycle models need to be complete (no missing time steps or missing input values within a time step). Observational data are always incomplete. We evaluated and implemented a statistical approach to fill in the data gaps. This approach includes searching for related values in both temporal and spatial contexts. More information about the gap filling methods and results

Demonstrating the value of using ARM data vs. other data sources

Part of the effort expended to �ǣsell�ǥ the usefulness of the data products to the carbon research community included using examples of the data products with carbon models. We conducted formal studies with widely used carbon models and compared the modeling results that would be obtained with regionally available measurements (e.g., weather station) vs. the more specific and intense ARM measurements. A daily aggregation of ARM data was used with the Biome-BGC model. The estimates of shortwave radiation based on ARM data were greater than regional estimates for this parameter. This difference was reflected by greater values of Net Ecosystem Production from this model when the ARM data were used as input. More information about the results of the study can be found at http://www.arm.gov/publications/proceedings/conf12/extended_abs/hargrove-ww.pdf .

Another study was started with the SiBD model with hourly data products extracted from ARM. Generating the hourly data files for the SiBD experiments forced us to address gap filling techniques. Results from the GAP filling for the SiBD model were presented in a poster at AGU in 2003 . The SiBD model is still being used to evaluate the usefulness of the soil water and temperature data collected by ARM. Access data products that are useful with SiBD

A final "evaluation exercise" included preliminary efforts to compare the soil water measurements contained in the SWATS data with the water profile estimated by simulation runs with the SIB-D model. This comparison is different type of use for ARM data. This experiment was in process when the project ended. Access SWATS data products and a few very preliminary results

Collaborative interactions

Attending professional meetings

Another approach to developing usage of the ARM data for carbon research included efforts to communicate and collaborate with carbon researchers. These efforts included attending professional meetings (see Table 1). Posters and presentations were presented at most of these meetings. Table 2 lists the references for these "publications."

Table 1 Professional meetings attended on behalf of ARM Carbon Collaborations.

Meeting name	2002	2003	2004
ARM Science Team Meeting	X	X	X
American Geophysical Union	X	X
Ameriflux Meeting		X	X
Ecological Society of America		X
MODIS Modland workshop		X	X
North American Carbon Program Meeting		X	X

Table 2 Posters and presentations at professional meetings.

.	Hargrove, W. W., C. C. Brandt, H. I. Jager, and R. A. McCord. 2002. A "Make-a-Difference" Experiment to Assess the Value of ARM Data in Carbon Cycle Models (poster and extended abstract; http://www.arm.gov/docs/documents/technical/conf_0204/hargrove-ww.pdf ). Twelfth ARM Science Team Meeting Proceedings, April 8-12, 2002, St. Petersburg, Florida
.	Hargrove, W. W., Brandt, C. C., Jager, H. I., Hanan, N., and McCord, R.A. 2003. Characterizing and Filling Temporal and Spatial Gaps in Time-Aggregated ARM Measurements for Use in Carbon Models. (abstract) Thirteenth ARM Science Team Meeting Proceedings, March 31 - April 4, 2003, Broomfield, Colorado.
.	Hargrove, W. W., C. C. Brandt, H. I. Jager, and R. A. McCord. 2002. A "Make-A-Difference" Experiment to Assess the Value of ARM Data in Carbon Cycle Models. Eos Trans., AGU, 83(47), Fall Meet. Suppl., Abstract A12A-0136.
.	McCord, R. A., W. W. Hargrove, H. I. Jager, C. C. Brandt, and N. Hanan. 2003. Characterizing and Filling Data Gaps in ARM Measurements for Carbon Models. Eos Trans. AGU, 84(46), Fall Meet. Suppl., Abstract B22A-0805

Organizing workshop and mixer

We also attempted to have a training workshop on the usage of ARM Carbon data products at the 2003 Ecological Society of America meeting. (This meeting was considered to be significant gathering of carbon researchers that had not been contacted at other meetings.) About 350 invitations to the workshop were sent by email. This email list was composed of participants at other carbon cycle research meetings. However, the registration of the workshop was insufficient and it was not held. A description of the intended workshop (look for "WK -9: Leveraging the Atmospheric Radiation Measurement (ARM) Data Archive for Carbon Modelers")

A professional mixer about ARM Carbon measurements and data products was collaboratively sponsored by ORNL and LBNL at the Fall 2004 AGU meeting. This event was attended by about 25 researchers. The workshop resulted in identification of potential future collaborations between ARM and NASA on space-based carbon observations. The proposed activities would use ARM data from SGP and TWP sites. More information on this proposed future collaboration can be found at these links: at SGP, at TWP

Collaboration with Colorado State University

We established a small consulting subcontract with Dr. Niall Hanan at Colorado State University. The purpose of the contract was to get detailed advice on the design of data products and assistance with interacting with groups of carbon researchers. Dr. Hanan also provided assistance with our use of the SiB model. Future collaborations beyond this project and contract are expected to continue the evaluation of the SiB model with the ARM soil water data.

Review of potential use of ARM facilities NACP activities

We prepared a review of the potential use of the ARM facilities for future North American Carbon Program activities. For several months, the potential of the ARM sites was included in the NACP planning documents. However, subsequent planning resulted in the interests of NACP being focused on other regions. Access the review

Status at the end of the project

In addition to the presentations and evaluations described in the previous sections, we accumulated several data products and data plots. The section provides overview documentation for each of the data products resulting from this project.

SiBD driver files

Data files with hourly statistics derived from ARM measurements and suitable for "driving" the SiB carbon model were generated from most of the ARM SGP extended facilities from 1996 through 2000. These files contain results that are filtered based on QA information and have complete records from "gap filling." These files are documented and available

Monthly Plots of SiB driver parameters

Data plots showing the SiB driver parameters were generated to confirm the reasonableness of the data products and evaluate the sufficiency of the gap filling estimates. These plots are being retained because the large scale (monthly) view of ARM data are not readily accessible from any other sources. These plots are documented and available

Extended statistics from SiB driver parameters

The preparation of the SIB driver files resulting in the merging of measurements from several ARM data streams. More extensive statistics (e.g., mean, min, max, standard deviation, etc.) were computed for each time interval and many related measurements (e.g., secondary radiation measurements). These extended statistics and measurements may be enable future use of ARM data with other carbon cycle models. These data files are available at: <<<< need to add reference after Craig gets the files generated and we have them stored >>>>>

Soil water and temperature files

The ARM program has collected a significant amount of soil water and temperature measurements (hourly data at 19 extended facility locations) since 1996. Data files that could be compared with the soil water estimates included in many carbon models are available. One version of these data is a "best estimate." These data products have been filtered to remove invalid or unreasonable values. The data products have also condensed to present the best single or averaged result from the redundant arrays of sensors at each location. A secondary version of the data product includes the "uestionable" data values with detailed QA flags. Additional documentation Access best estimate data files Access detailed data files

Yearly plots of soil water and temperature files

Data plots showing the soil water and temperature "best estimate" data product were generated to confirm the reasonableness of this data summary. These plots are useful to observe the systematic differences in soil moisture profiles and seasonal trends between the many installations. The data plots also provide a long time scale view of the measurements that is not readily available from other sources. Access plots.

Suggestions for future activities

There are many other applications of ARM data for carbon cycle research that could be pursued in the future. This section provides a brief description of future activities that we had planned to pursue.

Spatially interpolated data products

The spatial spread and intensity of the ARM extended facilities is unusual for the combination of measurements being collected. We suggest that generating spatially interpolated estimates of the measurements and using them with a regional model (multiple points, or area logic) could have interesting results. The addition of routine measurement land surface conditions could result in data set that is not replicated in any other location.

Comparison with latent heat measurements

Latent heat measurements are representative of evapotranspiration. This environmental parameter may have significant relationship to carbon flux. ARM has been collecting this measurement with the EBBR instrument at 9 locations for over 10 years. We considered preparing data products from the ARM EBBR data streams that could be compared to another portion of the carbon cycle models. However, an adjustment is needed to accommodate known spikes in the measurement process. A version of this adjustment was implemented in the ARM Value Added Product (VAP) for the EBBR. However, the implementation of this adjustment has some uncertainty and this VAP is currently not in production.

More use of the soil water and temperature data

The soil water and temperature data are currently being considered for model validation. Another research project could be to use these measurements for input into the models. This would require significant (but possibly beneficial) revisions to most of the current carbon cycle models. Methods to fill data gaps for these types of measurements would also need development.

Integration of ARM Carbon measurements with other ARM data

Measurements collected by the ARM Carbon project at LBNL include CO2 flux and high precision CO2 concentrations. These measurements should be combined with other ARM measurements. A data set similar to Ameriflux data products could be developed from this combined source of measurements. These measurements combined with the new ECOR data (discussed below) have the potential to provide a relatively dense (spatially) collection of this type of measurements. Most of the Ameriflux locations are wide distributed.

Use of future ARM ECOR data

In the Fall of 2003, ARM upgraded all of its Eddy Correlation (ECOR) instruments to include CO2 Flux measurements. (Regrettably the ARM information still does not explain this change, but the CO2 are included in currently generated data files.) These instruments are distributed among 9 of the SGP extended facilities. As this data collection accumulates, these measurements should be useful for comparison with carbon cycle models of the ARM facilities. Because the facilities represent replicates of a few land use types (grassland / pasture, crops, and mixed), the replication and density of these measurements could provide some new research opportunities.

Conclusion

The results of this project are considered "mixed."

Some aspects of the project were positive:

We demonstrated that data products that are useful with carbon cycle models could be produced.
We documented that these data products could contribute to discoveries that were different and perhaps more solidly based than more general data sources for similar information.
A few continuing, collaborative activities related to the combined meteorology, radiation, and soil water measurements have the potential to measurable contributions.

Some aspects of the project were negative:

We were never fully successful in convincing the carbon research community to come and work with our data products (although the level of interest increased within the carbon community right throught the end of the project). Most carbon researchers are already entrenched in studies at other locations. They were unwilling to transfer their research efforts to the ARM locations solely based on the availability of data. We encountered several inquires about "when or where can research proposals be submitted to get funding to work with your data products?" This type of funding was not available.
The project depended on a variation of the assumption that "if you build it, they will come." This assumption does not appear to be valid in this case.

It is possible that future efforts similar to this project may occur in the future. The "additional measurements" portion of the ARM Carbon Program (operated by LBNL) continues. The continuing accumulation of these measurements has the potential to spur future interest in the types of efforts conducted for this project. ARM has also been designated as User Facility and has some expectations to "reach out" to other disciplines with its infrastructure resources. Preparing data products for carbon cycle, water cycle, or other environmental disciplines one possible mechanism for "outreach." The lessons learned from this project should be valuable input to any similar projects in the future.