GTE - Data Archive Format Document

Abstract:

This document describes the GTE Data Archive Format, the native format of all Global Tropospheric Experiment datasets. The information contained in this document is necessary to correctly decode the GTE Archive Datasets.

Table of Contents:

Figures and Tables:

I. General Instructions and Rationale

The data format described herein is designed to provide a file structure that effectively utilizes magnetic storage media for the exchange of data for scientific analysis. Recognizing that the variety of GTE measurement techniques span a large dynamic data range, every attempt has been made to meet the needs of scientists that generate either large or small data sets. The GTE Data Archive Format is intended to satisfy both the data archive and data exchange requirements of the GTE field expeditions.

I.A. Suitable Media:

Files must be in ASCII. All data should be numeric, separated by commas. Each aircraft PI shall submit a minimum of one file per flight. Whenever possible, data shall be submitted to the GTE Data Archive via electronic transfer. Suitable media include File Transfer Protocol (FTP), electronic mail (E-mail), and IBM-compatible diskettes (3.5" High Density preferable). The file size should be compatible with the submitting media limitations. Therefore, if the ASCII file size is greater than 1.44 mb, it should either be compressed before being submitted on 3.5" disks, or it should be submitted via FTP or E-mail. If data are compressed, either the un-compressing programs must be included with the data, or the data should be in executable form (.EXE file extension). Large files can be most easily accommodated via electronic submissions directly into the GTE Archives (Refer to Section V. FTP Protocol for GTE Data Exchange). As mediums of data exchange are constantly being upgraded and amended, and as new mediums of exchange introduced, the GTE Data Archive Office will continue to upgrade and change its data exchange processes when feasible. Any medium of exchange differing from those mentioned herein should be approved by the GTE Data Archive Office prior to data submittal.

I.B. Common File Structure:

A GTE Archive file is composed of two sections: a HEADER SECTION, which contains all the information with which to process the file; and the DATA SECTION, which contains the data in the prescribed GTE Data Archive format. The file naming convention, header and data format descriptions are detailed in Section II. of this document.

In order to facilitate the exchange of data among the GTE investigators and the scientific community at large, the GTE requires that all datasets be referenced by time. Time data is composed of the Julian day of the year and the seconds of the day, both referenced to Greenwich Mean Time (GMT). When a reporting period extends past GMT midnight (86,400 seconds), increment the Julian day of year and reset the time to zero. See Section II.C. Dataset Types for an explanation of the different time reporting options.

Aircraft PI's should submit a file for each aircraft mission. If no data are submitted to the archive for a particular mission, the PI shall in any case submit a file containing the Header section of the GTE Data Archive format. The PI shall use comment lines to explain the reason no data are provided for the mission.

II. Header Record Format

This section contains a detailed description of the format for each header record. The header records should supply all information needed to read and interpret the data records. Even if the actual data are not included in the file, the header records should include the parameters as noted and information needed to identify and locate the data files.

II.A. Definition of Header Records:

Each line of the header section is reserved for specific information. These headers will be scanned by computer programs; therefore, all data must be in the format as described in order to be correctly interpreted. The following sections define each line of the Header Record Format.

Line 1. Number of Lines in the Header (NH)

The first line in the header contains the number of lines in the header section (NH). The first line of data = NH + 1.

Line 2. File Name

The second line in the header contains the name of the dataset file. The data file naming convention for the GTE is as follows:

PPTTTLXX.EXP where:

NOTE: For aircraft investigators, each landing constitutes a new flight. Flights are named by integer numbers only (i.e. there can not be a flight named 21A or 21.1). If there are several flights on a day, each flight will receive a separate numeric identifier.

Examples follow:

Line 3. Principal Investigator

Line 3 contains the Experimenter Last Name, First Name, and Institution. Always separate items by commas.

Line 4. Species Measured / Technique Used

Line 4 contains a brief summary of the species measured and/or the method employed for this sampling. For example, non methane hydrocarbons should be entered here rather than listing each species. Use only one line.

Line 5. Expedition Name

The Official name for this expedition. Please refer to Table 2. for Expedition Names and their abbreviations.

Line 6. Data Start Date, Date Last Revised (YY,MM,DD,YY,MM,DD)

Line 6 contains two dates. Report the start date for this dataset, followed by the date of last revision, both in the format YY,MM,DD

Line 7. Number for this Flight, or Sequential Dataset Number (NF)

Line 7 contains a unique number for this dataset (NF). For aircraft datasets, this number is the number for this flight. For ground datasets, this is usually a sequential number starting at "01" for the first dataset submitted.

Line 8. Number of Variables Reported in this Dataset (NV)

For most standard datasets, the number of variables remains constant for each line in the dataset (Dataset Types 1, 2,4 and 6). In those cases, NV = number of variables for each record of data submitted. Several dataset types are defined which include a matrix of data whose dimensions may vary between dataset records. For these datasets, the matrix variable is considered as a single variable and is always the last variable reported in the list. The variable or variables immediately preceding the dimensioned variable will contain the number of elements in the matrix. In these cases, NV = number of unique variables for each record in the dataset, with the matrix being considered as a single variable. Refer to Figure 3 and Figure 5 for a complete explanation and examples of this variable.

Line 9. Number of Comments (NC)

Line 9 contains the number of comment lines included in the header for this dataset. Comment lines should include any pertinent information needed to help the scientific community correctly interpret the dataset.

Line 10. Dataset Type (DT)

The GTE Data Archive Format currently supports six (6) Standard Dataset Types, numbered from 1 to 6, with 0 = non-standard dataset type. The six Standard Dataset Types are:

Refer to Section II.C. Dataset Types for a complete description and examples of the six dataset types.

Line 11. Data Averaging Period

Report the Data Averaging Period, in SECONDS, in the 11th header record. For a 10-second averaging period, report data at 5-second time ticks; i.e., 5, 15, 25,... GMT seconds. Thus, the data value reported at 15 seconds represents the average value in the interval equal to or greater than 10 seconds and less than 20 seconds.

For 60-second averaged data, the reporting times should be at 30-second time ticks; i.e., 30, 90, 150,... GMT seconds. Thus, the data value reported at 90 seconds represents the average value in the interval equal to or greater than 60 seconds and less than 120 seconds.

For other constant time intervals, time ticks of the data reporting interval should be consistent with the 10-sec or 60-sec format. For a non-standard averaging period, enter "0" for this line.

Line 12. Data Sampling Frequency (Hertz)

Enter the instrument sampling frequency, in hertz. Data recorded at 1 sample/second = 1 Hz; 5 samples/second = 5 Hz; 1 sample every 5 seconds = .2 Hz. ; 1 sample / minute = 1/60 = .0167 Hz. Enter "0" here for instruments having non-standard sampling periods. For Data Type =6, replace this variable with the number of Samples per Trajectory.

Line 13. Definition of First Variable in the Dataset: Var(1)

This variable must be the Julian Day of the Year (GMT) that the sample was taken. (Refer to Section II.C. for further explanation.) A minimum of eight parameters are necessary for each Variable Definition Line. They are:

If Limits of Detection (LOD's) are used in the dataset, then an additional four parameters are added to the end of the above definition line:

Refer to Section II.B. Data Variable Definition for a complete explanation of each element of the variable definition.

Line 14. Definition of Second Variable in the Dataset: Var(2)

The second variable must be a time in seconds, GMT. For Data Type 1, this is the median time of the sampling period. For all other Data Types, this is the Start Time of the sample. (The start time will be null (-999) in the case of Data Type 5). Refer to Section II.B. for further explanation.

Line 15. Definition of Third Variable in the Dataset: Var(3)

Refer to Figure 1., Figure 2., Figure 3., Figure 4., Figure 5., and Figure 6. for any standard definitions for this record.

Line 13+NV-1: Definition of the last variable in the dataset, Var(NV)

Line 13+NV: First line of Comments

This line contains the first line of comments for this header. If NC = 0, then there are no comment lines in the dataset.

Line 13+NV+NC-1: Last line of Comments

This is the last line of comments for this header.

Line 13+NV+NC: First line of Data

This is the first line of data for this dataset. This location is the same as Number of Lines in the Header (NH)+1 (Header line 1 defines NH)

II.B. Data Variable Definition:

Each variable is defined on a separate line in the header beginning with Header Record 13. Each variable definition is composed of a minimum of eight (8) parameters, separated by commas. The eight required parameters are:

If Limits of Detection (LOD's) are used in the dataset, then an additional four parameters are added to the end of the above definition line:

Each item in the Variable Definition line is explained below.

Variable Definition in the Header:

Scale Factors and Offsets must be applied to ALL data submitted to the archives, including data summary values reported in the variable definition section of the header. This includes Minima, Maxima, and Limits of Detection for the data. It is preferable that data be reported in engineering units such that Scale Factors = 1 and Offsets = 0.

Item 5. Minimum value for this variable in this dataset.

The minimum value that is found for this variable for THIS dataset, not an absolute minimum value that could be experienced for this variable. For TIME, use the first sample time reported in the dataset. Minimum value is reported using the Scale Factor and Offsets given for the variable, such that:

ReportedMinimumValue * ScaleFactor + Offset = Minimum Value (Engineering Units)

Item 6. Maximum value for this variable in this dataset.

The maximum value for a variable that is found in THIS dataset. For TIME, use the last sample time reported in the dataset. Maximum value is reported using the Scale Factor and Offsets given for the variable, such that:

ReportedMaximumValue * ScaleFactor + Offset = Maximum Value (Engineering Units)

Item 7. Code for missing or bad data.

This code is composed of a series of negative nines which will be outside the range of actual data for this variable. No scale factoring or offsets are applied.to this code. The code reported here should be to the same number of significant digits as the actual Reported Data Value.

Item 8. Limit of Detection code (LOD Code):

Acceptable values are 0, 1, or 2.

Items 9-12 are used only when Limits of Detection are given (LODCode > 0).

Item 9. Lower Limit of Detection Code (LLOD Code):

This code is composed of a series of negative eights (-8's) which will be outside the range of actual data for this variable. No scale factoring or offsets are applied to this code. The code reported here should be in the same format as the actual Reported Data Value.

Item 10. Lower Limit of Detection (LLOD) (Value or Variable column number)

If LOD Code = 1, LLOD = variable column number containing the actual LOD for this sample. If LOD Code = 2, LLOD = value of the LLOD for this dataset.

LLOD value is reported using the Scale Factor and Offsets given for the variable, such that :

Reported LLOD Value * ScaleFactor + Offset = LLOD Value (Engineering Units)

Item 11. Upper Limit of Detection Code (ULOD Code):

This code is composed of a series of negative sevens (-7's) which will be outside the range of actual data for this variable. No scale factoring or offsets are applied to this code. The code reported here should be to the same number of significant digits as the actual Reported Data Value.

Item 12. Upper Limit of Detection (ULOD) (Value or Variable column number).

If LOD Code = 1, ULOD = variable column number containing the actual LOD for this sample.

If LOD Code = 2, ULOD = value of the ULOD for this dataset.

ULOD value is reported using the Scale Factor and Offsets given for the variable, such that :

Reported ULOD Value * ScaleFactor + Offset = ULOD Value (Engineering Units)

Example of LOD Codes:

This code assumes the Limits of Detection for this flight were changing while in flight. Assume this is Var(7) definition. (Based on 1 to NV variables). Data reported has one significant digit, with a Min = 2.1 and a Max = 155.8.

NOTE: When Var(7) contains real data, Var(8) will contain the Code for Missing Data (-9.9)

LOD Code = 2

This code is used for defining constant LOD's for the entire dataset. Refer to the above section (LOD Code = 1) for general information on the Limits of Detection. In this case, when Var(7) contains LLOD Code (-8.8), the actual Lower LOD for this dataset is the LLOD from the Data Definition line of the header. Likewise, if Var(7) contains ULOD Code (-7.7), ULOD contains the actual Upper LOD.

Like all other data values reported to the archive, scale factors and offsets must be applied to the reported value to arrive at the true data value, in engineering units.

II.C. Dataset Types (DT):

Sample Description 1. (DT = 1): Standard Dataset, Constant time increment, Time reported as the midpoint of sampling time. (See Constant Interval Time Reporting for a discussion of midpoint reporting.) Most instruments using a constant time interval between samples can utilize this dataset type. The first two variables must be:

Refer to Figure 1 for an example of this dataset type.

Sample Description 2. (DT=2): Standard Grab Sample, Times are non-standard. This dataset type should be used by experimenters who take samples at non-constant sampling times and periods. The first four variables must be:

Refer to Figure 2 for an example of this dataset type.

Sample Description 3. (DT = 3): Vertical Profile Sample, with a variable number of points in the sample. Variable "nv" defines the number of variables, with the last variable being an array dimensioned Var(nv). For example, if nv = 8, then variable 8 contains the number to dimension the profile array variable. The first two variables must be:

The final two variables must be:

Refer to Figure 3 for an example of this dataset type.

Sample Description 4. (DT = 4): Vertical Profile Sample, Standard Sonde Data. This dataset type follows the example of DT 1, except that time is not constant. The first two variables must be:

Refer to Figure 4 for an example of this dataset type.

Sample Description 5. (DT = 5): Standard Gridded Data, Gridded Map Data. This dataset type is constructed to enable investigators to submit satellite data to the Archive, for example, AVHRR Fire Count data maps. It is assumed that these datasets are geographically rectified to a given latitude and longitude at a given time. This data type requires the following variables:

			 DIM PixelData(NRows,NCols)
			 FOR i = 1 to NRows
						 ' read in a line of data for a constant Latitude
						 For j = 1 to NCols
							 INPUT filenumber, PixelData(i , j)
						 NEXT j
			 NEXT i
	

The dimensioned variable PixelData will now contain the entire matrix of data for this image. Geographically rectify the image using the Start Latitude and Start Longitude and incrementing each correctly. Refer to Figure 5 for an example of this type of dataset.

Sample Description 6. (DT = 6) : Standard Trajectory Dataset. Trajectory data report samples from a particular air parcel over a finite period of time. Each trajectory in the dataset must cotain the same number of data points, defined by Header Line 12. The first data line contains data from the trajectory end point, corresponding to a selected aircraft time and position. Each following line is the trajectory's position on the prvious time step. Time steps prior to the start time are all normalized according to the Data Averaging Period defined in Header Line 11. Refer to Figure 6. for an example of this type of dataset.

III. Data Record Format

The Data Record section begins on line NH+1 (one line after the header). For each record, data should be reported in the sequence as stated in the Header Record. Each reported value in the dataset utilizes the scale factor and offset for that variable so that

Reported DataValue * ScaleFactor + Offset = Engineering Units

Only numeric values are acceptable as data for the standard archive data types. Please contact the GTE Data Management Office if your data cannot meet this criteria. It is preferable that data be reported in engineering units such that Scale Factors = 1 and Offsets = 0. If data is missing, or is above or below the Limits of Detection (LOD), use the appropriate code as stated in the variable definition for that variable in the header. These special codes should be entered exactly as stated in the Variable Definition Section so that computer programs can correctly determine validity. For example, if the null value is defined as -9.99 for a variable in the Variable Definition Section, that value should be used in the data for the null value. Scale factors and offsets are NOT applied to any codes in the data.

Note that uncertainties associated with a given measurement may be included in the archive as a variable.

III.A. Time Reference:

All data should be referenced to day of the year (Julian Day) and Greenwich Mean Time (GMT seconds of the day). When a reporting period extends past GMT midnight (86,400 seconds), increment the Julian day of year and reset the time to zero. Please refer to Section II.C. Dataset Types for a complete list of required variables for each dataset.

III.A.1 Constant Interval Time Averaging Reporting:

Report the Data Averaging Period in the 11th header record. For a 10-second reporting interval, report data at 5-second time ticks; i.e., 5, 15, 25,... GMT seconds. Thus, the data value reported at 15 seconds represents the average value in the interval equal to or greater than 10 seconds and less than 20 seconds.

For 60-second data (1-minute averages), the reporting times should be at 30-second time ticks; i.e., 30, 90, 150,... GMT seconds. Thus, the data value reported at 90 seconds represents the average value in the interval equal to or greater than 60 seconds and less than 120 seconds. Data reported in 60-second time intervals would be reported in the same format as for 10-second reporting, except the data reporting interval in header record 12 would be 60.

For other constant time intervals, time ticks of the data reporting interval should be consistent with the 10-sec or 60-sec format.

III.A.2 Variable Interval Time Reporting:

Report "0" for the Data Averaging Period in the 11th header record. When reporting intervals are irregular, it is necessary to report the sample start time, end time, and the midpoint time for each record.

IV. Sample Data Files

Header Sample Description 1

Header Sample Description 2

Header Sample Description 3

Header Sample Description 4

Header Sample Description 5

Header Sample Description 6

V. FTP Protocol for GTE Data Exchange

DATA ARCHIVE MEMO: FTP File Transfer to/from the GTE Archives

Some GTE Data Archive datasets are now located on ANONYMOUS FTP at the TYPHOON machine at NASA Langley and may be accessed using standard FTP protocol. Type the quoted (" ") commands to logon and access the various on-line files. Note that you can now submit your data files via FTP. Follow the instructions given below to logon and transfer files to/from the GTE Archives. NOTE: The proprietary archives (those not yet released to the public) are protected from being read by the public. Please notify the Archive Facility prior to accessing these files, so that we may give you a time period for data access.

LOGGING ON TO THE GTE ARCHIVE

LOCATING THE GTE ARCHIVE FILES YOU WISH TO TRANSFER TO YOUR COMPUTER

TRANSFER COPIES OF THE FILES TO YOUR COMPUTER

SENDING FILES TO THE GTE ARCHIVE USING FTP

*** END OF INFO ON FTP TRANSFER ***

Table 1. Principal Investigator Codes

Table 2. GTE Expedition Codes