root/cf-conventions/trunk/docbooksrc/description-of-the-data.xml

<chapter>
  <title>
    Description of the Data
  </title>

  <para>
    The attributes described in this section are used to
    provide a description of the content and the units
    of measurement for each variable. We continue to
    support the use of the 
    <varname>units</varname> 
    and 
    <varname>long_name</varname> attributes
    as defined in COARDS. We extend COARDS by adding the
    optional 
    <varname>standard_name</varname> 
    attribute which is used to provide
    unique identifiers for variables. This is important for
    data exchange since one cannot necessarily identify a
    particular variable based on the name assigned to it by
    the institution that provided the data.
  </para>
  
  <para>
    The 
    <varname>standard_name</varname> 
    attribute can
    be used to identify variables that contain coordinate
    data. But since it is an optional attribute, applications
    that implement these standards must continue to be
    able to identify coordinate types based on the COARDS
    conventions.
  </para>

  <section id="units">
    <title>Units</title>
    <para>
                The <varname>units</varname> attribute is required for all variables 
        that represent dimensional quantities (except for boundary variables 
        defined in <xref linkend="cell-boundaries"/> and climatology variables 
        defined in  <xref linkend="climatological-statistics"/>). The value of 
        the <varname>units</varname> attribute is a string that can be 
        recognized by UNIDATA"s Udunits package <biblioref linkend="udunits"/>, 
        with a few exceptions that are given below. 
        The <ulink url="http://www.unidata.ucar.edu/software/udunits/">Udunits package</ulink> includes a file 
        <filename>udunits.dat</filename>, 
        which lists its supported unit names. Note that case is significant in the <varname>units</varname> strings.
    </para>

    <para>
                The COARDS convention prohibits the unit 
        <constant>degrees</constant> altogether, but this unit is not 
        forbidden by the CF convention because it may in fact be appropriate 
        for a variable containing, say, solar zenith angle. The unit 
        <constant>degrees</constant> is also allowed on coordinate variables 
        such as the latitude and longitude coordinates of a transformed grid. 
        In this case the coordinate values are not true latitudes and 
        longitudes which must always be identified using the more specific 
        forms of <constant>degrees</constant> as described in 
        <xref linkend="latitude-coordinate"/> and <xref linkend="longitude-coordinate"/>.
    </para>

    <para>
      Units are not required for dimensionless quantities. A variable with no units attribute is assumed to be dimensionless. However, a units attribute specifying a dimensionless unit may optionally be included. The Udunits package defines a few dimensionless units, such as <constant>percent</constant>, but is lacking commonly used units such as ppm (parts per million). This convention does not support the addition of new dimensionless units that are not udunits compatible. The conforming unit for quantities that represent fractions, or parts of a whole, is "1". The conforming unit for parts per million is "1e-6". Descriptive information about dimensionless quantities, such as sea-ice concentration, cloud fraction, probability, etc., should be given in the <varname>long_name</varname> or <varname>standard_name</varname> attributes (see below) rather than the <varname>units</varname>.
    </para>

    <para>
                The units <constant>level</constant>, <constant>layer</constant>, and <constant>sigma_level</constant> are allowed for dimensionless vertical coordinates to maintain backwards compatibility with COARDS. These units are not compatible with Udunits and are deprecated by this standard because conventions for more precisely identifying dimensionless vertical coordinates are introduced (see <xref linkend="dimensionless-vertical-coordinate"/>).
    </para>

    <para>
      The Udunits syntax that allows scale factors and offsets to be applied to 
      a unit is not supported by this standard. The application of any scale 
      factors or offsets to data should be indicated by the 
      <varname>scale_factor</varname> and <varname>add_offset</varname> 
      attributes. Use of these attributes for data packing, 
      which is their most important application, 
      is discussed in detail in <xref linkend="packed-data"/>.
    </para>

    <para>
      Udunits recognizes the following prefixes and their abbreviations.
      <table id="table-supported-units" frame="all"><title>Supported Units</title>
        <tgroup cols="7" align="left" colsep="1" rowsep="1">
          <thead>
            <row>
              <entry>Factor</entry>
              <entry>Prefix</entry>
              <entry>Abbreviation</entry>
              <entry></entry>
              <entry>Factor</entry>
              <entry>Prefix</entry>
              <entry>Abbreviation</entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>1e1</entry>
              <entry>deca,deka</entry>
              <entry>da</entry>
              <entry></entry>
              <entry>1e-1</entry>
              <entry>deci</entry>
              <entry>d</entry>
            </row>
            <row>
              <entry>1e2</entry>
              <entry>hecto</entry>
              <entry>h</entry>
              <entry></entry>
              <entry>1e-2</entry>
              <entry>centi</entry>
              <entry>c</entry>
            </row>
            <row>
              <entry>1e3</entry>
              <entry>kilo</entry>
              <entry>k</entry>
              <entry></entry>
              <entry>1e-3</entry>
              <entry>milli</entry>
              <entry>m</entry>
            </row>
            <row>
              <entry>1e6</entry>
              <entry>mega</entry>
              <entry>M</entry>
              <entry></entry>
              <entry>1e-6</entry>
              <entry>micro</entry>
              <entry>u</entry>
            </row>
            <row>
              <entry>1e9</entry>
              <entry>giga</entry>
              <entry>G</entry>
              <entry></entry>
              <entry>1e-9</entry>
              <entry>nano</entry>
              <entry>n</entry>
            </row>
            <row>
              <entry>1e12</entry>
              <entry>tera</entry>
              <entry>T</entry>
              <entry></entry>
              <entry>1e-12</entry>
              <entry>pico</entry>
              <entry>p</entry>
            </row>
            <row>
              <entry>1e15</entry>
              <entry>peta</entry>
              <entry>P</entry>
              <entry></entry>
              <entry>1e-15</entry>
              <entry>femto</entry>
              <entry>f</entry>
            </row>
            <row>
              <entry>1e18</entry>
              <entry>exa</entry>
              <entry>E</entry>
              <entry></entry>
              <entry>1e-18</entry>
              <entry>atto</entry>
              <entry>a</entry>
            </row>
            <row>
              <entry>1e21</entry>
              <entry>zetta</entry>
              <entry>Z</entry>
              <entry></entry>
              <entry>1e-21</entry>
              <entry>zepto</entry>
              <entry>z</entry>
            </row>
            <row>
              <entry>1e24</entry>
              <entry>yotta</entry>
              <entry>Y</entry>
              <entry></entry>
              <entry>1e-24</entry>
              <entry>yocto</entry>
              <entry>y</entry>
            </row>
          </tbody>
        </tgroup>
      </table>
    </para>

  </section>
  <section id="long-name">
    <title>Long Name</title>
    <para>
      The <varname>long_name</varname> attribute is defined by the NUG to contain a long descriptive name which may, for example, be used for labeling plots. For backwards compatibility with COARDS this attribute is optional. But it is highly recommended that either this or the <varname>standard_name</varname> attribute defined in the next section be provided to make the file self-describing. If a variable has no <varname>long_name</varname> attribute then an application may use, as a default, the <varname>standard_name</varname> if it exists, or the variable name itself.
    </para>
  </section>

  <section id="standard-name">
    <title>Standard Name</title>
    <para>
      A fundamental requirement for exchange of scientific data is the ability to describe precisely the physical quantities being represented. To some extent this is the role of the <varname>long_name</varname> attribute as defined in the NUG. However, usage of <varname>long_name</varname> is completely ad-hoc. For some applications it would be desirable to have a more definitive description of the quantity, which would allow users of data from different sources to determine whether quantities were in fact comparable. For this reason an optional mechanism for uniquely associating each variable with a standard name is provided.
    </para>
    
    <para>
                A standard name is associated with a variable via the attribute <varname>standard_name</varname> which takes a string value comprised of a standard name optionally followed by one or more blanks and a standard name modifier (a string value from <xref linkend="standard-name-modifiers"/>).
    </para>

    <para>
      The set of permissible standard names is contained in the standard name table. The table entry for each standard name contains the following:
    </para>

    <variablelist>
      <varlistentry>
        <term>standard name</term>
        <listitem>
          <para>
            The name used to identify the physical quantity. A standard name contains no whitespace and is case sensitive.
          </para>
        </listitem>
      </varlistentry>
      <varlistentry>
        <term>canonical units</term>
        <listitem>
          <para>
                  Representative units of the physical quantity. Unless it is dimensionless, a variable with a <varname>standard_name</varname> attribute must have units which are physically equivalent (not necessarily identical) to the canonical units, possibly modified by an operation specified by either the standard name modifier (see below and <xref linkend="standard-name-modifiers"/>) or by the <varname>cell_methods</varname> attribute (see <xref linkend="cell-methods"/> and <xref linkend="appendix-cell-methods"/>).
          </para>
        </listitem>
      </varlistentry>
      <varlistentry>
        <term>description</term>
        <listitem>
          <para>
            The description is meant to clarify the qualifiers of the fundamental quantities such as which surface a quantity is defined on or what the flux sign conventions are. We don"t attempt to provide precise definitions of fundumental physical quantities (e.g., temperature) which may be found in the literature.
          </para>
        </listitem>
      </varlistentry>
    </variablelist>    

    <para>
      When appropriate, the table entry also contains the corresponding GRIB parameter code(s) (from ECMWF and NCEP) and AMIP identifiers.
    </para>

    <para>
        The standard name table is located at 
        <ulink url="http://cf-pcmdi.llnl.gov/documents/cf-standard-names/standard-name-table/current/cf-standard-name-table.xml">http://cf-pcmdi.llnl.gov/documents/cf-standard-names/standard-name-table/current/cf-standard-name-table.xml</ulink>
        , written in compliance with the XML format, as described in 
        <xref linkend="standard-name-table-format"/>. 
        Knowledge of the XML format is only necessary for application 
        writers who plan to directly access the table. A formatted text 
        version of the table is provided at 
        <ulink url="http://cf-pcmdi.llnl.gov/documents/cf-standard-names/standard-name-table/current/standard-name-table">http://cf-pcmdi.llnl.gov/documents/cf-standard-names/standard-name-table/current/standard-name-table</ulink>
        , and this table may be consulted in order to find the standard 
        name that should be assigned to a variable. Some standard names 
        (e.g. <varname>region</varname> and <varname>area_type</varname>) 
        are used to indicate quantities which are permitted to take only 
        certain standard values. This is indicated in the definition of the 
        quantity in the standard name table, accompanied by a list or a link 
        to a list of the permitted values.
    </para>

    <para>
                        Standard names by themselves are not always sufficient to describe a quantity. For example, a variable may contain data to which spatial or temporal operations have been applied. Or the data may represent an uncertainty in the measurement of a quantity. These quantity attributes are expressed as modifiers of the standard name. Modifications due to common statistical operations are expressed via the <varname>cell_methods</varname> attribute (see <xref linkend="cell-methods"/> and <xref linkend="appendix-cell-methods"/>). Other types of quantity modifiers are expressed using the optional modifier part of the <varname>standard_name</varname> attribute. The permissible values of these modifiers are given in <xref linkend="standard-name-modifiers"/>.
    </para>
    
    <example>
    <title>Use of <varname>standard_name</varname></title>
      <programlisting>
float psl(lat,lon) ;
  psl:long_name = "mean sea level pressure" ;
  psl:units = "hPa" ;
  psl:standard_name = "air_pressure_at_sea_level" ;
      </programlisting>
        <para>
          The description in the standard name table entry for <varname>air_pressure_at_sea_level</varname> clarifies that "sea level" refers to the mean sea level, which is close to the geoid in sea areas.
        </para>
    </example>

    <para>
      Here are lists of equivalences between the CF standard names and the standard names from the 
      <ulink url="http://cf-pcmdi.llnl.gov/documents/cf-standard-names/ecmwf-grib-mapping">ECMWF GRIB tables</ulink>,  the 
      <ulink url="http://cf-pcmdi.llnl.gov/documents/cf-standard-names/ncep-grib-code-cf-standard-name-mapping">NCEP GRIB tables</ulink>, and the 
      <ulink url="http://cf-pcmdi.llnl.gov/documents/cf-standard-names/pcmdi-name-cf-standard-name-mapping">PCMDI tables</ulink>.
    </para>
  </section>

  <section id="ancillary-data">
    <title>Ancillary Data</title>
    <para>
                When one data variable provides metadata about the individual values of another data variable it may be desirable to express this association by providing a link between the variables. For example, instrument data may have associated measures of uncertainty. The attribute <varname>ancillary_variables</varname> is used to express these types of relationships. It is a string attribute whose value is a blank separated list of variable names. The nature of the relationship between variables associated via <varname>ancillary_variables</varname> must be determined by other attributes. The variables listed by the <varname>ancillary_variables</varname> attribute will often have the standard name of the variable which points to them including a modifier (<xref linkend="standard-name-modifiers"/>) to indicate the relationship.
    </para>

    <example><title>Instrument data</title>
      <programlisting>
  float q(time) ;
    q:standard_name = "specific_humidity" ;
    q:units = "g/g" ;
    q:ancillary_variables = "q_error_limit q_detection_limit" ;
  float q_error_limit(time)
    q_error_limit:standard_name = "specific_humidity standard_error" ;
    q_error_limit:units = "g/g" ;
  float q_detection_limit(time)
    q_detection_limit:standard_name = "specific_humidity detection_minimum" ;
    q_detection_limit:units = "g/g" ;
      </programlisting> 
    </example>
  </section>

  <section id="flags">
    <title>Flags</title>
    <para>
      The attributes <varname>flag_values</varname><emphasis role="newtext">, 
      <varname>flag_masks</varname></emphasis> and <varname>flag_meanings</varname> 
      are intended to make variables that contain flag values self describing. 
      <emphasis role="newtext">Status codes and Boolean (binary) condition flags
      may be expressed with different combinations of <varname>flag_values</varname>
      and <varname>flag_masks</varname> attribute definitions.</emphasis>      
    </para>
    <para>
      <emphasis role="newtext">The <varname>flag_values</varname> and <varname>flag_meanings</varname>
      attributes describe a status flag consisting of mutually exclusive coded values.</emphasis>
      The <varname>flag_values</varname> attribute is the same type as the variable to which 
      it is attached, and contains a list of the possible flag values. 
      The <varname>flag_meanings</varname> attribute is a string whose value is a blank 
      separated list of descriptive words or phrases, one for each flag value. 
      If multi-word phrases are used to describe the flag values, then the words within 
      a phrase should be connected with underscores. <emphasis role="newtext">The following example illustrates
      the use of flag values to express a speed quality with an enumerated status code.</emphasis>
    </para>
    <example><title>A flag variable<emphasis role="newtext">, using <varname>flag_values</varname></emphasis></title>
      <programlisting>
  byte current_speed_qc(time, depth, lat, lon) ;
    current_speed_qc:long_name = "Current Speed Quality" ;
    <emphasis role="newtext">current_speed_qc:standard_name = "sea_water_speed status_flag" ;</emphasis>
    current_speed_qc:_FillValue = -128b ;
    current_speed_qc:valid_range = <emphasis role="newtext">0b, 2b</emphasis><emphasis role="deletedtext">-127b, 127b</emphasis> ;
    current_speed_qc:flag_values = 0b, 1b, 2b ;
    current_speed_qc:flag_meanings = "quality_good sensor_nonfunctional 
                                      outside_valid_range" ;
      </programlisting>
    </example>
    <para>
      <emphasis role="newtext">The <varname>flag_masks</varname> and <varname>flag_meanings</varname> 
      attributes describe a number of independent Boolean conditions using bit field notation by setting 
      unique bits in each <varname>flag_masks</varname> value.  <varname>The flag_masks</varname> attribute 
      is the same type as the variable to which it is attached, and contains a list of values matching unique 
      bit fields.  The <varname>flag_meanings</varname> attribute is defined as above, one for each 
      <varname>flag_masks</varname> value.  A flagged condition is identified by performing a bitwise AND
      of the variable value and each <varname>flag_masks</varname> value; a non-zero result indicates a 
      <varname>true</varname> condition.  Thus, any or all of the flagged conditions may be <varname>true</varname>, 
      depending on the variable bit settings. The following example illustrates the use of <varname>flag_masks</varname>
      to express six sensor status conditions.</emphasis>
    </para>
    <example>
      <title><emphasis role="newtext">A flag variable, using <varname>flag_masks</varname></emphasis></title>
      <programlisting><emphasis role="newtext">
  byte sensor_status_qc(time, depth, lat, lon) ;
    sensor_status_qc:long_name = "Sensor Status" ;
    sensor_status_qc:_FillValue = 0b ;
    sensor_status_qc:valid_range = 1b, 63b ;
    sensor_status_qc:flag_masks = 1b, 2b, 4b, 8b, 16b, 32b ;
    sensor_status_qc:flag_meanings = "low_battery processor_fault
                                      memory_fault disk_fault
                                      software_fault
                                      maintenance_required" ;</emphasis>
      </programlisting>
    </example>
    <para>
      <emphasis role="newtext">The <varname>flag_masks</varname>, <varname>flag_values</varname> and 
      <varname>flag_meanings</varname> attributes, used together, describe a blend of independent Boolean 
      conditions and enumerated status codes.  The <varname>flag_masks</varname> and <varname>flag_values</varname>
      attributes are both the same type as the variable to which they are attached.  A flagged condition 
      is identified by a bitwise AND of the variable value and each <varname>flag_masks</varname> value; 
      a result that matches the <varname>flag_values</varname> value indicates a <varname>true</varname> 
      condition.  Repeated <varname>flag_masks</varname> define a bit field mask that identifies a number 
      of status conditions with different <varname>flag_values</varname>.  The <varname>flag_meanings</varname>
      attribute is defined as above, one for each <varname>flag_masks</varname> bit field and 
      <varname>flag_values</varname> definition.  Each <varname>flag_values</varname> and 
      <varname>flag_masks</varname> value must coincide with a <varname>flag_meanings</varname> value.  
      The following example illustrates the use of <varname>flag_masks</varname> and <varname>flag_values</varname>
      to express two sensor status conditions and one enumerated status code.</emphasis>
    </para>
    <example>
      <title><emphasis role="newtext">A flag variable, using <varname>flag_masks</varname> and <varname>flag_values</varname></emphasis></title>
      <programlisting><emphasis role="newtext">
  byte sensor_status_qc(time, depth, lat, lon) ;
    sensor_status_qc:long_name = "Sensor Status" ;
    sensor_status_qc:_FillValue = 0b ;
    sensor_status_qc:valid_range = 1b, 15b ;
    sensor_status_qc:flag_masks = 1b, 2b, 12b, 12b, 12b ;
    sensor_status_qc:flag_values = 1b, 2b, 4b, 8b, 12b ;
    sensor_status_qc:flag_meanings =
         "low_battery
          hardware_fault
          offline_mode calibration_mode maintenance_mode" ;</emphasis>
      </programlisting>
    </example>
    <para>
      <emphasis role="newtext">In this case, mutually exclusive values are blended with Boolean values 
      to maximize use of the available bits in a flag value.  The table below represents the four binary 
      digits (bits) expressed by the <varname>sensor_status_qc</varname> variable in the previous 
      example.</emphasis>
    </para>
    <para>
      <emphasis role="newtext">Bit 0 and Bit 1 are Boolean values indicating a low battery condition 
      and a hardware fault, respectively. The next two bits (Bit 2 and Bit 3) express an enumeration 
      indicating abnormal sensor operating modes.  Thus, if Bit 0 is set, the battery is low and if 
      Bit 1 is set, there is a hardware fault - independent of the current sensor operating mode.</emphasis>
    </para>
    <table frame="all"><title><emphasis role="newtext">Flag Variable Bits (from Example)</emphasis></title>
      <tgroup cols="4" align="left" colsep="1" rowsep="1">
        <colspec colwidth="50pt"/>
        <colspec colwidth="50pt"/>
        <colspec colwidth="50pt"/>
        <colspec colwidth="50pt"/>
        <thead>
          <row>
            <entry><emphasis role="newtext">Bit 3 (MSB)</emphasis></entry>
            <entry><emphasis role="newtext">Bit 2</emphasis></entry>
            <entry><emphasis role="newtext">Bit 1</emphasis></entry> 
            <entry><emphasis role="newtext">Bit 0 (LSB)</emphasis></entry> 
          </row>
        </thead>
        <tbody>
          <row>
            <entry></entry>
            <entry></entry>
            <entry><emphasis role="newtext">H/W Fault</emphasis></entry>
            <entry><emphasis role="newtext">Low Batt</emphasis></entry>
          </row>
        </tbody>
      </tgroup>
    </table>
    <para>
      <emphasis role="newtext">The remaining bits (Bit 2 and Bit 3) are decoded as follows:</emphasis>
    </para>
    <table frame="all"><title><emphasis role="newtext">Flag Variable Bit 2 and Bit 3 (from Example)</emphasis></title>
      <tgroup cols="3" align="left" colsep="1" rowsep="1">
        <colspec colwidth="50pt"/>
        <colspec colwidth="50pt"/>
        <colspec colwidth="100pt"/>
        <thead>
          <row>
            <entry><emphasis role="newtext">Bit 3</emphasis></entry>
            <entry><emphasis role="newtext">Bit 2</emphasis></entry> 
            <entry><emphasis role="newtext">Mode</emphasis></entry> 
          </row>
        </thead>
        <tbody>
          <row>
            <entry><emphasis role="newtext">0</emphasis></entry>
            <entry><emphasis role="newtext">1</emphasis></entry>
            <entry><emphasis role="newtext">offline_mode</emphasis></entry>
          </row>
          <row>
            <entry><emphasis role="newtext">1</emphasis></entry>
            <entry><emphasis role="newtext">0</emphasis></entry>
            <entry><emphasis role="newtext">calibration_mode</emphasis></entry>
          </row>
          <row>
            <entry><emphasis role="newtext">1</emphasis></entry>
            <entry><emphasis role="newtext">1</emphasis></entry>
            <entry><emphasis role="newtext">maintenance_mode</emphasis></entry>
          </row>
        </tbody>
      </tgroup>
    </table>
    <para>
      <emphasis role="newtext">The "12b" flag mask is repeated in the <varname>sensor_status_qc</varname>
      <varname>flag_masks</varname> definition to explicitly declare the recommended bit field masks to 
      repeatedly AND with the variable value while searching for matching enumerated values. An application 
      determines if any of the conditions declared in the <varname>flag_meanings</varname> list are 
      <varname>true</varname> by simply iterating through each of the <varname>flag_masks</varname> and 
      AND'ing them with the variable. When a result is equal to the corresponding <varname>flag_values</varname>
      element, that condition is <varname>true</varname>. The repeated <varname>flag_masks</varname> enable 
      a simple mechanism for clients to detect all possible conditions.</emphasis>
    </para>
  </section>
</chapter>