The form of the national configuration data set has been designed in a joint effort between the NWS and FSL, and is still under development. The national configuration data set currently contains information about counties, cities, zones, topography, rivers, etc., and is maintained at NWS headquarters in cooperation with the local offices. Changes to these basic data sets will be distributed to each office. Each office is able to incorporate these changes into all of their local configuration data sets by running a single script.
The real beginnings of the localization process (although at the time it was not called that) were when we removed these dependent configuration data sets from source control and began generating them on a regular basis in our software builds. What this did, in effect, was to generate a default Denver localization each time we did a software build. This created a problem in that our software directory structure was designed to handle the interrelationships in our source code well, but it did not well reflect the interrelationships between our dependent data sets and the programs that created them. As such the data targets in our software builds got to be very messy, so some sort of separate process for generating the dependent data sets would have eventually been developed even if national deployment was not an issue.
When it became known that the WFO-Advanced system was going to be deployed nationwide, it was immediately apparent that developing an automated localization process independent of our software build process was absolutely essential. Design for this started in October 1996 and development began in December that year. By mid January 1997, enough of the localization process was working that we were able to remove the default Denver localization from the software build process. In late January the first of the data sets provided by NWS headquarters were incorporated into the localization process. The WFO-Advanced system that went to the AMS conference in February 1997 was the first deployment of a WFO-Advanced system that relied on the localization process for its configuration data. The WFO-Advanced upgrade installed in the Denver WFO at the end of February 1997 was based on the localization process, as was the first major software hand-off the NWS in March 1997. In October 1997, a formal methodology for including local customization was developed. As of this writing, the basic structure of the localization scripts is complete, and nearly all the major data sets that make up the national configuration data set are implemented.
In our source code control system in $FXA_HOME/src, there is a subdirectory called `localization' where all of the scripts, national data, and local data used by the localization process reside. Source code for programs used by the localization scripts are distributed amongst the other source directories; the localization directory only contains scripts, configuration data, and documentation.
There are five subdirectories in the localization source directory; `scripts,' `nationalData,' `documentation,' `localData,' and `realizations.' As the names would suggest, the scripts directory is where the localization scripts reside, the nationalData directory is where the national configuration data set resides, and the documentation directory is where user and developer documentation is kept. The localData directory is where the site-specific control files are kept, and the realizations directory is where special site specific control files that change the behavior of groups of localizations are kept.
Running the `data' target in the localization source directory causes all of the files needed to create localizations to be moved to the directory $FXA_HOME/data/localization. When localizations are created, they use the data and scripts from $FXA_HOME/data/localization, not from the source tree. This is because it is necessary to create localizations in the field, and field sites will not have a source tree, only $FXA_HOME/bin and $FXA_HOME/data. It is important to remember that the act of running the data target in the localization source directory only moves files over to the directory $FXA_HOME/data/localization; it does not create a usable localization.
Upon running the data target in the localization source directory, the files in the nationalData, scripts, and documentation subdirectories get moved directly to subdirectories of the same name in the directory $FXA_HOME/data/localization.
The files in src/localization/localData get exploded into a directory structure. The name of every site specific control file in localData looks like LLL-blahblah.blah, where LLL is the identifier of a localization (not necessarily 3 characters long). A file in src/localization/localData called LLL-blahblah.blah ends up being moved to the directory named data/localization/LLL by the data target in the localization source directory. The reason for this is that while it is painful to add directories in our source code control system, we want what is installed in the field to have a separate site control directory for each localization. Because a `-' is used as the delimiter that separates the localization identifier from the rest of the file name for site specific control files, one should never try to use a `-' in a localization identifier.
Files in src/localization/realizations behave similarly. The name of every realization file in realizations looks like RRR--blahblah.blah, where RRR is now the realization identifier. Such a file ends up in data/localization/realizations/RRR. Two `-' characters are used as the delimiter so it is harder to mistake a realization file from a local site specific file. Realizations are mentioned here so that the reader will understand their general function. Not much else will be said about them until much later because most localizations are not associated with a realization.
In order to make use of the results of the localization process, all of the WFO advanced workstation and data ingest processes use a single software module, called InfoFileServer, to locate static metadata files. This module always checks a series of predefined directories, some specific to a given localization and some generic, in a well defined order when trying to locate a file. Incorporated into most of the software that actually reads static metadata files is the ability to interpret C/C++ style #include statements, along with automatic environment variable translation in these statements. This means that any new software written that reads static metadata files should always use the InfoFileServer class to locate these files.
All of the discussion in the next two sections is predicated on the user having access to a full build of the WFO Advanced software tree.
The NWS has provided us with a file that contains information about county warning areas. The identifiers of all county warning areas are automatically defined to be valid localization identifiers. One can produce a list of these identifiers by running the command
It is also possible to define additional localizations by creating certain types of site specific control files. As the reader will recall from the previous section, site specific control files exist in the source tree in the directory src/localization/localData, have file names that look like LLL-blahblah.blah, and get moved to data/localization/LLL when the software tree is built. A localization can be defined with a site specific control file that is named either LLL-mainConfig.txt or LLL-cwa.asc. The cwa.asc file is just a one line file with a latitude and longitude to center the WFO scale around. Files with names that look like LLL-blahblahConfig.txt contain what are referred to as directives (see directives.html), which are lines in a file that begin with @@@. In order for a mainConfig.txt file to properly define a localization, it must contain a @@@WFO directive, a @@@CLONE directive, or a @@@REALIZATION directive. The argument of a @@@WFO directive must be a county warning area identifier, and the argument of a @@@CLONE directive must be the identifier of some other defined localization which is not itself a clone (also see 6.1). The argument of a @@@REALIZATION directive must be the identifier of a correctly defined realization (also see 8.0).
Here we have only mentioned two possible types of site specific control files and two possible directives; there will be much more on this later.
FXA_NATL_CONFIG_DATA = ${FXA_HOME}/data/localization
FXA_LOCALIZATION_SCRIPTS = $FXA_HOME/data/localization/scripts
FXA_LOCALIZATION_ROOT = $FXA_HOME/data/localizationDataSets
FXA_LOCAL_SITE = LLL
FXA_INGEST_SITE = III
FXA_NATL_CONFIG_DATA points to where the source data for localizations
reside, and FXA_LOCALIZATION_ROOT points to where the finished
localization subdirectories go. FXA_LOCALIZATION_SCRIPTS is where all
of the localization scripts are run from. FXA_LOCAL_SITE is meant to
point to the localization currently being used, and FXA_INGEST_SITE is
meant to point to the localization currently being run on the data
server.There is also a bare minimum set of programs that must be built in order for one to run all tasks of a localization. Programs not needed to run the default list of tasks are noted with an asterisk. The function of these programs will be described in detail later. For now, the list of programs is:
$FXA_HOME/src/dm/shapefile/shp2bcd $FXA_HOME/src/dm/grid/initCdlTemplate $FXA_HOME/src/dm/grid/makeGridKeyTables $FXA_HOME/src/dm/grid/processStyleInfo $FXA_HOME/src/dm/grid/testGridKeyServer $FXA_HOME/src/dataMgmt/fileMover $FXA_HOME/src/dataMgmt/keyMunge $FXA_HOME/src/dataMgmt/pasteUtil $FXA_HOME/src/util/textBufferTest $FXA_HOME/src/geoLib/bcdProc $FXA_HOME/src/geoLib/maksuparg $FXA_HOME/src/geoLib/test_grhi_remap $FXA_HOME/src/geoLib/maksubgrid $FXA_HOME/src/geoLib/rangeAzimuth $FXA_HOME/src/mapping/testDepictorTable $FXA_HOME/src/mapping/makthermo $FXA_HOME/src/mapping/makxsect $FXA_HOME/src/staticPlotData/va_driver $FXA_HOME/src/staticPlotData/masterToGoodness $FXA_HOME/src/geoTables/newGELTmaker $FXA_HOME/src/geoTables/makeGeoTables $FXA_HOME/src/geoTables/GELTtest $FXA_HOME/src/geoTables/image_mask $FXA_HOME/src/dm/point/reformatTest $FXA_HOME/src/tstorm/localize/create_radarLoc * $FXA_HOME/src/tstorm/localize/sitefinder.c *In an environment with no source tree, all of these programs will, of course, be in $FXA_HOME/bin.
In order to actually create a localization, go to the directory $FXA_HOME/data/localization/scripts and issue the command `mainScript.csh LLL LLL'. This will build the localization with the identifier LLL. mainScript.csh takes 5-20 minutes to run, depending on the type of machine one is running on and the complexity of the localization. It will produce quite a few diagnostics, which will be discussed later. The files for this localization are put in a directory named $FXA_HOME/data/localizationDataSets/LLL.
To run a workstation or an ingest process, the environment variable FXA_LOCAL_SITE must be set to the identifier of a successfully created localization. This is because there is a single software module used throughout the WFO Advanced software to find static metadata files. When asked to find a file, it first looks in $FXA_LOCALIZATION_ROOT/$FXA_LOCAL_SITE, then in $FXA_NATL_CONFIG_DATA/nationalData, then in $FXA_HOME/data, and if all these fail it will try the current working directory.
The environment variable FXA_INGEST_SITE is important because it is possible for a data server to be running one localization when a workstation is running a different one. In general, the data server will run the same localization over time, where workstations might run different localizations. If, when building a localization, one issued the command `mainScript.csh LLL III', this will build a localization named LLL on the assumption that its data server is using the localization named III. Issuing the command `mainScript.csh' is the same as issuing the command `mainScript.csh $FXA_LOCAL_SITE $FXA_INGEST_SITE', and the command `mainScript.csh LLL' will produce the same result as running the command `mainScript.csh LLL $FXA_INGEST_SITE'. What this means is that FXA_INGEST_SITE is set to be whatever localization the data server is running, so that when localizations are built using a single localization identifier, they will be able to properly interpret data from the data server.
The script mainScript.csh is mostly an executive; it delegates the creation of localization data sets to subordinate scripts. mainScript.csh verifies that the correct localization environment exists and that the localization that the user is attempting to build is viable. mainScript.csh also performs a couple of functions that are hard to isolate in a single subordinate scripts; assuring that any out-of-date mapping tables are disposed of and removing any files created in the localization data set that are identical to files in $FXA_HOME/data or nationalData/. It also allows the user to select which portions of the entire localization function are to be performed.
The usage of mainScript.csh is as follows:
The `n' option must be used if one is changing the customization environment for an existing localization. This means when the value of the environment variables FXA_CUSTOM_FILES or FXA_CUSTOM_VERSION changes. See section 9.0 on customization for more information about this. The `n' flag must also be used if one wants to rerun an existing localization using a different ingest site.
There is now some logic in localization that allows it to detect when certain files are up to date and thus avoid recreating those files. If the `f' option is present, this logic is disabled.
When mainScript.csh runs, it will echo to the user the list of localization tasks it is performing. Also, it will tell the user which subordinate script it is currently running, and some of the subordinate scripts produce some diagnostics of their own. The `v' option will cause it to echo individual commands executed in the subordinate scripts. Currently, the complete list of default task options is as follows:
A task option of `-all' will cause all default and non-default tasks to run. Additionally, the arguments `-WS', `-DS', and `-AS' will result in running only those tasks absolutely necessary for localizations that reside on the workstation, data server, and application server, respectively. The task -WWA (upper case as opposed to lower case) will run just enough localization tasks to support running warnGen for a localization id that has not yet been run. A leading `t' option will cause it to only verify that the localization id selected is valid, list the tasks that would be run, and verify the path to each of the subordinate scripts being used.
The task selection functionality of mainScript.csh would be subverted should the user choose to define localization identifiers that begin with a `-' or `+', so this should never be done.
The user should note that the task options cannot change the order in which the subordinate scripts are run; it can only specify which subordinate scripts are run. This is because some tasks are dependent on previous tasks to function correctly. In general, one can expect to get meaningful results from running a given task only when the all of the preceding tasks have been successfully completed.
Within mainScript.csh it will also try to run subordinate scripts called test1.csh, test2.csh, or test3.csh if the task options `-test1', `-test2', or `-test3' are supplied. The user can place a test script by any of these names into $FXA_LOCALIZATION_SCRIPTS and have them run within the localization environment. These tasks always run after the regular tasks have completed, and will not be activated by the `-all' task option. Another task that behaves just like the test tasks is (run last, not with -all) is the `fixGeo' task. Running this task on the data server will remedy the situation where the template files in the gridded data directories do not contain the correct geographic information.
The environment variable FXA_LOCALIZATION_LOG, if set, will cause the diagnostic output from mainScript.csh to be written to that file. Also, this will cause additional diagnostic output to be generated, including lists of files changed since the last time a localization was run. Using the `t' or `v' flags will cause logging to be turned off.
Sections 4.1.3 through 4.1.13 will talk about all of the subordinate scripts in turn.
4.1.2) Some generic utility scripts
There are three generic utility scripts that are used by virtually every subordinate script to help do their work. getPath.csh is a generic file finding script that is very analogous to the module InfoFileServer found in $FXA_HOME/src/foundation. fileGrab.csh is used to move groups of site specific control files with similar names from $FXA_HOME/data/localization to their proper place in $FXA_HOME/data/localizationDataSets. doPatches.csh is actually meant to be sourced rather than run as an independent script. In a subordinate script named blahblahblah.csh, sourcing doPatches.csh will cause the site specific control file LLL-blahblahblah.patch to be sourced if it exists. This allows an easy way to add functionality to subordinate scripts that cannot be done with a file replacement operation. There is also an analog to doPatches.csh called doPatchesI.csh that can be sourced to do the same thing based on the ingest localization instead of the display localization.
There are four other generic utility scripts that are not as widely used, but nonetheless are very important. The script configValue.csh is used to obtain the value of directives. The script newerUtil.csh provides a means to determine if a dependent file is older that the file it is created from, and thus must be recreated. The script scaleSup.csh is used to get the geographic information file (depictor file) for a given scale index. The script stdErr.ksh gives a unified way of sending text to standard error; this is the only way to produce diagnostics in scripts that output data to standard output as their primary function.
The script makeDataSups.csh is controlled by the `dataSups' task option. The purpose of this script is to create all of the geographic information files (depictor files) that describe the geographic characteristics of various data sets. In order to do its job, this script makes use of the programs maksuparg and makthermo and the script raobUtil.csh.
The script makeScales.csh is controlled by the `scales' task option. The purpose of this script is to create all of the geographic information files (depictor files) that describe the geographic characteristics of displays. This includes display scales, of course, but also includes the default state of the movable cross section, time section, and sounding depictors. It is also where the scaleInfo.txt file is handled. In order to do its job, this script makes use of the programs maksuparg, makxsect, makthermo, and rangeAzimuth. This is the only subordinate script where the generic .patch file replaces functionality instead of adding functionality when available and sourced.
Unlike the other subordinate scripts in the localization process, this one has an optional command argument. The optional argument is meant to be a localization identifier. When the optional argument is supplied, the depictor files created for each scale will be tagged with that localization identifier instead of named generically.
The script makeClipSups.csh is controlled by the `clipSups' task option. When the `clipSups' task is invoked, makeScales.csh is first called with the ingest localization identifer ($FXA_INGEST_SITE by default) as an argument. This creates a set of scale depictors that are specifically identified as being associated with the localization that is assumed to be running on the data server. Then, makeClipSups.csh is called, which actually generates those scale dependent data set depictor files that are other than just tagged scale depictors, such as the one for the clipped mesoEta grids. This is necessary because some data sets are created in a manner that is dependent on a display scale, such as the regional satellite sector, the mesoEta grids, or the LAPS grids. If a workstation were running a different localization than the data server and only had access to its own generically named depictor files, it would attempt to map these clipped data sets as if they were clipped according to its own localization, not the data server's. The user should note that it is within this second invocation of makeScales.csh that the decision is made as to whether to use east or west satellite.
The script assembleTables.csh is controlled by the `tables' task option. This script performs many functions. It sees to it that files containing the manually defined data and depictable keys are in place, and creates keys for satellite products. Satellite keys are adjusted for east vs west CONUS sectors and to use the regional clip area of the data server through the inclusion of tagged depictor files. assembleTables.csh grabs files containing locally defined data and depictable keys as well. For product buttons, it also verifies that the default manually defined version of that table is in place and grabs the file with locally defined buttons, if needed. For menus, it makes use of the raobUtil.csh script to construct a local RAOB menu with about 5 to 10 close to the center of the localization, and to place the 2 to 4 closest RAOBs directly on the upper air menu. It sees to it that the default menus are in place and will move any locally defined menus into the localization data set. assembleTables.csh is where file override is implemented for design files and lookup table files, which control plan view plotting (see adaptivePlanViewPlotting.html for more info). This script is also where color tables are handled.
The script makeTextKeys.csh is controlled by the `text' task option. Using the textUtil.csh script, makeTextKeys.csh determines the proper AFOS CCC to use and places it in the fxa.config file. It also does distance based assignment of the number of versions of certain text products to keep, and automatically generates data and depictable keys for text depictables that only use some regional or local text products.
The Local Analysis and Prediction System (LAPS) is a high resolution hourly surface and 3D analysis that normally runs on the application server. LAPS is treated more or less as a COTS by the WFO advanced system, but LAPS still needs some localization generated geographic information in order to be properly configured. Invoking the non-default `laps' task option will result in LAPS being configured to run over the local area associated with the ingest localization.
The script makeTopoFiles.csh is controlled by the `topo' task option. This script makes the topography data for high resolution topographic images and for those gridded data sources whose geography is dependent on the location of the display scales. This script has 2.5 minute world topography and 1km US topography at its primary data sets to derive other topography data from. It also has 1km Alaska, 30 second Pacific and 30 second Carribean data sets available. For creating certain data sets, the maksuparg program is used to determine whether the area in question overlaps the US, Pacific, Alaska, or Carribean topography, otherwise it will use the world topography. The program test_grhi_remap is what is used to make the derived data sets. Topography data for gridded data sources can also be made available by moving *.topo files from site specific control files into the localization data set.
4.1.10) makeGridSourceTable.csh and updateGridFiles.csh
The scripts makeGridSourceTable.csh and updateGridFiles.csh are controlled by the `grids' task option. These two scripts collectively perform all the tasks necessary to make gridded data available through the volume browser. The script makeGridSourceTable.csh, whose job it is to create the final version of gridSourceTable.txt, runs before any other task because other tasks need that file. The file gridSourceTable.txt, defines some characteristics for gridded data sources and specifies which ones are currently active.
The script updateGridFiles.csh is used to create template files from the .cdl files for active gridded data sources, and build the depictable keys and data keys associated with gridded data. updateGridFiles.csh also creates style information (for example, contour intervals) from style rules files, generates volume browser selection menus for sources and cross sections, and generates the depictor files that represent the predefined lat/lon cross section baselines. updateGridFiles.csh uses the program testGridKeyServer to generate lat/lon baselines and volume browser menus, the program ncgen to make .cdlTemplate files, the program initCdlTemplate to place geographically dependent static information into those .cdlTemplate files, the program makeGridKeyTables to create the depictable and data keys, and the program processStyleInfo to generate style information.
This script has the capability to utilize non-default versions of the files dataFieldTable.txt, dataLevelTypeTable.txt, gridPlaneTable.txt virtualFieldTable.txt, and the *.rules files. How these files are used to manage gridded data and the volume browser is a complex enough subject that it is treated separately in the documents gridTables.html and styleRules.html.
The script updateRadarFiles.ksh is controlled by the `radar' task option. This script is used to create the data, depictable, product button and multi-load keys associated with nexrad radar data, as well as the needed menu structures. This is the only subordinate script written in the korn shell, and is probably the most complex, as it has six subordinate scripts of its own.
There are four files that control which radars are ingested and displayed and on which scales, all of which will have a default version generated if an override version is not supplied. radarsInUse.txt and radarsOnMenu.txt control, respecitvely, which radars are ingested and which radars appear on the menu. If needed, the default versions of these are constructed using radar metadata in the shapefile nationalData/fsl-w88d plus some distance tests in a utility script called radarUtil.csh. The files mosaicScales.txt and mosaicInfo.txt control which on-the-fly mosaics are available on which scales. The default versions of these files will set up a single mosaic available on scale 4 (usually the state scale) containing the nine nearest radars on the menu.
updateRadarFiles.ksh calls genRadarDataKeys.ksh to make data keys, genRadarDepictKeys.ksh to make depict keys, and makeRadarSups.csh to generate depictor files for all of the radars known to the ingest. It also calls genRadarDataMenus.ksh to make menus, genRadarProdButtonInfo.ksh to make product buttons, and genRadarMultiLoadKeys.ksh to make multi-load entries for each radar on the menu. Finally, it calls doMosaicProcessing.ksh to create depict keys, product buttons, and menus for selected products that are displayed as on-the-fly mosaics.
updateRadarFiles.ksh and its subordinate scripts make use of several utility programs; shp2bcd, keyMunge, pasteUtil, and maksuparg.
The script makeMapFiles.csh is controlled by the `maps' task option. The primary function of this script is to make files that the workstation can read to draw vector map backgrounds. It does this in three ways: first, it uses the program shp2bcd to convert shape files into either .bcd files (binary cartographic data files) or .bcx files (extended binary cartographic data files), which are the two formats the workstation can read to draw vector map backgrounds. Second, it uses the program bcdProc to perform operations such as clipping and thinning on existing .bcd and .bcx files. Finally, it can move .bcd and .bcx files that are site specific control files into the localization data set.
The script makeWWAtables.csh is controlled by the `wwa' task option. The purpose of the makeWWAtables.csh script is to construct the geographic entity lookup tables (GELTs) that are needed by the warnGen program, as well as the template files that are used to describe the specific format of each type of product that can be generated by warnGen.
This script manipulates two types of files to make the files that serve as templates for the individual types of wwa products. Files in nationalData/ that have .preTemplate extensions are used to create files that have .template extensions in the localization data set. These files describe how to make generic blocks of text that can be used by many different wwa products, such as county, city, and zone lists, or UGC code headers. These files are brought into the specific templates for wwa products through an include mechanism. Files in nationalData/ that have .preWWA extensions are used to create files that have wwaProd extensions in the localization data set, and these files are the specific templates for warnGen products. The manner in which both the list templates and the product specific templates are created is controlled by several different directives (see 6.1.3). For a description of how wwa product templates are interpreted by the warnGen program, see TextTemplate.html.
This script also uses the utility program newGELTmaker (see href="newGELTmaker.doc.html">newGELTmaker.doc.html) to create geographic entity lookup tables (GELTs). These tables form the basis for how the warnGen program is able to create descriptions of the area covered by a watch, warning, or advisory. newGELTmaker uses binary cartographic data files, shape files, and other files with ascii lists of locations and identifiers (such as the CitiesInfo.txt file) as the raw material from which to construct geographic entity lookup tables. Currently, makeWWAtables.csh tries to construct eight different GELTs: a table for the county warning area of the localization, a wwa county table, a regional county table, a wwa zones table, a regional zones table, a warning cities table, a watch cities table, a wwa marine zones table, and a regional marine zones table. The files that control how these tables are made are called gelt script files. The default gelt script files live in nationalData/ and have file names that look like *_gsf.txt.
The script makeStationFiles.csh is controlled by the `station' task option. The purpose of the makeStationFiles.csh script is to construct station lists that contain progressive disclosure information in them. There are two types of station lists created here, station plot info files (.spi files), which are meant to be used to control the progressive disclosure for plots or stochastically distributed hydrometeorological data, and location plot info files (.lpi files) which are just meant to be used for constructing map backgrounds with location information.
The source data for these are so called goodness files, which are files with a .goodness extension and have two possible formats, compatible with either creating .spi or .lpi files. It is also possible to create .lpi files from the .id file that is a component of a GELT or from the data in the file $FXA_HOME/data/CitiesInfo.txt file. The goodness files are so named because they contain a `goodness factor' for each location, which is an arbitrary integer that says how desirable it is to see a given station at a low zoom level; the higher the number the more desirable it is to see the station at a low zoom. The program va_driver is used by makeStationFiles.csh to convert these goodness factors into progressive disclosure parameters. The progressive disclosure parameter used in WFO-Advanced is the distance in kilometers to the nearest other station which is at least as likely to be seen at a given zoom.
makeStationFiles.csh uses the program shp2bcd to construct goodness files from shape files, the program GELTtest to restrict station lists to only those in a given geographic entity (such as a county warning area), and bcdProc to restrict station lists to only those within the area of a depictor file (which describes a display scale, for instance).
Station plot info (.spi), location plot info (.lpi), .goodness files, and the CitiesInfo.txt are all plain ascii, and so can be managed manually. The preferred method is to change the .goodness files and then rerun makeStationFiles.csh, rather than to edit .spi or .lpi files directly.
Finally, makeStationFiles.csh uses the utility program reformatTest to create any predefined static plan view plot data sets. For those instances where there are corresponding *.cdl and *.dat files, and there is a data key for an adaptive plan view plotting data set with a file name but no directory, it will attempt to generate a blank *.nc file using ncgen on the *.cdl file and then use the contents of the *.dat file and the program reformatTest to initialize the file with data.
The script makeDirectories.csh is controlled by the non-default `dirs' task option. The main purpose of the makeDirectories.csh script is to create the data directory for every entry in the dataInfo.txt table, including all of its include files. It also creates the directories for gridded data files and moves the *.cdlTemplate files for each gridded data source into those directories and names them `template', and creates the template files for plan view point data sets stored in netCDF.
The script createAuxFiles.csh is controlled by the non-default `auxFiles' task option. The main purpose of the createAuxFiles.csh is to create directories and files on $FXA_DATA that are not standard data directories. Currently, this script sets up default RPS lists, creates the correct acquisition patterns, creates some files needed by the thunderstorm application, and copies some localization files to the ldad server.
4.1.17) fxatextTriggerConfig.sh
The script fxatextTriggerConfig.sh is controlled by the non-default `trigger' task option. The purpose of this script is to create text triggers, which can cause an action to occure upon arival of a given text product.
Site specific control files and realization files will be referred to with the prefixes `LLL-' and `RRR--' as before, where LLL is an arbitrary localization identifier and RRR is and arbitrary realization identifier (again not necessarily three characters). Files in the national data set that exist in $FXA_HOME/data will be referred to with the prefix `data/' and ones that exist in localization/nationalData will be referred to with the prefix `nationalData/'. Occasionally, a file will be referred to with the prefix `III-', which is similar to the `LLL-' prefix except that it refers to the identifier of the localization being used on the data ingest machine. All other first references to file names are assumed to be files in the final localization data set.
One concept that is very important to this section is file override. Here is an idealized example of how file override works. Suppose there is some file needed by the localization called locFile.dat. Very commonly, the existence of the following files will be checked: data/locFile.dat, nationalData/locFile.dat, RRR--locFile.dat, and LLL-locFile.dat. The last of these files that actually exists is what is used by the localization. An additional understanding of how file override works can be gained by reading the header documentation of the getPath.csh and fileGrab.csh scripts.
In what will be referred to as functional override, the file is merely located using the getPath.csh script, in which case the file will be used as input data to create some other file or control some aspect of the localization process. Sometimes, functional override will be implemented by copying the files into the localization data set and the removing them once they have been used. In what will be referred to as copy override, the file will actually be moved using the fileGrab.csh script to create the file locFile.dat in the localization data set. In what will be referred to as replacement override, the override file is moved into the localization data set only when the RRR-- or LLL- version is present, and the workstation will rely on the InfoFileServer class to find it directly in the national data set otherwise. The fileGrab.csh script can also operate in append mode, which means it would try to concatenate each of these files to locFile.dat in the localization data set with the >> redirect operator. That way, the national data set version will be at the beginning of the file in the localization data set and the local override version would be at the end. This final override method is referred to as append override.
When a file is subject to copy override or append override, a version of that file is always created in the localization data set. When a file is subject to functional override, no version of that file is ever permanently created in the localization data set. When a file is subject to replacement override, a version of that file is created in the localization data set only when the selected version is a local or realization file. This point is very important; it is possible for the same file to be subject to different types of override from different sources. For example, a file might be subject to copy override if an RRR-- version was found, but then subject to append override if an LLL- version was found.
File override selection always operates on a file by file basis. The fileGrab.csh script can operate on groups of files, so it is possible that one use of the fileGrab.csh script would result in files originating from several different source directories all ending up in the localization data set.
For every file mentioned in the national data set catalog, what type of file override it is subject to will be mentioned. Some files are not subject to any override, and this will be stated as well. When a file is `renamed', this means that it is copied to the localization data set and given a new name; files in the national data set are never removed or changed by the localization process. When a file is renamed, it is usually changed by sed or some other program.
Although this is not the usual procedure, it is also possible to obtain versions of satDepictKeys.txt and satDataKeys.txt directly through copy override.
Because of their size, the files c11-zone.shp, usa_cwa.shp, and uscounty.shp are kept in the source tree in compressed form. Thus, when new versions of these files are made available from NWS headquarters, they must be compressed before being checked into the source tree. The software that reads them will automatically uncompress them when required. In an environment without a source tree, it is perfectly OK to just replace these files directly in localization/nationalData without worrying about compressing them.
Shape files control a number of things. They are the primary source of data for creating map backgrounds. The county warning area information in the shape file usa_cwa provides the basis for defining the scales for all of the different localizations. Shape files are also the main source of data for creating geographic entity lookup tables, which form the basis for how the warnGen program is able to create descriptions of the area covered by a watch, warning, or advisory.
Currently, all .bcx files and many .bcd files are created using the utility program shp2bcd with shape files as input. The utility program bcdProc can also perform many other operations on .bcd and bcx files. The .bcd files that do not come from shape files all exist in the $FXA_HOME/data directory. These files are all subject to replacement override, and are used to describe data sets that we do not yet or will not have available from shape files. Some examples of this are continental and other international boundaries, snowfall contours for an orographic snow model covering the mountains of Colorado, some highway and river data for Colorado, and ARTCC boundaries.
If a III-gridSourceTable.txt file is available for the data server localization, then it will be used as the core of the source table, otherwise, the file nationalData/gridSourceTable.template will provide the core. The localization will also append the file nationalData/tdlSourceTable.template to the core and attempt to append III-localGridSourceTable.txt to the core. Both gridSourceTable.template and tdlSourceTable.template are subject to functional override. To create the gridSourceTable.txt file, a sed command is invoked on the core to tag the depictor files for LAPS and mesoEta with the data server localization identifier. gridSourceTable.template also has all gridded data sources marked as inactive. The file nationalData/activeSources.txt contains a list of unique ids for those gridded data sources to activate. It is subject to copy override for realizations and append override for localizations. The file inactiveSources.txt, if present as a site specific control file, can be used to turn off sources.
The structure of the netCDF files that contain the gridded data for each source are defined in .cdl files. The .cdl files for all of the default data sources exist in the $FXA_HOME/data directory, and are subject to replacement override.
Briefly, it is the job of the utility program makeGridKeyTables to use these five tables and the .cdl files for the individual gridded sources to create the files gridDataKeys.txt, and gridDepictKeys.txt. It is the job of the utility program testGridKeyServer to create the depictors and volume browser menus for the predefined latitude and longitude cross section baselines, as will as the volume browser menus for selecting the data source. To see a detailed write-up of how this works, please see gridTables.html, makeGridKeyTables.doc.html, and testGridKeyServer.doc.html.
The file nationalData/imageStyle.txt is managed in assembleTables.csh and is also subject to replacement override. It contains manually maintained style information for all images except those from gridded data, nexrad data, and topography data. Style information for gridded data images is in gridImageStyle.txt and style information for nexrad images usually comes from the data itself. Style information for topography images is written to the file topoImageStyle.txt and is generated by the same test_grhi_remap utility program that generates the images. There is a #include statement in imageStyle.txt that is used to bring in the topoImageStyle.txt file. There is also extensive header documentation in imageStyle.txt that is meant to provide information needed to maintain the file.
There are several .template files in nationalData/ that control what key, button, and menu entries for each nexrad radar look like. In these files the string `@@@@' is a stand in for an arbitrary radar id, and a single @ is a stand in for a key list. All of the keys in these files are generic keys in the range 10000-65535. Except for areal composite product keys, all of these keys are converted into radar specific keys based on the arbitrary immutable index for each radar. radarDataKeys.template contains generic entries for nexrad data keys, radarDepictKeys.template contains generic entries for nexrad depict keys, and radarProductButtonInfo.template contains generic entries for nexrad product buttons. The file radarMultiLoadInfo.template contains generic entries for multi-loads, which is how the workstation handles such things as reflectivity/velocity combo and four-panel displays. The file radarDataMenus.template serves as a template for creating the menu entries for one nexrad radar. The files radarMenuHeader.txt and radarMenuFooter.txt contain menu entries for products other than nexrad radar products that the user might want to appear at the top or bottom, respectively, of the `Radar' menu, which is the way the user also accesses dial radars and mosaics. The files mosaicDataMenus.template, mosaicDepictKeys.template, and mosaicProductButtons.template contain generic entries of menus, depict keys and product buttons for areal composite products. All of the files mentioned in this paragraph are subject to functional override, except for radarMenuHeader.txt and radarMenuFooter.txt, which are subject to replacement override.
The files in nationalData/ with names like wwa_blahblah_blah.preWWA are source data for the default set of warnGen product templates. Each file like this gets converted by the makeWWAtables.csh script into a file that look like wwa_blahblah_blah.wwaProd, and these files are the actual warnGen product templates. The *.preWWA files are subject to functional override. In the conversion process, sed commands are used to substitute the value of several directives in these files (see section 6.1). For every warnGen product template, entries are made in the wwa.config file, which are the name of the file and the title of the product; this registers the file and its description to the warnGen program. The title of the product must be present in a commented line (// style) in the .preWWA file with the title in quotes. Warngen templates are sorted on the warngen menu based on the title; any text in the title that occurs before an optional vertical bar(|) is only used for sorting purposes and will not appear on the warngen menu. The function of the warnGen product template files is described in TextTemplate.html.
warnGen product templates can access through #include statements one or more files with names like wwa_blah_blah.template. These files contain unified mechanism for generating UGC codes, or lists of items such as counties, cities, or zones. The source data for these files are files in nationalData/ with names like wwa_blah_blah.preTemplate. The makeWWAtables.csh script converts the *.preTemplate files into the *.template files. In the process, sed commands are used to substitute the value of several directives in these files (see section 6.1). The *.preTemplate files are not subject to any override functionality. The functionality that performs this conversion is somewhat complex; sometimes the wwaUtil.csh is used to put several different versions of a *.preTemplate together into one *.template file.
Related to these files are some files in nationalData/ with names like sls*.preTemplate. These files are used to generate the product template for the makeWRKSLS program. Some of the same directives used by the warnGen product template conversion are used here. The file nationalData/sls.preTemplate is converted to the file sls.template, which is the main product template. The rest are used to generate the file sls_county_list.template, which is brought into sls.template through a #include statement and is used to create the text of the county list. Another related group of files are the nationalData/*.abrev files, of which there are currently three, areas.abrev, county_type.abrev, and state.abrev. These files are used by warngen to translate abreviations into plain language, and are subject to copy override only from realization files. They refer to parts of states, how to describe a county, and two letter postal codes.
Files in nationalData/ that have file names like *_gsf.txt are refered to as GELT script files. These files are subject to functional override from all places. They are used by the program newGELTmaker to create Geographic Entity Lookup Tables. Also many of the files that the program newGELTmaker typically reads in are subject to various types of override, as well.
An additional source of information for controlling stochastic progressive disclosure is the file data/CitiesInfo.txt. This file is the main source of information for constructing the cities map background, as well as the warnGen cities table and the warning cities map background. The format of this file is also documented in va_driver.doc.html.
Another type of file that affects stochastic progressive disclosure is files that have a `.primary' extension. These are lists of station or location identifiers that are to be made visible at lower zoom factors regardless of the goodness value associated with them. These files are subject to reverse append override.
Two related files are data/anchors.txt and data/selsAnchors.txt. These are used for labeling cross section baselines and decoding SAW products respectively. These are not subject to any file override and (especially selsAnchors.txt) should normally be centrally maintained, but it is useful to know about these files in case either of these functions fails to work as expected.
This discussion will use the same file identification convention as the previous section. Specifically, the prefixes `LLL-' and `RRR--' will refer to site specific control files and realization files, the prefixes `data/' and `nationalData/' will refer to national files in the directories $FXA_HOME/data and localization/nationalData, and all other references to file names are assumed to be files in the final localization data set.
The main purpose of this file is to contain directives that can define additional localizations. The are two directives that can be used to define a localization. The first is the `WFO' directive, which must have as its argument one of the county warning area identifiers in the usa_cwa shape file in localization/nationalData. Executing the script cwaIds.csh in localization/scripts will provide a list of these county warning area identifiers. The second is the `CLONE' directive. The argument of the CLONE directive must be the identifier of some other valid localization which is not itself defined by a clone directive. A localization defined by the WFO directive will not inherit any site specific files from the localization with the same name as the county warning area identifier, whereas a localization defined by the CLONE will inherit any site specific files from the localization it is cloned from. Any site specific files belonging to the new cloned localization will override any from the localization it is cloned from.
The final type of directive that can define a localization is the `REALIZATION' directive. The REALIZATION directive has as its argument the name of the realization to associate this localization with (see chapter 8). For a complete list of usable directives, the readed is directed to directives.html.
This file does not contain directives, but it is related to the WFO directive in LLL-mainConfig.txt. This file can be used to define a localization centered about some arbitrary point or area, not just one of the predefined county warning identifiers. This file contains any number of latitude-longitude points. Each point is on one line in the file, and is in ascii format, space delimited. The localization ends up being centered around that point or group of points. It is not recommended to define a localization with both a LLL-cwa.asc file and either a WFO or CLONE directive in the LLL-mainConfig.txt file.
This file contains directives that change how the warnGen functions. A file in nationalData/ called wwaDefaults.txt contains default settings for some of these directives, but any instances of these directives in LLL-wwaConfig.txt will override those in wwaDefaults.txt. For a complete list of the directives that one can place in the wwaConfig.txt file, one is again directed to directives.html.
6.2.1) Depict keys, data keys, and product buttons
The file nationalData/dataInfo.manual has a #include statement for bringing in the file localDataKeys.txt, which must come from the file LLL-localDataKeys.txt. This allows any arbitrary additional data keys to be added that are specific to the localization. Also, in nationalData/depictInfo.manual, there is an analogous #include statement that refers to localDepictKeys.txt. Finally, in the file nationalData/productButtonInfo.txt, there is a #include statement for the file localProductButtons.txt, which can bring in any arbitrary additional product buttons that are specific to the localization.
Unlike depict keys, data keys, and product buttons, which are order insensitive, entries in the menus files are order sensitive. Thus, there are many different places in the default menus where site specific control files can be brought in by #include statement. The file nationalData/backgroundMenus.txt, which is the default map background menu, has a #include statement for the file otherBackgroundMenus.txt. In nationalData/dataMenus.txt, there are many #include statements designed to bring in site specific control files. The files otherToolMenus.txt and otherVolumeMenus.txt allows localization specific additions to the Tools and Volume menus. The file ldadMenus.txt is where non-default LDAD data plots are added to the surface menu. The file analysisMenus.txt allows localization specific surface analysis menus to be brought in, and otherSurfaceMenus.txt allows any other arbitrary additions to the end of the surface menu. Finally, otherUaMenus.txt and otherSatMenus.txt allow for arbitrary additions to the end of the upper air and satellite menus.
Any file in the localization specific files (LLL- files) or realization specific files (RRR- files) that has a file name pattern like `*Menus.txt', `*MenuHeader.txt', or `*MenuFooter.txt' will be moved into the localization data set. However, to become a useful menu, one of the existing default menus must be overrode and a #include statement for the new menu file added.
Any file in the localization specific files or realization specific files that has a file name pattern like `*.bcd' or `*.bcx' will be moved into the localization data set. A new file such as this cannot be used for drawing a map background until data key, depict key, product button, and menu entries have been defined. In this case, this would usually be handled using the files LLL-localDataKeys.txt, LLL-localDepictKeys.txt, LLL-localProductButtons.txt, and LLL-otherBackgroundMenus.txt, respectively.
6.3.3) Topography and cdl files for gridded data sources
Any file in the localization specific files or realization specific files that has a file name pattern like `*.cdl' or `*.topo' will be moved into the localization data set. These files are associated with gridded data sources. The .cdl files are used to define the exact variables and levels stored for a given gridded data source, and the .topo files contain the surface topography used for a given gridded data source. Before new files like these can be used by the system, one must also add a new gridded data source to the system that specifically refers to these new .cdl and/or .topo files. This can be done by functionally overriding the file in nationalData/ called gridSourceTable.template, or by providing a file called III-gridSourceTable.txt, which will be used as a direct replacement for the grid source table.
6.3.4) Product templates for warnGen
Any file in the localization specific files or realization specific files that has a file name pattern like `wwa_*.preWWA' will be moved into the localization data set. These files are used to create analogous files with names like `wwa_*.wwaProd' that serve as warnGen product templates. In order for a new file like this to be used by the warnGen program, one must make sure it has a title line, which is a commented out line (// style) with the title in quotes immediately after the comment. 6.3.5) GELT script files.
GELT script files are files that contain instructions that are used by the program newGELTmaker to create Geographic Entity Lookup Tables, which are used by warngen to identify which counties, cities, etc., fall within a warned area. GELT script files have file names that look like *_gsf.txt, and the defaults live in nationalData/. Any file in the site specific, realization, or customization files that has a file name pattern like this will be used to try to create a GELT. These override files can be used to change how a default GELT is created or to generate additional GELTs.
Any file in the localization specific files or realization specific files that has a file name pattern like `*.goodness' and `*.primary' will be used to create either station plot information (.spi) files or location plot information (.lpi) files in the localization data set, depending on the format of the .goodness file. A .primary file only modifies how a .goodness file is interpreted, so a .primary file by itself is meaningless (unless it refers to a .goodness file already in the national data set). In order for the resulting .lpi or .spi files to be used to create point based map backgrounds, entries must be made for data keys, depict keys, product buttons, and menus. In this case, this would usually be handled using the files LLL-localDataKeys.txt, LLL-localDepictKeys.txt, LLL-localProductButtons.txt, and LLL-otherBackgroundMenus.txt, respectively.
The files removeMenuItems.txt and preserveMenuItems.txt are subject to copy override from realization files and to append override from site specific files. They allow a localization to have certain menu items removed from the user interface without having to edit or override files in the default menus. After applying file override to build versions of these files, menu items listed in removeMenuItems.txt but not in preserveMenuItems.txt will be removed from the user interface. Each menu item listed contains an object to delete from menus plus an optional leading comma delimited file name. If the file name exists, then that will be the only that file name will be searched for items to delete. The object to delete can be a button number, an appButton name, an include file name, or a submenu name. If a submenu name, use a '.' for spaces or metacharacters. In the case of a submenu name, it will delete the whole submenu, accounting for nesting, but will not do the right thing for files where the submenu statement is in a different file from the endSubmenu statement.
Finally, many of the key tables in WFO advanced are actually put together at run time through the use of an include mechanism. The program textBufferTest (see textBufferTest.doc.html) can be used to locate and output to standard output the entire contents of such a table.
The creation of a GELT naturally creates a location list, and as such GELTs are often used as a source of information for such lists. The program GELTtest (see GELTtest.doc.html), while primarily a unit test for the geographic entity lookup table software, is used in the localization to restrict the contents of station lists based on the contents of a geographic entity lookup table. Program masterToGoodness (see masterToGoodness.doc.html) can convert a station list from a GELT into a form that can be interpreted by the program va_driver.
Currently, there are four valid realizations defined, `RFC', `NC', `radonly', and `cwb'. The `RFC' realization is used to create localizations suitable for use in river forecast centers. The `NC' realization is used to create localizations suitable for use by national centers. The `cwb' realization is used to create localizations suitable for use by the Central Weather Bureau in Taiwan. The `radonly' realization is used to create localizations suitable for displaying Nexrad radar data only, occasionally being used in Alaska and Hawaii.
There are two additional environment variables that are important to customization: FXA_CUSTOM_FILES and FXA_CUSTOM_VERSION. FXA_CUSTOM_FILES can point to any arbitrary directory in the file system; it could be either a local or remote mounted disk. FXA_CUSTOM_VERSION needs to point to a subdirectory in the directory pointed to by FXA_CUSTOM_FILES. By default, FXA_CUSTOM_FILES points to the directory $FXA_DATA/customFiles. Since this is a cross-mounted disk, a change made here will affect future localizations run on any machine. It is important to note that the current configuration does not allow any customization files to directly affect the operation of the workstation or ingest software; customization files can only change the way a localization is built.
In order to make use of the customization features, the environment variable FXA_CUSTOM_FILES (and, optionally, FXA_CUSTOM_VERSION) must be set prior to running any localization that one wishes to be customized. Unlike FXA_LOCAL_SITE and FXA_INGEST_SITE, there is no way to set FXA_CUSTOM_FILES or FXA_CUSTOM_VERSION on the command line. If either are left unset, they will normally default to their previous values the last time a given site's localization was run. If one wants to rerun a site's existing localization using different values of FXA_CUSTOM_FILES or FXA_CUSTOM_VERSION, one must use the leading `n' flag in the command line of mainScript.csh.
Customization works by extending the existing file override capabilities to also include files in $FXA_CUSTOM_FILES or $FXA_CUSTOM_FILES/$FXA_CUSTOM_VERSION. If both FXA_CUSTOM_FILES and FXA_CUSTOM_VERSION are defined, then the following file paths can also participate in file override:
$FXA_CUSTOM_FILES/* $FXA_CUSTOM_FILES/$FXA_LOCAL_SITE-* $FXA_CUSTOM_FILES/$FXA_CUSTOM_VERSION/* $FXA_CUSTOM_FILES/$FXA_CUSTOM_VERSION/$FXA_LOCAL_SITE-*If append override is being used, as is very often the case with customization files, then this is the order in which they get appended. If using copy override, then this is the order in which they are copied so the last in the list that exists is what is used. Not all files in the localization data set are subject to file override from customization files. Here is a current list of those that are, along with the type of override to which they are subject for customization (a more detailed table covering this is available in fileChanges.html):
Append override:
localD*Keys.txt
localMultiLoadInfo.txt
*.abrev
localImageStyle.txt
satProductButtons.txt
localProductButtons.txt
*Menus.txt
*MenuHeader.txt
*MenuFooter.txt
mainConfig.txt
*.spi
*.lpi
*.goodness
*.primary
wwaConfig.txt
localGridSourceTable.txt
*activeGridSources.txt
*.rules
virtualFieldTable.txt
dataFieldTable.txt
gridPlaneTable.txt
dataLevelTypeTable.txt
*.config
browser*Menu.txt
radarInfoMaster.patch
LocalCitiesInfo.txt
*MenuItems.txt
Functional override:
wwa_*.preWWA
*_gsf.txt
*_*.txt
Replacement override:
radarsInUse.txt
radarsOnMenu.txt
pupId.txt
dialRadars.txt
mosaicScales.txt
mosaicInfo.txt
portInfo.txt
*.topo
*.sup
*.bcd
*.bcx
localColorMaps.mark
Here is an example of how this might be used. Suppose one wanted to move the regional area 200 km to the south in the summer. In directory $FXA_CUSTOM_FILES/summer, one could create a file mainConfig.txt, containing the entry `@@@REGNORTH -200' (see directives.html for more information). Then all localizations would be rerun twice a year, spring and fall. In the fall, FXA_CUSTOM_VERSION would be undefined, and in the spring it would have a value of `summer'. What this would do is move the regional scale 200 km south of the default for the warm season and put it back to the default for the cool season.