RXTE GOF |
XTE FITS Database Management for Users Recipes from the RXTE Cook Book |
RXTE FAQ |
---|
Introduction
The design of the XTE FITS Database (XFDB) is described in other documents with varying degrees of detail. This document aims to be a purely practical guide. It will cover three topics:
-
The function of the Master Index (FMI) files and how XDF uses them - to provide background information for XDF and the next topic.
How to create and fix FMI files - e.g. if you want to combine observations or build a database with multiple proposals and AOs, but don't have a combined FMI file.
The calibration database (CALDB) - a brief introduction.
xdf: XTE Data Finder, shipped with Ftools. $XFDB: Environment variable, to be set to the top of the database (the location of the FMI file). recofmi: Tool to repair or create a higher level FMI file.The necessary tools and files can be downloaded via anonymous FTP from legacy.gsfc.nasa.gov, in the directory xte/calib_data.
FMI files simply contain a list of observations, providing information
like:
Typically, the mission database contains FMI files at three levels: the full mission; a single proposal (in the "P" directories); and a single ObsId. Consequently, a proposal level FMI contains all the combined information of the FMI files in the underlying ObsId directories, while the full mission FMI contains all information from all proposal level FMI files underneath it.
Therefore, it does not make any difference whether XDF is pointed to the mission, proposal, or ObsId directory: as long as there is an FMI file, it will be able to locate the data files for that ObsId.
One should realize, though, that the file pointers in all index files
are relative. If the FMI in ObsId directory
The best thing to do is to set the environment variable XFDB to the
directory that you want to use as the top of the database; make sure
that it contains an FMI file. In C shell:
FMI files are crucial for XDF's working correctly. In some cases you may have to fix one or create a new one; for instance, if you have a number of observations from your proposal, but no up-to-date proposal level FMI file; or if you want to combine observations from different proposals.
The ftool for creating or updating FMI files is recofmi. The general usage is simple; for instance, to update an existing FMI file in proposal level directory P10408 after you've added observations which aren't listed in the current FMI:
Normally, rows are only added. If rows with identical ObsIds are found, the original is left in place. The option "overwrite=yes" will force such rows to be overwritten. The "delete=yes" option first wipes the FMI file clean and then fills it only with whatever ObsIds it finds in lower level FMI files (for instance, if you have only a few of the listed observations.) The "check=yes" option provides only a report and will not make any changes.
If you want to create a new FMI in a directory that does not have one (say you have some loose ObsIds and want to put them under on common directory), first copy an existing FMI file (it does not matter which) to that directory and then run "recofmi delete=yes" on it.
For a more complicated example, suppose you need to create a top level FMI file under the FD directory, which you will probably want to do once you have datasets from multiple proposals and AOs; you then simply add the "level=2" option. This will tell the tool to look two directory levels down (through AOs to the proposals) and find the information in those FMIs:
The caltools are a subset of the FTOOLS which require installation of CALDB in order to find the proper files for their execution. Other XTE-specific tools use the CALDB as well, such as pcabackest and pcarsp.
In all but early AO1 processed data, your tape comes with the full database structure starting with FD as the top level directory containing an AOn directory and a Cal directory, as well as Clock and OrbitEphem. Under AOn are the proposal level directories, while the Cal directory contains all of the XTE calibration files. For early processing, your tape begins with the proposal level directory, and under each ObsId is a cal directory. Newer processing puts all of the calibration files in one higher level directory to save space, replacing the files in each ObsId with symbolic links.
1. Master Index (FMI) Files
ObsId
Source
User
RA and Dec
Start and stop time
Pointers to index files for all subsystems
Subsystem index files, in turn, contain pointers to the actual data files. Thus, XDF can locate any data file, as long as it is given the path to the directory where the FMI file resides.
00023-02-01-00
points to PCA index
FIPC_3eb6dcb-3eb922d,
the reference in the proposal directory P00023 will be:
00023-02-01-00/FIPC_3eb6dcb-3eb922d,
and in the mission FMI:
AO0/P00023/00023-02-01-00/FIPC_3eb6dcb-3eb922d.
One should take care not to move FMI files from one directory to another. On the other hand, moving entire directory trees does not affect FMI file in the underlying structure.
setenv XFDB /data/kaah3/xteguest/xtegst1/data/P00023
XDF will use $XFDB as a first guess for its database path.
2. Repairing or Creating an FMI File
recofmi dirpath=P10408
This reconstructs the file P10408/FMI according to the contents of FMI files in underlying directories. If level=1 (default), it will go only to the first level subdirectories (i.e. ObsIds in this case); if level=2, it will go only to the subdirectories two levels deep. Typically, level=1 is used in proposal directories to include the information from their ObsId directories; level=2 is used to incorporate information from proposal level FMI files into the mission database's FMI file.
cd $XFDB
cp AO1/P10033/FMI .
recofmi dirpath=$XFDB level=2 delete=yes
In this case, recofmi will look in all AOn directories and all Pxxxxxx directories under those to find the proposal level FMIs. The resulting top level FMI will list all the observations from all AOs and proposals which are in your database.
3. The calibration database (CALDB)
If you have a question about RXTE, please send email to one of our
help desks.