How to find out what data is avaliable ====================================== E. Buckley-Geer 23-Apr-1993 Contents ======== 1. Introduction 2. What types of data exist, where is it and how to find out about it 3. How to get luminosity information 4. How to allocate a user tape 5. FATMEN and the SILO 6. The TPID bank 1. Introduction =============== This document is an attempt to collect together all the information on what data is where and how to get information about it. The information is taken from many mail messages and info messages that have been posted by many people. 2. What types of data exist, where is it and how to find out about it ===================================================================== The following table summarizes the types of data that currently exist. Data is either on disk, on 8mm tape or in the STK Silo. For data produced through the production group, information can usually be found in the Production_manager (PM) database or the FATMEN (FM) catalog. Data type Location Where to find info ========================================================================== Express line RAW data Disk Look at disk area for new files +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Express line DSTs STK Silo FM catalog Disk Look at disk area for new files ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 6.1 production only Inclusive Express Disk Look at disk area line PADs 8mm tape for new files ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Split Express line PADs Disk Look at disk area STK Silo for new files ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Full Production DSTs 8mm tape PM database ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Split STREAM1 DSTs 8mm tape PM database ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Split STREAM1 PADs Disk Look at disk area STK Silo FM catalog 8mm tape PM database ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 6.01 production only Split STREAM2 DSTs Disk Look at disk area STK Silo FM catalog ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Before Run 41936 (6.0)/After Run 42294 (6.01/6.1) Global STREAM2 DST 8mm tape PM database ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ a) Express line raw data ========================= These are located on disk in FNALD: CDF$TRS_DATA:[RAW] CDFSGA: /fnald/cdfstrip7/trs0/raw They should remain there for about 5 days, disk space permitting. b) Express line DSTs ==================== These files are stored in the Silo after a few days. They do not remain for ever in the Silo but are deleted periodically to make way for new files. They can be accessed via the FATMEN catalog. They can be found in SILO: //fnal/cdf/prod/exp/oth/dst The most recent files are located in FNALD: CDF$EXPRESS_DATA:[stream_2.dst] CDFSGA: /fnald/cdfstrip9/express0/stream_2/dst File extensions are STRX_1F for 6.01 and STRX_3F for 6.1. They are also copied to 8mm tape. A list can be found in C$DOC:EXPRESS_TAPE_LIST.TXT which includes the luminosity for each file. c) Inclusive Express line PADs ============================== These are located in FNALD: CDF$EXPRESS_DATA:[stream_2.pad] CDFSGA: /fnald/cdfstrip9/express0/stream_2/pad The files are concatenated per run. The names have the following syntax E46728A_.STRX_3P These files are also staged to 8mm tape. At present you can ask Stephan Lammel for the list of tapes. This will be made public soon. d) Split Express line PADs ========================== Pads produced with 6.01 are located in the silo. //FNAL/CDF/PROD/EXP/PAD01/AAAX-1P //FNAL/CDF/PROD/EXP/PAD02/AAAX-1P where AAA = QCD, PHO, ELE, MUO, DIL, PSI, EXO. Pads produced with 6.1 are located in CDF$PDM3_DATA:[EXPRESS.STK.aaaX_3P] where aaa = QCD, PHO, ELE, MUO, DIL, PSI, EXO, OTH. The files are concatenated per run. The names have the following syntax E46728A_.aaaX_3P They are copied into the silo and can be found in //FNAL/CDF/PROD/SPTX3/PAD01/AAAX-3P (note that in the FATMEN catalog the "_" becomes a "-" (Note that there is a limit of 1000 files per FATMEN directory so as more files are added you will see PAD02, PAD03 etc.) e) Full production DSTs ======================= These are available on 8mm tape. The information can be obtained from the PM database: $ setup production_manager $ @production_manager$programs:p_m_tape_report STR1_1F (For 6.0 files the database to search is DST, for 6.01 use STR1_1F and for 6.1 use STR1_3F.) This produces a file in the users current directory called STR1_1F_TAPE_INDEX.TXT the contents of which looks as follows: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Production_Manager STR1_1F Tape Index generated 5-OCT-1992 16:48:20.78 File_Name Tape Date_On_Tape --------- ------ ------------ R38417AA.STR1_1F CC2769 28-JUL-1992:22:08:22.18 GOOD R38418AA.STR1_1F CC2769 28-JUL-1992:22:08:23.54 GOOD R38418AB.STR1_1F CC2769 28-JUL-1992:22:08:24.79 GOOD R38418AC.STR1_1F CC2769 28-JUL-1992:22:08:26.68 GOOD R38418AD.STR1_1F CC2770 28-JUL-1992:23:27:46.20 GOOD T38417AA.STR1_1F CC2776 29-JUL-1992:17:00:38.18 GOOD T38418AA.STR1_1F CC2776 29-JUL-1992:17:00:40.45 GOOD T38418AB.STR1_1F CC2776 29-JUL-1992:17:00:42.51 GOOD T38418AC.STR1_1F CC2776 29-JUL-1992:17:00:44.87 GOOD S38417AA.STR1_1F CC2777 29-JUL-1992:18:05:33.92 GOOD S38418AA.STR1_1F CC2777 29-JUL-1992:18:05:37.14 GOOD ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ You can also look at the file C$DOC:PRODUCTION_TAPE_LIST.TXT. This file also contains information on the luminosity per file and whether the file has been split. f) Split STREAM1 DSTs ===================== These are available on 8mm tape. Before run 41936 the files are split into streams but all files are collected on a single tape. After run 41936 each stream is collected on its own set of tapes. (adapted from an INFO posted by JJ). The following streams exist at present STREAM 1 DST FILENAMES Trigger selection Filename extension ----------------- ------------------- 6.0 6.01 6.1 --- ---- --- JET_100 J1Q1_0F J1Q1_1F J1Q1_3F JET_70 J2Q1_0F J2Q1_1F J2Q1_3F JET_50 J3Q1_0F J3Q1_1F J3Q1_3F JET_20 J4Q1_0F J4Q1_1F J4Q1_3F SUMET_175 ETQ1_0F ETQ1_1F ETQ1_3F TOP -> Multijets TJQ1_0F TJQ1_1F TJQ1_3F 16 GeV photons, non-isolated C1P1_0F C1P1_1F C1P1_3F 6 GeV photons, non-isolated C2P1_0F C2P1_1F C2P1_3F 15 GeV Plug photons PMP1_0F PMP1_1F PMP1_3F Electrons ELE1_0F ELE1_1F ELE1_3F Central muons CMU1_0F CMU1_1F CMU1_3F Forward muons FMU1_0F FMU1_1F FMU1_3F High Pt Dileptons DIL1_0F DIL1_1F DIL1_3F Jpsis PSI1_0F PSI1_1F PSI1_3F Exotics XOX1_0F XOX1_1F XOX1_3F Minbias MBO1_0F MBO1_1F MBO1_3F Information about these streams can be found in the production_manager database: $ setup production_manager $ @production_manager$programs:p_m_tape_report xxx where xxx is the stream name, e.g. ETQ1_0F This produces a file in the users current directory called xxx_TAPE_INDEX.TXT the contents of which looks as follows: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Production_Manager ETQ1_0F Tape Index generated 6-OCT-1992 11:17:15.45 File_Name Tape Date_On_Tape --------- ------ ------------ R40692AA.ETQ1_0F disk 06-OCT-1992:08:20:03.82 GOOD These files U40690AH.ETQ1_0F disk 06-OCT-1992:06:28:52.34 GOOD have not been U40690AI.ETQ1_0F disk 06-OCT-1992:08:16:19.10 GOOD staged to tape U40690AJ.ETQ1_0F disk 06-OCT-1992:09:36:11.26 GOOD yet. R39882AA.ETQ1_0F CC2929 23-AUG-1992:13:13:28.51 GOOD R39882AB.ETQ1_0F CC2929 23-AUG-1992:15:06:56.95 GOOD R39882AC.ETQ1_0F CC2929 23-AUG-1992:15:04:15.28 GOOD R39882AD.ETQ1_0F CC2929 23-AUG-1992:16:44:11.61 GOOD +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ At present the list of files in the database does not necessarily correspond to the order on the tape. g) Split STREAM1 PADs ===================== Most of these are located in the Silo and are split by streams into separate files (adapted from an INFO posted by JJ). Some of the larger streams are only available on 8mm tape. STREAM 1 PAD FILENAMES Trigger selection Filename extension ----------------- ------------------- 6.0 6.01 6.1 --- ---- --- JET_100 J1Q1_0P J1Q1_1P J1Q1_3P JET_70 J2Q1_0P J2Q1_1P J2Q1_3P JET_50 J3Q1_0P J3Q1_1P J3Q1_3P JET_20 J4Q1_0P J4Q1_1P J4Q1_3P SUMET_175 ETQ1_0P ETQ1_1P ETQ1_3P TOP -> Multijets TJQ1_0P TJQ1_1P TJQ1_3P 16 GeV photons, non-isolated C1P1_0P C1P1_1P C1P1_3P 6 GeV photons, non-isolated C2P1_0P C2P1_1P C2P1_3P 15 GeV Plug photons PMP1_0P PMP1_1P PMP1_3P Electrons ELE1_0P ELE1_1P ELE1_3P Central muons CMU1_0P CMU1_1P CMU1_3P Forward muons FMU1_0P FMU1_1P FMU1_3P High Pt Dileptons DIL1_0P DIL1_1P DIL1_3P Jpsis PSI1_0P PSI1_1P PSI1_3P Exotics XOX1_0P XOX1_1P XOX1_3P Minbias MBO1_0P MBO1_1P MBO1_3P The following FATMEN directories exist for these files (note that in the FATMEN catalog the "_" becomes a "-"): For 6.0 pads: //FNAL/CDF/PROD/STREAM1/PAD/MBO1-0P //FNAL/CDF/PROD/STREAM1/PAD/CMU1-0P //FNAL/CDF/PROD/STREAM1/PAD/ELE1-0P //FNAL/CDF/PROD/STREAM1/PAD/PSI1-0P //FNAL/CDF/PROD/STREAM1/PAD/XOX1-0P //FNAL/CDF/PROD/STREAM1/PAD/C1P1-0P //FNAL/CDF/PROD/STREAM1/PAD/C2P1-0P //FNAL/CDF/PROD/STREAM1/PAD/J1Q1-0P //FNAL/CDF/PROD/STREAM1/PAD/FMU1-0P //FNAL/CDF/PROD/STREAM1/PAD/J3Q1-0P //FNAL/CDF/PROD/STREAM1/PAD/PMP1-0P //FNAL/CDF/PROD/STREAM1/PAD/J4Q1-0P //FNAL/CDF/PROD/STREAM1/PAD/J2Q1-0P //FNAL/CDF/PROD/STREAM1/PAD/ETQ1-0P //FNAL/CDF/PROD/STREAM1/PAD/TJQ1-0P //FNAL/CDF/PROD/STREAM1/PAD/DIL1-0P For 6.01 pads: //FNAL/CDF/PROD/STREAM1/PAD01/MBO1-1P //FNAL/CDF/PROD/STREAM1/PAD02/MBO1-1P //FNAL/CDF/PROD/STREAM1/PAD01/CMU1-1P //FNAL/CDF/PROD/STREAM1/PAD02/CMU1-1P //FNAL/CDF/PROD/STREAM1/PAD01/ELE1-1P //FNAL/CDF/PROD/STREAM1/PAD02/ELE1-1P //FNAL/CDF/PROD/STREAM1/PAD01/PSI1-1P //FNAL/CDF/PROD/STREAM1/PAD02/PSI1-1P //FNAL/CDF/PROD/STREAM1/PAD01/XOX1-1P //FNAL/CDF/PROD/STREAM1/PAD02/XOX1-1P //FNAL/CDF/PROD/STREAM1/PAD01/C1P1-1P //FNAL/CDF/PROD/STREAM1/PAD02/C1P1-1P //FNAL/CDF/PROD/STREAM1/PAD01/C2P1-1P //FNAL/CDF/PROD/STREAM1/PAD02/C2P1-1P //FNAL/CDF/PROD/STREAM1/PAD01/J1Q1-1P //FNAL/CDF/PROD/STREAM1/PAD02/J1Q1-1P //FNAL/CDF/PROD/STREAM1/PAD01/FMU1-1P //FNAL/CDF/PROD/STREAM1/PAD02/FMU1-1P //FNAL/CDF/PROD/STREAM1/PAD01/J3Q1-1P //FNAL/CDF/PROD/STREAM1/PAD02/J3Q1-1P //FNAL/CDF/PROD/STREAM1/PAD01/PMP1-1P //FNAL/CDF/PROD/STREAM1/PAD02/PMP1-1P //FNAL/CDF/PROD/STREAM1/PAD01/J4Q1-1P //FNAL/CDF/PROD/STREAM1/PAD02/J4Q1-1P //FNAL/CDF/PROD/STREAM1/PAD01/J2Q1-1P //FNAL/CDF/PROD/STREAM1/PAD02/J2Q1-1P //FNAL/CDF/PROD/STREAM1/PAD01/ETQ1-1P //FNAL/CDF/PROD/STREAM1/PAD02/ETQ1-1P //FNAL/CDF/PROD/STREAM1/PAD01/TJQ1-1P //FNAL/CDF/PROD/STREAM1/PAD02/TJQ1-1P //FNAL/CDF/PROD/STREAM1/PAD01/DIL1-1P //FNAL/CDF/PROD/STREAM1/PAD02/DIL1-1P (Note that there is a limit of 1000 files per FATMEN directory so as more files are added you will see PAD03 etc.) For 6.1 pads: //FNAL/CDF/PROD/SPT13/PAD01/MBO1-1P //FNAL/CDF/PROD/SPT13/PAD01/XOX1-1P //FNAL/CDF/PROD/SPT13/PAD01/C1P1-1P //FNAL/CDF/PROD/SPT13/PAD01/C2P1-1P //FNAL/CDF/PROD/SPT13/PAD01/J1Q1-1P //FNAL/CDF/PROD/SPT13/PAD01/FMU1-1P //FNAL/CDF/PROD/SPT13/PAD01/J3Q1-1P //FNAL/CDF/PROD/SPT13/PAD01/PMP1-1P //FNAL/CDF/PROD/SPT13/PAD01/J4Q1-1P //FNAL/CDF/PROD/SPT13/PAD01/J2Q1-1P //FNAL/CDF/PROD/SPT13/PAD01/ETQ1-1P //FNAL/CDF/PROD/SPT13/PAD01/TJQ1-1P //FNAL/CDF/PROD/SPT13/PAD01/DIL1-1P (Note that there is a limit of 1000 files per FATMEN directory so as more files are added you will see PAD02 etc.) While files are being accumulated they are available on disk in the following directories CDF$PDM2_DATA:[SPT13.STK.C1P1_1P]*.C1P1_3P CDF$PDM2_DATA:[SPT13.STK.C2P1_1P]*.C2P1_3P CDF$PDM2_DATA:[SPT13.STK.DIL1_1P]*.DIL1_3P CDF$PDM2_DATA:[SPT13.STK.ETQ1_1P]*.ETQ1_3P CDF$PDM2_DATA:[SPT13.STK.FMU1_1P]*.FMU1_3P CDF$PDM2_DATA:[SPT13.STK.J1Q1_1P]*.J1Q1_3P CDF$PDM2_DATA:[SPT13.STK.J2Q1_1P]*.J2Q1_3P CDF$PDM2_DATA:[SPT13.STK.J3Q1_1P]*.J3Q1_3P CDF$PDM2_DATA:[SPT13.STK.J4Q1_1P]*.J4Q1_3P CDF$PDM2_DATA:[SPT13.STK.MBO1_1P]*.MBO1_3P CDF$PDM2_DATA:[SPT13.STK.PMP1_1P]*.PMP1_3P CDF$PDM2_DATA:[SPT13.STK.TJQ1_1P]*.TJQ1_3P CDF$PDM2_DATA:[SPT13.STK.XOX1_1P]*.XOX1_3P The ELE, CMU and PSI streams are staged to 8mm tape directly from the splitting nodes and are not available cluster wide. In addition there have been efforts to remove the larger streams from the silo for 6.01 data as well. Please see the physics group convenors for details of this. h) Split STREAM2 DSTs ===================== These are available for 6.01 production only. They are stored in the SILO. They are NOT available on 8mm tape. The FATMEN directories are as follows: //FNAL/CDF/PROD/STREAM2/DST01/DIL2-1F Dileptons //FNAL/CDF/PROD/STREAM2/DST01/ELE2-1F Central and plug electrons //FNAL/CDF/PROD/STREAM2/DST01/EXO2-1F Exotics //FNAL/CDF/PROD/STREAM2/DST01/MBO2-1F Min bias //FNAL/CDF/PROD/STREAM2/DST01/MUO2-1F Central Muons //FNAL/CDF/PROD/STREAM2/DST01/PHO2-1F Photons //FNAL/CDF/PROD/STREAM2/DST01/PSI2-1F J/psi //FNAL/CDF/PROD/STREAM2/DST01/QCD2-1F Jets and sumet h) Global STREAM2 DSTs ====================== These are available for 6.0 production before run 41936 and for 6.01/6.1 production after run 42294 and are located on 8mm tape. The information can be obtained from the PM database: $ setup production_manager $ @production_manager$programs:p_m_tape_report STR2_0F (for 6.01 production use STR2_1F and for 6.1 use STR2_3F) This produces a file in the users current directory called STR2_0F_TAPE_INDEX.TXT or STR2_1F_TAPE_INDEX.TXT the contents of which looks as follows: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Production_Manager STR2_0F Tape Index generated 6-OCT-1992 14:23:10.27 File_Name Tape Date_On_Tape --------- ------ ------------ R40014AB.STR2_0F CC2966 28-AUG-1992:00:31:56.83 GOOD R40015AA.STR2_0F CC2966 28-AUG-1992:00:51:29.32 GOOD R40021AA.STR2_0F CC2966 28-AUG-1992:04:43:17.27 GOOD R40021AB.STR2_0F CC2966 28-AUG-1992:04:59:58.03 GOOD R40021AC.STR2_0F CC2966 28-AUG-1992:05:13:40.40 GOOD R40023AB.STR2_0F CC2966 28-AUG-1992:06:35:23.76 GOOD R40023AC.STR2_0F CC2966 28-AUG-1992:07:39:43.02 GOOD R40023AD.STR2_0F CC2966 28-AUG-1992:08:35:39.32 GOOD R40023AE.STR2_0F CC2966 28-AUG-1992:08:50:40.85 GOOD R40041AB.STR2_0F CC2966 30-AUG-1992:01:35:32.64 GOOD R40041AC.STR2_0F CC2966 30-AUG-1992:04:09:04.60 GOOD ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3. How to get luminosity information ==================================== A separate stream is kept which has the luminosity information for each file. The following is from an Info posted by Mark Bailey. A new command file has been set up to drive all the varieties of luminosity-summing routines. To run this file, do the following: $ setup run_sum development all $ lum_control You will then be asked which of the available options you wish to use. The first option is whether to use luminosity from express or production(default). Then you will be asked whether to sum the luminosity by file or by run(default). If you opt to sum by run, you will be asked the beginning and ending run numbers. If you opt to sum by file, you will be prompted for the name of a file which contains the list of files for which you want luminosity totals. The file should have only one file name per line. Duplicate file names will be discarded. Examples of acceptable files are (express) E40100AA E41771AB E42371AA E40100AB ... or (production) S41000AQ T41000AQ R40100AA R40100AB T41771AC R41771AC ... The last option is whether or not to specify the trigger name. This is useful if you want the live luminosity for a trigger that has been pre-scaled. If you select this option, you will be prompted for the trigger name. All letters should be capitalized. Standard wildcarding is allowed (% for single characters, * for character strings). Up to 20 matches will be stored for ambiguous trigger names. If more than one of these triggers is present in the same run, only the first match will be used, so the resulting sum will be useless. The only practical application is for different versions of the same trigger, like MY_FAVORITE_TRIGGER_V%. Other lum* commands will not be maintained. Please use Lum_Control instead. I will very much appreciate reports of any problems encountered, or suggestions for enhancements. Mark If you have questions or find problems with luminosity information then send mail to RUN_SUM$CONSULTANT. Additional information on the luminosity stream can be found in c$doc:lum00.txt. 4. How to allocate a user tape ============================== A list of blank VAX labelled tapes that are avaliable for users are stored in a tape database. To access this database type $ @CDF$ROOT1:[CDF_PDM.CDF$TAPES]MASTER_MENU or $ TMASTER (this is defined when you setup offline) This puts you into a menu. Option L puts you into the menu for selecting a user tape (RK tapes) to use on FNALD that will be vaulted. To get tapes to use in B0/CDF Trailers see Rich Krull, or supply your own and label them yourself on a workstation with a tape drive - see the VMS command INITIALIZE, i.e., INITIALIZE device-name[:] volume-label. 5. FATMEN and the SILO ====================== There a number of documents and infos about how to use FATMEN to access the files in the STK SILO. Lingfeng Song is the expert in this area (FNALD::LINGFENG). To use FATMEN and access files in the Silo you need to do the following: $ Setup FATMEN $ Setup VAXTAP Keep an eye on CDFNEWS to see whether development or production should be used - the lines above will give you production. CDFNEWS will also list changes and improvements. To find old bulletins you can do "set folder cdf" and then "set archive" when inside CDFNEWS. A list of documents about FATMEN: The FATMEN manual - available from the Computer Divison Library PU0131 FATMEN$DOC:INTRO.TXT FATMEN$EXAMPLES:FULL_TEST.MEMO C$DOC:SILO.TXT Some useful tips for the SILO (from INFOs posted by Eric Wicklund) a) You can see what data is on real disk by using the following commands: $ FM FM> CD PROD/VAL/DST FM> LS * TTY R and look for the "Staged: Y" notation. Clearly, your jobs will run much faster if they use data that is already on real disk. b) One can find out which files have recently been put into the STK Silo with commands similar to: $ FM FM> CD PROD/VAL/DST FM> SEARCH * CREATED=920926- 6. The TPID bank ================ The events that pass through full production (not express-line) have their history recorded in the TPID bank. Below is an example of a dump of this bank. It allows the user to locate the full production file name on which the event can be found. Once the file name is known the production manager database can be searched to locate the tape on which this file can be found. DUMP OF TPID BANK ============================================= Online Tape Drive device Name : MKA200: Raw Data Tape Label : CB2215 Raw Data File Name : T43455AC.RAW File # 2 of production chain is : T43455AC.STR File # 3 of production chain is : T43455AC.STR1_1F