FATMEN at D0 Short User's Guide by Krzysztof L. Genser (Send remarks to FNALD0::FATMEN_MGR) (Updated 08-NOV-1994) *) What is FATMEN ? FATMEN stands for Distributed File and Tape Management System It is a file database plus database query tools and file access tools. It uses the concept of the so called generic names which point to the real files. *) How to use FATMEN ? To access files simply specify the generic name (in upper case) as the input file name for CALOR_OFF, D0USER or EXAMINE. The rest is done by the system for you. Some of the above frames accept lists of files and in these cases one can give the lists in the generic form. Refer to the frame manuals for the specific details. In order to use FATMEN make sure the program is linked correctly. Before you link it do put FATMEN library in official/test/beta mode depending on what level you want. The specific comands are: (official mode is the default one) $ LIBTEST FATMEN $ LIBBETA FATMEN If you do not have symbol LIBBETA defined do: $ @D0$BETA_UTIL:SETUP_LOCAL_CMS If you want to avoid problems (or simply to be able to do anything) use FATMEN version: FATMEN 1.85/07 940110 11:51 D0 CERN PROGRAM LIBRARY FATMEN= or newer. If you do not see "D0" in the FATMEN header printed it means you are not using the correct version. *) How to obtain a list of generic file names ? Use either FATMEN FM shell or FAT_LIS a COMPACK based query program searching the FATMEN Catalogue and prompting you what is available. **) How to use FAT_LIS ? Do $ @D0$FATMEN:SETUP $ FAT_LIS position the cursor using the arrow keys press DO (PF1) key while the cursor points to "FATMEN selection" press DO (PF1) key while the cursor points to "setting the path" this will prompt you to select the "data origin" position the cursor on the data origin you are interested in TBDA - stands for Test Beam Data CMDA - Data from D0 Commissioning CLDA - Collider Data CLMC - Collider MC now you may press DO key twice or RETURN key once. You may want to use HELP (PF2) key also. The program will present you the current default generic name for the specific data origin. Position the cursor on the field you would like to change. Press DO key once. If the program says there are no files like this, it means the path on the left side of the cursor excludes all the files. You may want to modify the fields on the left side then. If the program returns to the same filed changing it or not and says only one element present, this is exactly what it means. The only element was placed in the filed you tried to change. It is OK. If the program presents you window with a list of names you may use arrow keys or PAGE_DOWN, PAGE_UP keys to move around it. To guide you better the last element in the list is shown in the message window. You may now chose one element from the list or try to select first field which is WILDCARD. This will prompt you for your selection. You may use wildcards like * - unspecified number of characters,% - one character, but also you may specify ranges of runs or partitions like R0413(1:20)7P(1:21) etc... Once you have established the search patch you may press BACK (PF4) key. Then perform "searching the catalog" this will prompt you for "output_file_name" if you just press RETURN it will be FAT_LIS.GEN After the search is successfully performed you may EXIT You may now see which files were found by looking in output_file_name.GEN The file can be defined as the input file list for CALOR_OFF and D0USER. FAT_LIS can be used in batch. The command sequence: $ @D0$FATMEN:SETUP $ FAT_LIS $ FAT_LIS_BATCH produces some instructions to be followed. FAT_LIS has a limit of 10000 files to be considered for the lists. Watch the message screen or batch log file if you did not reached the limit. *) How to use FATMEN Shell FM ? One can also use the FATMEN shell FM to make more sophisticated queries of the catalogue. It has quite extensive help. The commands of interest are ld, ls, cd, pwd, find, fc, search. Doing help command will give you usually sufficient information. Here are some command examples: $ FM invokes the shell FM> LD lists directories FM> CD //FNAL/D0/CLDA/92P1E18/DST01REU1117/ALLALL00WZEXX20 sets the default directory FM> FC counts files in the directory FM> LS R043238P(01:05) -b prints list of files with location and usage information FM> LS R043238P(01:05) -r checks whether files are on the disk or not (use with caution, opening files via DFS takes some time...) FM> set/mediat 1 sets media type to disk for the LS and SEARCH commands FM> set/mediat 4,5 sets media type to single and double density 8mm tapes FM> set/mediat 1,4,5 sets media type back to default General hint using FM: if you ever hit ^C or ^Y while in FM exit and enter FM again. KUIP (also used by interactive PAW) gets extremely confused by ^C or ^Y. *) How to copy a file to your local area ? it is done using the FM Find command. This will stage the file on the file server if necessary. the exact syntax is: FM> FIND generic_file_name unit_number e.g. FM> FIND R043229P01 88 This should create a logical e.g. FOR088 (if unit_number=88) assigned to the file of interest. After you exit the FM FM> exit you can do: $ copy FOR088 your_newly_created_file_in_your_area If you are planing to use many files from a tape it is advisable to use prestage see below. *) How to read many tape files **) Prestaging or the so called Server Staging Mode The Server Staging Mode was designed to decrease the number of tape mounts if more than one file is to be read from the same tape and to save the space on the staging disk to use it only for the files which are really needed. The server tries to combine all the file stage requests from the same tape during the time the staging job is pending or executing, reducing the number of mounts and by doing so shortening the overall staging time and diminishing the wear of tapes. The Prestaging is another concept which allows to prepare the files to be analysed ahead of time, so that the analysing job does not wait for files to be staged or at least lets the files to be prestaged while the analysis job is reading the first files. Now, how to invoke Prestaging. All what needs to be done is to create a file e.g. FAT_LIS.GEN with the list of files to be analysed as usually for CALOR_OFF or D0USER and then do: $ LIBTEST FATMEN ! if not done before $ LIBBETA FATMEN ! if not done before $ PRESTAGE FAT_LIS.GEN queue_characteristics and then submit your normal job with the same queue_characteristics. Note that PRESTAGE just makes a request for files to be placed on the staging disk. IT DOES NOT PHYSICALLY STAGE THEM. One should keep in mind that the lifetime of the files on the staging disk is about 12h, so it does not make sense to prestage more files than can be read within this period of time as it will effectively decrease the lifetime of files even more. *) Answers to some commonly asked questions: **) How to do FATMEN search for files from a given run Here is a way to find ALL stream STA files from a given run e.g. 56639 $ FM FM> ls //FNAL/D0/CLDA/92P1E18/STA01REU%%%%/ALLALL00NONEX00/*/R056639P%% -wi For completenes this is how to find an INSPILL stream RAW files from this run: FM> ls //FNAL/D0/CLDA/92P1E18/RAW01DAQ0001/INSALL00NONEX00/*/R056639P%% -wi If you do not know antything more then the run number (and you do not want to go to the controll room to find the type of run) you can try the time consuming command: FM> ls //FNAL/D0/CLDA/92P1E18/RAW01DAQ0001/*/*/R056639P01 -w The same wildcarding applies when using FAT_LIS **) How to get a list of the most recent files from a specific stream To find e.g. EXP DST files reconstructed with version 101% D0RECO catalogued between Feb 1st, 1993 and Dec 31, 1995 FM> logl -1 FM> search //FNAL/D0/CLDA/92P1E18/DST01REU101%/EXPALL00NONEX00/*/* _ FM>_ cataloged=930201-951231 -gix output=FAT_LIS.GEN FM> exit Note the underscore "_" to indicate that the next line is a continuation line. The above search is very resource and time consuming. The output created is quite similar to the one of FAT_LIS, but contains empty lines which can be removed using an editor. *) Where to find (much more detailed, but not D0 specific) information ? FATMEN is fully described in the manual (PU0131) available from the Computing Division Library (High Rise, 8-th floor). The manual is fairly descriptive, contains index and tables of all FM commands.