environment for the objectivity should be set properly
1) To run lock server run
> oolockserver
2) To check lock Server is running or not
> oocheckls
3) To kill lock server
> ookillls -force
some time lock server just start hanging,
for that - stop AMS first to stop the AMS, see the AMS
steps.
- try to kill the lock server using ookillls -force,
if it does not work then do a "ps -el | grep ools"
and kill each ools process like
> ps -el | grep ls
8 S 9811 14067 13946 0 51 20 65a62fc0 735 65a63030 ? 2:22
ools
8 S 9811 26548 14067 0 41 20 62771648 735 ef7efc74 ?
0:00 ools
> kill -9 14067 26548
> ps -el | grep ls - run oolockserver and then run oocheckls to see that lock server is running properly.
- cleanup the locks ( see step 4) - run the AMS server again ( see AMS steps) 4) To cleanup locks - oocleanup
AMS: ---- -
environment for the objectivity should be set properly
1) To run AMS - oostartams for enstore-AMS run
/afs/fnal.gov/files/home/room2/cmsprod/enstore_ams/AMS/3.4.5.0/ams-server
2) To check that AMS is running or not
> oocheckams
3) To kill/stop the AMS
oostopams
some times AMS just start hanging or refuse to stop due to some open lock, in this case you have to kill the AMS processes like this one
"ps -el | grep ooams"
and kill each ooams process like
> ps -el | grep ls
8 S 9811 14067 13946 0 51 20 65a62fc0 735 65a63030 ? 2:22
ooams
8 S 9811 26548 14067 0 41 20 62771648 735 ef7efc74 ?
0:00 ooams
> kill -9 14067 26548
> ps -el | grep ls
> on cmsun1 system as AMS process also start encp process
to transfer files from enstore to disk, so also look for any encp processes
for AMS-enstore. e.g
on cmsun1 system just killing ooams processes is not
enough, for cmsun1, first kill all encp processes and then kill ooams process.
Once all ooams processes are dead, cleanup the locks (
see lock steps ) and start the AMS again.
Attaching DB files from one Federation to another: -------------------------------------------------
There are 3 steps for attaching/checking DB files from one federation to an other
1) Attachment of DB files
2) Copy DB files (if needed)
3) Check the proper attachment of DB files
1) Attachment of DataBase files from Production Federation to User Federation:
Let's say, we have produced some Hits or/and Digis data and we want to attach this data to the user federation. There are some things that we have to do before we can attach data to user federation. As the User federation is using AMS-Enstore interface, the files which we want to attach with the user federation should exist under the enstore directory which is known to AMS-Enstore.
In our case all User Federations share the same enstore directory for file staging. This directory is
/pnfs/cms/UserFederation/data
We have to backup their DataBase files only ( not META data) in such a way that they exist under /pnfs/cms/UserFederation/data directory.
Currently we backup our data to /pnfs/cms/production/Federation_backups/cms_production/jetmet_production/data directory but we create a link
/pnfs/cms/UserFederation/data/jetmet_production -> /pnfs/cms/production/Federation_backups/cms_production/jetmet_production/data
so every thing which is backed up in
/pnfs/cms/production/Federation_backups/cms_production/jetmet_production/data
directory will also be visible from
/pnfs/cms/UserFederation/data/jetmet_production
Once the backup is complete, we need these parameters
a)Production Federation Name, lets say gallo.fnal.gov::/data/jetmet_production/C_federation/jet0900.boot
b) Dataset Name, lets say jm_sm_ttj
c) Owner of Hits, let say jetHit120_452_FNAL_C
d) If Digis are also done then Owner of Digis, lets say 2e33_jetDigis120_452_FNAL_C
e) Which User Federation to use, as there are 2 or 3 user federations. We should be clear which user federation to attach? lets say cmsun1.fnal.gov::/usr/cmsun1/r5data08/UserFederation_jetmet/jet0900.boot
f) Path where the DataBase files are Backed up, lets say /pnfs/cms/production/Federation_backups/cms_production/jetmet_production/data
g) Path where the system files are backed up, lets say /pnfs/cms/production/Federation_backups/cms_production/federation1/system h)Now what we have to do is to create a file (let say p2u.config) which should have the following set entries
#boot file of the User Federation
DESTINATION_FEDERATION=cmsun1.fnal.gov::/usr/cmsun1/r5data08/UserFederation_jetmet/jet0900.boo#
#Number of threads you want to run
THREADS=6
#script directory path
SCRIPT_DIR=/afs/fnal.gov/files/home/room1/muzaffar/script/cms_production/db_tools
#copy attach script
SCRIPT=copyattach_DB.pl
#name of the new host for the DB files
NEW_HOST=cmsun1.fnal.gov
#Production federation boot file
SOURCE_FEDERATION=gallo.fnal.gov::/data/jetmet_production/C_federation/jet0900.boot
#directory where these DB files would be available
NEW_DIR=/usr/cmsun1/r5data08/UserFederation_jetmet
#any valid PERL filter for files
FILTER=--filterpath "META\.(jetHit120_452_FNAL_C|2e33_jetDigis120_452_FNAL_C)\."
#different options for the copy script, like either only attach or copy and attach
OPTIONS=--attachonly
#either has "" value or DONE, if DONE is provided then this entry will be skipped
STATUS=
#end of entry
START=YES
This file can have more then one entries. e.g. a sample file could look like ( an actual config file is available in
/afs/fnal.gov/files/home/room1/muzaffar/Production_work/jetmet_p2u_attach.conf
) Note: If we not provide any parameter for any entry in this file then the parameters of last entry will be used. So one can provide common parameters on top of this file. Also Filter parameter is a regular Perl expression, if you want to match a '.' or '/' then you should put a '\' before it.
#Start of File
#some common parameters
DESTINATION_FEDERATION=cmsun1.fnal.gov::/usr/cmsun1/r5data08/UserFederation_jetmet/jet0900.boot
SCRIPT_DIR=/afs/fnal.gov/files/home/room1/muzaffar/script/cms_production/db_tools
SCRIPT=copyattach_DB.pl
NEW_HOST=cmsun1.fnal.gov
THREADS=4
#end of common parameters
#Common parameter for a particular production federation
############################################################################### #From
velveeta.fnal.gov::/data/jetmet_production/D_federation/jet0900.boot
################################################################################
SOURCE_FEDERATION=velveeta.fnal.gov::/data/jetmet_production/D_federation/jet0900.boot
################################################################################ #specific parameters for each entry #start of
1st entry NEW_DIR=/usr/cmsun1/r5data08/UserFederation_jetmet
FILTER=--filterpath "META\.2e33_jetDigis120_452_FNAL_D\."
OPTIONS=--attachonly
STATUS=DONE
START=YES
#End of 1st entry
################################################################################
#Start of 2nd entry
NEW_DIR=/usr/cmsun1/r5data08/UserFederation_jetmet
FILTER=--filterpath
"META\.2e33_jetDigis120_454_FNAL_D\."
OPTIONS=--attachonly
STATUS=DONE
START=YES
#End of 2nd entry
################################################################################
>br>#Start of 3rd entry
NEW_DIR=/usr/cmsun1/r5data08/UserFederationDB/jetmet_production
FILTER=--filterpath
"(\/2e33_jetDigis120_452_FNAL_D\/jm_hlt1520\/)|(jm_hlt1520\.2e33_jetDigis120_452_FNAL_D\.)"
OPTIONS=--attachonly --cmsdir
STATUS=DONE
START=YES
#End of 3rd entry
################################################################################
#Start of 4th entry
NEW_DIR=/usr/cmsun1/r5data08/UserFederationDB/jetmet_production
FILTER=--filterpath "(\/(2e33_jetDigis120_454_FNAL_D|jetHit120_454_FNAL_D)\/jm_pi_pt30_eta395405\/)|
(jm_pi_pt30_eta395405\.(2e33_jetDigis120_454_FNAL_D|jetHit120_454_FNAL_D)\.)"
OPTIONS=--attachonly --cmsdir
STATUS=DONE
START=YES
#End of 4th entry Now as all of the above entries have "STATUS=DONE", this mean these entries will be skipped.
If you want to add a new entry for the same Production Federation then you have to just provide the NEW_DIR,FILTER,OPTIONS, STATUS and START parameters. like NEW_DIR=/usr/cmsun1/r5data08/UserFederationDB/jetmet_production FILTER=--filterpath "(\/(2e33_jetDigis120_454_FNAL_D|jetHit120_454_FNAL_D)\/jm_tttau\/)|(jm_tttau\.(2e33_jetDigis120_454_FNAL_D|FILTER=--filterpath "(\/( OPTIONS=--attachonly --cmsdir STATUS= START=YES all other values will be used from the previous entry. If you want to attach DB files from a different Production Federation then for new entry you also have to provide the SOURCE_FEDERATION, like
SOURCE_FEDERATION=gallo.fnal.gov::/data/jetmet_production/C_federation/jet0900.boot NEW_DIR=/usr/cmsun1/r5data08/UserFederation_jetmet FILTER=--filterpath "META\.2e33_jetDigis120_454_FNAL_C\." OPTIONS=--attachonly STATUS= START=YES Once you have created this file,
then you have to run the "p2u_attach.pl" script with a single arg i.e the p2u.config file path. You can find this script in the
/afs/fnal.gov/files/home/room1/muzaffar/script/cms_production/db_tools
directory or can checkout from the CVS
cvs -d :pserver:anonymous@cdcvs.fnal.gov:/cvs/cd_read_only login
Password is "anoncvs"
cvs -d :pserver:anonymous@cdcvs.fnal.gov:/cvs/cd_read_only checkout cms_production/db_tools cd cms_production/db_tools ./install-scripts.pl
Make sure that AMS/Lock server for user and production federations are running. "p2u_attach.pl p2u.config" script will attach the files by reading the production federation and writing in the User federation. if you run "p2u_attach.pl p2u.config" and you get error for some files then you just have to rerun it. Once every file of an entry is done then edit this config file again and put STATUS=DONE for that entry, so next time this entry can be skipped At the end of this script you will find a ./log directory. this directory has 4 sub directories fail done dead status if you did not see any file in log/fail directory then this means that all files are correctly attached. if you see some log/fail files then read those files are see what was those errors and try to remove those errors. once you are sure that those errors are gone, delete the log directory and re-run the same job again.
2) Copy DB files on Disk:
We only need METAData DB files on disk for User Federations as the actual DB files are fetched via AMS-Enstore. To copy the META Data DB files you have to know the enstore directory where the backup of these files were taken and the path on the disk where you want these files. e.g. in our above config file NEW_HOST=cmsun1.fnal.gov NEW_DIR=/usr/cmsun1/r5data08/UserFederation_jetmet FILTER=--filterpath "META\.2e33_jetDigis120_452_FNAL_D\." the path of META data DB files is /usr/cmsun1/r5data08/UserFederation_jetmet on cmsun1.fnal.gov.
So we have to copy files from enstore to this disk directory. let say the enstore backup directory path is
/pnfs/cms/production/Federation_backups/cms_production/D_federation/system/010921
then there is another script which can be used to copy the required files from enstore to disk. e.g.
md2dd.pl --enstoredir /pnfs/cms/production/Federation_backups/cms_production/D_federation/system/010921 \ --diskdir /usr/cmsun1/r5data08/UserFederation_jetmet \ --threads 4 \ --filter "META\.2e33_jetDigis120_452_FNAL_D\."
this will copy all *META.2e33_jetDigis120_452_FNAL_D.* files from
/pnfs/cms/production/Federation_backups/cms_production/D_federation/system/010921
enstore directory to the /usr/cmsun1/r5data08/UserFederation_jetmet disk directory by running 4 encp processes. this will also create a ./log directory and you can see the log/fail directory for any failed jobs, if there are any fail files in this directory then read those files and try to remove those errors, once errors are removed, delete the log directory and re-run the same job i.e
"md2dd.pl --enstoredir ...."
If you have to copy the actual DB files too on disk then just select the proper disk and enstore directory and use the proper filter and run the md2dd.pl again. this is useful for the snickers.fnal.gov ntuple federation, as on that system the AMS is not an enstore-AMS so we have to copy all DB and MetaData files.
3) Checking the DB files.
Once the DB files are attached and required DB files are copied on disk, run the dsDump to see if this dataset is readable or not. dsDump could be run from any system , just se your OO_FD_BOOT environment to point to the User Federation and run it. if dsDump ran ok, then now we are ready to run some ntuple maker jobs for this dataset.