Command to run D0 framework executable. Usage: rund0exe [...] Command line options: General ------- -h - Get help. Global parameters ----------------- -exe=exe - Name of executable -rcp=rcp - Name of framework rcp script -rcppkg=rcppkg - Name of framework rcp package -outpath - Send the output to this directory. The default is the current directory -initscript=script - Initialization script for executable (executed script) -userscript=script - User provided initialization script for executable (executed script) -sourcescript=script - Initialization script for executable (sourced script) -usersourcescript=script - User provided initialization script for executable (sourced script) -userpostscript=script - User provided finalization script after executable - must be in $PWD -localbuild - Use local build instead of official version -localrcp - Use local rcp's in addition to official ones -localfwkrcp - Use local version of main framework rcp -localrundata - Use $SRT_PRIVATE_CONTEXT/rundata properly whne not using the -localrcp option -local - Same as -localbuild -localrcp -localfwkrcp -localrcp -name=name - Override default job name -rcpfiledb[=] - Set up a particular version of rcpfiledb. With no argument use the current - version. Otherwise use the specified version. Executables built with shared library support --------------------------------------------- -shlib - Use the version of the executable built using shared libraries To build an executable using shared libraries you should do the following: setup D0RunII ........ [ -O SRT_QUAL=maxopt] setenv LINK_SHARED yes gmake all rund0exe -shlib ..................... Run Time Environment options ---------------------------- -userte - Use run time environment (rte) directory -rtepkg= - Name (lc) of exe (yes!) within rte to be invoked (rtepkg will be changed to rteexe in a future release) -rtepath= - Path from current working directory to rte directory. if submitting to remote node this will be tarred and copied to the remote node unless -rtecached is set -rtecached - use if submitting to remote node which has a local copy. If so, -rtepath should be the local path on remote node. (if you have not setup D0RunII, you need to set some variables: setenv SRT_PUBLIC_CONTEXT p14.02.00 // or whatever is current setenv SRT_SUBDIR Linux2.4-KCC_4_0 setenv MACH_OS Linux as d0tools likes to know what it is doing... ) Modes of running (default mode is interactive) ---------------- -batch - Use batch queue [default = short] -debug=debugger - Use interactive debugger: available options totalview (recommended for p14), ddd (recommended for p16 and later), gdb. -maxopt - Use optimized build. -maxopt-strip - Use optimized stripped build (smaller) -jobname=jobname - Specify batch job name [default = rund0exe] clued0 (PBS) batch system switches: ------ -cput=time - Specify maximum CPU time. [default = 3:00:00] -mem=mem - Specify maximum memory. [default = 500Mb] -node=node - Specify node where job should be run [default = system chooses] -thisnode - Run job on the current node. -nomail - Tell PBS not to send E-mails to the user upon job completion. cab (PBS) switches: --- -cabsrv1 - Use the CAB (central analysis backend) batch system. -cabsrv2 Detailed documentation available at: http://www.nuhep.nwu.edu/~schellma/cab/cab_doc_v2.html CAB can get input either from SAM (preferred), cross mounted disks (project disks or home directories), or local disks. In the last case files are transferred to CAB prior to the job start (with a 5 GB limit). -fzk - Use CAB style jobs at the computing center in Karlshruhe (FZK) You must be logged on d0.fzk.de to submit jobs to FZK -q= - submit to a specific batch queue -scratch= - need to have this as you have to make a tar file with your code in it. Output is returned to this area as well -cabtar - use a pre-existing tarball which is already in -scratch DO NOT INCLUDE THE PATH! If cabtar is not set, a tarball is made from the contents of $PWD. To prevent the creation of the tarball use -cabtar=empty (but then you are limited to the official compiled version of the D0 code and the official RCP parameters). -jobs=num - number of parallel jobs you can run with single consumer works only with SAM input. -caboutpath - path for the output, defaults to -scratch (synonym for -outpath) -cabouthost - host for the output path, defaults to submission node -cabmonitor - check every 5 minutes the events.read file and print the total number of events read so far and the name of the last opened file in the job logfile -cabusenfs - when combined with the -filelist option read the input files through NFS without making a local copy (works only with input from project disks and from home directories) -cabskipchecks - do not perform checks on input files when reading from disk (ONLY FOR ADVANCED USERS, to be used with -filelist -cabusenfs) -cabproduction - allow the use to run separately the SAM start project, the SAM stop project and the workers (use start, stop or continue as options). When the start or stop options are used no worker is submitted. When you use this option you have to ensure that the workers are started after the SAM project has been properly started. And stop the project only when you're sure that all the workers have finished. -cabnodetype - specify properties for the nodes the batch jobs are to run on. Multiple values should be separated by ':'. The special value 'any' will override any defaults and force an empty property list. -cabdebug - debug the job submission to CAB, extensive printouts no jobs are submitted -nomail - Tell PBS not to send E-mails to the user upon job completion. Special *temporary* CAB feature: If you only want to include the ONE executable specified via your run command in your CAB tar file (thereby reducing its size and speeding up transfer), set the environmental variable CABTARSINGLEEXE to a non-null value. Options controlling input / output ---------------------------------- -filelist=filelist.dat - List of files/datasets to process -defname=defname - SAM project definition name -filecut=num - Maximum number of files to be delivered in a SAM project -jobfilelimit=num - Maximum numer of files to deliver to a single job. This does not work with older releases. -out=file - Write processed events to specified file Options to control which events are processed --------------------------------------------- -num=num - Specify number of events to process [default = all] -skip=num - Specify number of events to skip [default = none] -eventlist=eventlist.dat - Process only these specific collision ids eventlist.dat is a file containing a list of collision ids. -skipruns=badruns.dat - Skip these runs badruns.dat is a file containing a list of run numbers -skipevents=badevents.dat - Skip these specific collision ids badevents.dat is a file containing a list of collision ids. Other framework options ----------------------- -suppress - Disable printing of individual error message in log file (Instead, a summary is printed at the end of the job) -memreport - Enable framework memory leak checking (turns on framework switches -memusage -memreport) -filter_triggers=trg - Specify a list of L3 trigger names (separated by commas) used to filter events during the processing (requires that the executable is linked with RegTriggerFilter) -copyexe - Copy the execuable to the execution directory (instead of symlinking it). This allows rebuilding of the executable while a job is running (this option is ignored when running on CAB). SAM options ----------- -group=group - Specify SAM group [default = dzero] -datasetversion=version - Specify version of the SAM dataset [default = new] Special options --------------- -idletimeout= - Kill the job after it has been idle for the specified number of hours. A time of 0 disables the monitoring. (An idle job is one which has not updated its out or log files.) -fpe - Turn on floating point exception handling on Linux -nofpe - Disable floating point exception handling on Linux -runscript=script - Specify a custom script to run The specified script will be run before actually running the executable. Note that this script will only work reliably if it contains command line commands. It is intended for advanced users. -fwkparams - Pass any set of parameters to a framework executable You may pass any set of parameters to the framework executable you are running by using the -fwkparams option. This option MUST be the last one on the command line, followed by the actual parameters you want to pass to the framework. For example, rund0exe ... -fwkparams -option1 xxx -option2 yyy Expert options -------------- -v - Print out what would be done, but don't do it -t0 - minimal talkiness (default) (debug echo) -t1 - a little more talky -t2 - even more talky Deprecated options ------------------ -forcercpoutfile - Use this switch to force the usage of the output files specified in the WriteEvent rcp (ignore -out switch) With v02-04-22, the default is to have this behavior. It is overridden if the user uses the "-out" option. Description ----------- `rund0exe' can run any D0 framework executable. The user must specify the executable (-exe), the main framework rcp file (-rcp) and the package where that rcp file resides (-rcppkg). It is assumed that the D0 environment has been setup correctly by the user (e.g. setup D0RunII xx.xx.xx) before running or that the user has supplied a run time environment and specified that via -userte, -rtepkg, and -rtepath. The command creates a new directory where the executable will be run, and all output from the job will be placed. By default, the name of the directory is a combination of executable name, release version and input file name. To override this default name, use the -name option. The command supports running the executable interactively (default) or in a batch queue (-batch). In addition, it supports running debug mode (-debug). To run in batch mode, use the -batch option (only available on platforms with a batch system). The default batch queue is 'short'. To override this default, use the -q=xxx option. To use the debug mode, use the -debug option. You must have first setup the totalview or the ddd debuggers. Use -debug=totalview to debug with totalview, -debug=ddd to debug with DDD and -debug=gdb to debug with GDB. The command supports processing files either available on local disk, or available within SAM. To process a set of local disk files, create an ASCII file (e.g. mydata.dat) containing a list of files to process (one line per file). Then specify this list using the -filelist option (e.g. -filelist=mydata.dat). NOTE: you must specify the complete path of each file. For example, if you wish to process the files myfile1.dat and myfile2.dat, which are located in /scratch/1/mydir, you would include the following two lines in your "mydata.dat" file: /scratch/1/mydir/myfile1.dat /scratch/1/mydir/myfile2.dat To process files within SAM, first use SAM to create a dataset definition, and then use the -defname option. For convenience, users may optionally place the SAM dataset definition name into an ASCII file, prefacing it with the string "SAMDEFNAME:", and then use the -filelist option. To specify the number of events to process, use the -num option (e.g. -num=100). To skip a certain number of events, use the -skip option. If you would like to process only specific collision ids ( ), you may use the -eventlist option. Create a file with a list of desired collision ids and use -eventlist=thatfile.dat. For example, to process event number 123456 in run 191919, create a file (eg. myevents.dat) with the line "191919 123456" and use -eventlist=myevents.dat. By default, the specified framework executable must reside in the official release. To use a locally built exe, use the -localbuild. When using this option, you should execute this command at the base of that local build. If you would like to use the executable in the official build, but make a few changes to some rcp parameters in your local area, use the -localrcp option. For example, if you would like to change a parameter in a rcp file from package xxx, first checkout that package's rcp directory using the command "cvs checkout xxx/rcp" and modify your local copy. Then issue the rund0exe command with the -localrcp option. When -userte is specified, the framework rcp's for the executable (!) specified by -rtepkg are required to be within the runtime environment directory at runtime/framework_rcps/. E.g., for D0reco_x, you would have -rtepkg=d0reco_x and the rcp's would be at runtime/framework_rcps/d0reco_x Example - reco_analyze ---------------------- To run reco_analyze out of the p05.00.04 production release, % setup D0RunII p05.00.04 % rund0exe -exe=RecoAnalyze_x \ -rcppkg=reco_analyze \ -rcp=runRecoAnalyze.rcp \ -filelist=mydata.dat where mydata.dat contains a list of files to process. To process files in SAM, % rund0exe -exe=RecoAnalyzeSAM_x \ -rcppkg=reco_analyze \ -rcp=runRecoAnalyzeSAM.rcp \ -defname=my-dataset-definition-name To use a local version of an emreco rcp file, % cd myrundir % cvs checkout emreco/rcp --- modify appropriate rcp script -- % rund0exe -localrcp \ -exe=RecoAnalyze_x \ -rcppkg=reco_analyze \ -rcp=runRecoAnalyze.rcp \ -filelist=mydata.dat NOTE: If you want to run an executable out of a production release, and wish to modify one of the rcp parameters associated with a given package in that release, you will need to checkout that particular version of the package. For example, to use p08.09.00 RECO, and modify a parameter in emreco, do % setup D0RunII p08.09.00 % cvs checkout -r p08-br emreco/rcp # Get emreco from the "p08 branch" --- modify appropriate rcp script -- % rund0exe -localrcp \ -exe=RecoAnalyze_x \ -rcppkg=reco_analyze \ -rcp=runRecoAnalyze.rcp \ -filelist=mydata.dat Example - rund0exe with an rte ----------------------------- A run time environment has only the executable, such as D0reco_x, and the files needed by it at run time--no source code. To run D0reco_x out of a particular run time environment directory (named, say, rte0229 residing on /scratch/10/joeuser): % setup d0tools % rund0exe -exe=D0reco_x \ -userte \ -rtepkg=d0reco_x \ -rtepath=/scratch/10/joeuser/rte0229 \ -filelist=mydata.dat \ -format=data where the file mydata.dat in the current working directory contains a list of files to process. (For example, it could be the single line (with a trailing newline, of course): /home/joeuser/all_0000158994_146.raw You would make up a run time environment directory for D0reco_x onto the directory /scratch/10/joeuser/rte0229 as follows: Make a local release: % setup D0RunII t02.29.00 % newrel -t t02.29.00 MyLocalRel % cd MyLocalRel % d0setwa % setup d0rte % d0rte t02.29.00 /scratch/10/joeuser/rte0229 "" rcplink d0reco_x At this time, you would need to use the latest version of d0reco as the one built into t02.29.00 does not have everything needed for rte. To do this, insert the following lines after the d0setwa: % addpkg -h d0reco % gmake This deficiency should be removed shortly which will make this unnecessary. You could make a run time environment in this way for any package's executable as long as it and the packages which supply it run time data follow the d0rte publish and subscribe rules describe further in the d0rte package's documentation. Known Bugs ---------- In order to override a framework rcp, you must checkout the package and make a local build of the executable, and then use the -localbuild switch. Please report problems to melanson@fnal.gov