SDSS use of FNALU cluster GETTING AN ACCOUNT ------- -- ------- If you do not have any accounts on any FNAL machines, you first have to read and sign the "Fermilab Policy on Computing" located at http://www.fnal.gov/cd/forms/policy_form.html. If you already have an account on any FNAL machine, then we already have your signature. You can then sign up for an account on the FNALU cluster at http://www.fnal.gov/cd/forms/acctreq_form.html Please fill in the information requested. Some hints: The Experiment is SDSS; the Affiliation is your home institution. For the supervisor/spokesperson, fill in Chris Stoughton with email stoughto@fnal.gov. (Unless I am on vacation or away, then use Steve Kent or someone else who can approve the request.) You are asking for an account on "UNIX Systems" so click that box and choose fnalu from the menu directly below it. You do not need a "project request form." Fill in a sentence about sdss int eh "briefly describe computing to be done" box. In the "Please include any additional informatin in the area below" box, it is also helpful to request being put in the unix group SDSS. WHAT MACHINES ARE THERE ---- ------- --- ----- Here are the Dec and SGI nodes available. speed memory swap scratch name os mode #proc Mhz MB MB GB DLT drives fsgi02 IRIX+6 b+i 16 250 512 2509 7.9 6(3) fsgi03 IRIX+6 b+i 16 195 2816 4096 8.9 3(2) fsgb01 IRIX+6 b 2 180 256 256 - - fsgb02 IRIX+6 b 2 180 256 256 - - fsgb03 IRIX+6 b 4 180 512 2048 - - fdei01 OSF1 b+i 4 275 621 767 - 4(3) fdei02 OSF1 i 3 200 495 ? 10(5x2) - fdeb01 OSF1 b 1 233 54 256 - - fdeb02 OSF1 b 1 233 54 256 - - fdeb03 OSF1 b 1 233 54 256 - - fdeb04 OSF1 b 1 233 54 256 - - fdeb05 OSF1 b 1 233 54 256 - - fdeb06 OSF1 b 1 233 54 256 - - fdeb07 OSF1 b 1 233 54 256 - - mode: "b" means batch jobs only; "i" means interactive only; "b+i" means both. scratch: locally available scratch space under /usr/scratch/ The node fsgi03 is special. In addition to sdss products it has idl installed. It also has 500 Gb of disk dedicated to eag sdss science analysis. Only FNAL people can write to these disks. Under /usr/sdss/data01/imaging there is the usual directory tree of imaging outputs. Everyone can read and copy these files to their home institution or to scratch areas for processing. If the CPU becomes saturated on fsgi03 we will ask non-FNAL people to please use fsgi02 or fdei02 for running jobs. PRODUCTS -------- As with all FNAL machines, before you use a program you (usually) have to set it up. So if you want to use emacs, first type "setup emacs" to get the correct version of emacs added to your path. SDSS products are located in a different place, so you need to add some magic to the top of your .cshrc file to let "setup" know where they are. Copy ~stoughto/.cshrc to your place or make your .cshrc look like this at the top: setenv UPS_EXTRA_DIR /afs/fnal.gov/files/code/sdss/p/upsdb if ( -f "/afs/fnal.gov/ups/etc/setups.csh" ) then source "/afs/fnal.gov/ups/etc/setups.csh" endif The order is important. WHY CAN'T I WRITE IN MY HOME AREA? --- ----- - ----- -- -- ---- ----- The home areas are AFS disks, and use a security scheme which involves tickets. If you mysteriously can no longer write in your area, the command "klog" will ask for your password and then give you a fresh ticket. You may have to do this after logging in, and after leaving a session idle for a long time. WHY SHOULD I USE BATCH QUEUES? --- ------ - --- ----- ------- This machine is shared with others at FNAL so there are a few ground rules we have to follow. The intention is to give you significant compute resources for science analysis, but not the entire machine! The first rule is that no interactive job can consume more than 30 minutes of CPU time. There is a system job which "bags" jobs which are longer than that. So if you process all of a sudden goes away, it probably got bagged. You run longer jobs with the fbatch facility. % setup fbatch % fbatch_queues -- this gives you a list of the queues. For example, the queue named "4hr" has a 4 hour CPU limit. (This might be normalized in some way to take into account the CPU speed.) % fbatch_sub -q 4hr -m fsgi03 job.csh Submit the executable file job.csh to the 4 hr queue. You get mail when it is finished. % fbatch_jobs -- see what jobs you have running % fbatch_peek -- take a look at the log file at it develops. You need some magic at the beginning of your job.csh file to make the various products work: # To set up any product (including IDL) setenv UPS_EXTRA_DIR /afs/fnal.gov/files/code/sdss/p/upsdb source "/afs/fnal.gov/ups/etc/setups.csh" # now setup what you need... setup astrotools setup idl The file ~stoughto/fbatch/example.csh is a "hello world" example that can be submitted with the command: fbatch_sub -q 4hr -m fsgi02 example.csh This will send you mail when it finishes. See the man page for fbatch_sub for more options. DO I HAVE TO TYPE IN A PASSWORD FOR EACH BATCH JOB I SUBMIT? -- - ---- -- ---- -- - -------- --- ---- ----- --- - ------ No. If you do the following: source $FBATCH_DIR/bin/fbatch_setpgp.csh it will ask for your password and set an environment variable with an encrypted ticket. WHERE CAN I RUN IDL? ----- --- - --- ---- IDL is licensed only on fsgi03. Various useful IDL libraries are included in the local/ directory included in IDL_PATH when you setup idl. WHERE IS THE DATA? ----- -- --- ----- On fsgi03:/usr/sdss/data01/imaging there is the usual directory structure.