PSUB(l)                                                                PSUB(l)



NAME
       psub - Submits batch job to LCRM

SYNOPSIS
       psub [-A  date-time]  [-a account] [-b bank] [-bgl attributes] [-c con-
            straints] [-cpn cpus_per_node] [-creds] [-d jobid] [-dm] [-e file]
            [-eo]  [-exempt [exlist]] [-expedite] [-g [tasks][switch][@layout]
            [-i command] [-ke] [-ko] [-lc limit] [-ld limit] [-lF limit]  [-lf
            limit]  [-lM  jsize]  [-ln  nodedist] [-lo limit] [-lr limit] [-ls
            limit] [-lt limit] [-mb] [-me] [-mn] [-nc] [-net protocol]  [-net-
            type  type]  [-nobulkxfer] [-np] [-nr] [-o file] [-p priority] [-r
            jobname] [-re] [-ro] [-s shell-name] [-standby]  [-tM  time]  [-tW
            time] [-v] [-x] [script [session-args]]

DESCRIPTION
       The  psub  utility  is used to submit batch jobs to LCRM. On successful
       submission, LCRM spools the job to the LCRM control host where it  will
       generally remain until it is selected for execution. So, after you sub-
       mit a script, you may then change the contents of the script file with-
       out  affecting  the  job.  Attributes of jobs are entered into the LCRM
       job database on submission. See pstat, palter, prm, and prel for infor-
       mation about how to modify attributes or otherwise manage jobs in LCRM.

       On successful submission of the job, psub will  respond  with  the  job
       identifier associated with the job. This identifier uniquely identifies
       the job to LCRM throughout the LCRM domain.

       By default, LCRM returns the files produced by your job to the host and
       directory from which the job is submitted, under the following names:

           stderr:   <jobname>.e<jobid>
           stdout:   <jobname>.o<jobid>

       <jobname>  is the string specified by the -r option or is the base name
       of the script file if the -r option is not used or is your user name if
       you don't use the -r option and use the -i option or type in the script
       directly to psub. The jobname is truncated to 15 characters if a longer
       name  is  specified  or  implied.  The names of the standard output and
       standard error files can be set to whatever you wish by use of  the  -e
       and -o options, respectively.

       When  LCRM  schedules a job to run, it "commits" it to the native batch
       system running on the targeted host. If a job attribute or job  related
       action  is specified via an option to psub and that attribute or action
       is not supported by the native batch system, it is ignored.

       Options can be embedded within a script file prior to  the  first  exe-
       cutable  command  of the shell script by preceding them with the string
       "# PSUB". The characters "PSUB" must be upper-case and must not be sep-
       arated by white spaces.

       For  example, specification that mail is to be sent on termination of a
       batch job is as:

           # PSUB -me

       If the same option appears both in the script file and on  the  command
       line, the command-line option takes precedence. Multiple occurrences of
       an option on the command-line will cause an  error.  However,  multiple
       occurrences  of  an  option within a job script will result in only the
       last occurrence of the option being recognized.

       Comments may be included within embedded options in the script file  by
       preceding  them  with  a  #  character.  Comments can be appended to an
       embedded option by preceding them with the # character thus:

           # PSUB -A "01/23/93 18:00:00" # Do not run job before
                                         # 6 p.m. on Jan 23, 1993

       A typical extended example of the use  of  embedded  options  within  a
       script file follows:

           #
           #  Batch job shell script example:
           #
           #  PSUB -r myjob    # Set jobname to myjob
           #  PSUB -me         # Send mail at job termination.
           #  PSUB             # No more embedded options.
           #
           nmake all

       For  other  examples  of typical LCRM batch jobs, refer to the EXAMPLES
       section.

       If you are bringing a script from a host that runs NQS (without  LCRM),
       you may leave the embedded # QSUB options intact, psub will ignore them
       as will qsub ignore the # PSUB statements.

OPTIONS
       Note: on paired options, there must be a space between the  option  and
       the option value.

       -A date-time
           Holds  the  batch  job until the specified date and time. If both a
           date and a time are specified, then both must be enclosed in quotes
           so  that the shell does not treat them as separate tokens (e.g., -A
           "12/15/93 3:30PM"). The date-time is specified as follows:

                   [date] time [time-zone]

               Permissible forms for date are:

                   YYYY-MM-DD      DD-Month-YY      Month DD YYYY
                   YYYY DD-Month   DD-Month-YYYY    MM/DD YYYY
                   YYYY Month DD   DD-Month YYYY    MM/DD/YY
                   YYYY MM/DD

               If date is not specified, the current day is assumed.

               The month is specified in full or as any acceptable  prefix  of
               the  calendar month that is at least three characters. Month is
               case insensitive.

               The day is specified in full or as any acceptable prefix of the
               day  of the week that is at least three characters. Day is case
               insensitive. An optional  "."  may  follow  the  specification.
               Alternative specifications are as follows:

                   today         tomorrow

               Permissible forms for time are as follows:

                   HH<meridian>  HH:MM<meridian>  HH:MM:SS<meridian>
                   noon          midnight

               Permissible forms for meridian are as follows:

                   am            pm               m (indicating noon)

               In the absence of meridian, a 24-hour clock interpretation pre-
               vails.

               The time-zone is specified as any North American time  zone  or
               "GMT".  If time-zone is not specified, the local time zone pre-
               vails.

               Valid date-time examples:

                   "01-Jan-1988 12am, PDT"
                   "Tuesday, 23:00:00"
                   "11pm tues."
                   "tomorrow 23 MST"

       -a account
           Account that is to be associated with all  resource  usage  reports
           for  the  job.   The  account name must have been created by a LCRM
           administrator.

       -b bank
           Bank from which allocated resources is to be drawn. If you  do  not
           use  this option, resources used by the job will be drawn from your
           default bank. (See defbank.) If you do not have a default bank,  or
           your  default  bank  does not permit batch mode usage, you must use
           this option to specify a bank that does permit  batch  mode  usage.
           (See  bac.)  You  must  have  been  granted  permission  to use the
           resources allocated to the selected bank.

       -bgl attributes
           BGL specific attributes.  This  option  is  ignored  on  all  other
           machine  types. It specifies the geometry and other characteristics
           required for a job that is to run  on  the  IBM  BGL  machine.  The
           attributes must take the form:

           attributes := [size] [rigidity] [conntype] [usage]

           where

           size  :=  geometry=N[xM[xO]]  rigidity  :=  [no]rotate  conntype :=
           conn_type=mesh|torus usage := node_use=virtual | coprocessor

           The attributes may be in any order. None are required. The  default
           values  for  M  and O are 1. If the size attribute is not used, the
           default value for N is the value used in the -ln option. If the -ln
           option  is  also  not  used, the default size is 1x1x1. The default
           rigidity is to permit rotation.  The  default  connection  type  is
           mesh. The default node usage is coprocessor. Note: the defaults are
           consequences of SLURM design.  If  SLURM  defaults  change,  LCRM's
           defaults will change without notice as well.

       -c constraints
           Job  constraints.  The  specified  constraints determine the set of
           hosts on which the job must run. In order for a host to be a candi-
           date  to  run a job, it must have been assigned features that "sat-
           isfy" the constraints. An implicit feature of every host is its own
           name.  If there is no host that satisfies the given constraint, the
           job submission fails and an error message is printed. (See the CON-
           STRAINTS  section.)  If you do not use this option, then LCRM looks
           for a "default constraint" that may have been set for the  resource
           partition of which the host where the job is submitted is a member.
           If a default constraint is found, it is used to determine the hosts
           that  may  run  the job. If a default constraint is not found, then
           LCRM looks for the setting of a "localization" configuration param-
           eter.  If  the  localization parameter is GLOBAL, then any host may
           run the job. If the localization paramater is PARTITION, then  only
           hosts  in  the same resource partition as the host where the job is
           submitted can run the job. If the localization parameter is  LOCAL,
           then  the  job  can run only on the host where it is submitted. See
           the EXAMPLES for sample uses of the -c option.

       -cpn cpus_per_node
           This option is primarily meant for hosts that are "cluster"  sched-
           uled  by  LCRM and is advisory in the sense that LCRM does not con-
           strain the job from using more than the  specified  cpus.  If  this
           option  is  not used or otherwise implied, a value of 1 is assumed.
           It is ignored on hosts that are "memory" scheduled  and  is  depre-
           cated  in  favor  of the -ln option on hosts that are "node" sched-
           uled. (See "Scheduling Modes" under the NOTES section below  for  a
           way  to  discover  the  host's  scheduling  type.) The LCRM cluster
           scheduler will schedule jobs on a host of a LCRM cluster only inso-
           far  as  the  sum  of  values  specified by or implied for the -cpn
           option for those jobs do not exceed the cpu count of the  host.  If
           used  for  a  job  that  is to run on a host that is node scheduled
           (e.g., IBM SP), the specification must not conflict  with  any  cpn
           specification in the -ln option.

       -creds
           This  option  causes psub to attempt to preserve the user's current
           credentials so they can be made available to the batch job when  it
           runs.   This capability is dependent on the presence of an external
           credentials agent and as such has some caveats.  In order  to  hand
           the  credentials  off to the the credentials agent, the user's cre-
           dentials must be forwardable (klist can be  used  to  determine  if
           your credentials are forwardable and kinit can be used to make them
           forwardable if they are not).  If  the  credentials  agent  is  not
           present  on  the  submission  host, the credentials can not be pre-
           served and the submission will fail.  If the credentials  agent  is
           not  present  on  the execution host, the job will execute but will
           not be provided with run-time credentials.

       -d jobid
           Job which must complete before this  job  is  run.  Default  is  no
           dependency. A dependency on jobid 0 is the same as no dependency on
           any job.

       -dm (LCRM supports no system that supports job checkpointing.  Documen-
           tation for this option is moot for now.) Delete the job once it has
           received its maximum time. This option has no effect on hosts  that
           do not support job checkpointing.  In the absence of this option on
           a host that supports job  checkpointing,  a  job  is  held  in  the
           USED>MAX  state  once  it  has  received the maximum requested time
           which gives its owner a chance to increase the job's  maximum  time
           limit.  If  this option is used, a job that would otherwise go into
           the USED>MAX state will be  deleted  instead  thus  freeing  up  an
           active  job slot that would otherwise limit the number of jobs that
           could be executing on the host.

       -e file
           Directs the standard error file (stderr) produced by the job to the
           specified  destination.  This  string  may  not contain a ':', this
           character has a special meaning to some  of  the  underlying  batch
           systems.

       -eo Directs  all  output that would normally be sent to the stderr file
           to the stdout file for the job. This option may not be used in con-
           junction with the -e file or -re options.

       -exempt [exlist]
           Makes  the  job  exempt  from  specified  administrative limits. If
           exlist is not specified, the job is made exempt from all limits the
           owner of the job is permitted to exempt. Otherwise, the job is made
           exempt from the statuses listed in  addition  to  statuses  it  may
           already be exempted from.

           exlist  is  a comma separated list of administratively imposed lim-
           its. An administratively imposed limit is specified by entering the
           status  the  job  might  take  if  not  exempt  from the limit. For
           instance, specifying -exempt QTOTLIM exempts a job from the  admin-
           istrative  limit  on the total number of jobs that are permitted to
           run on a host.

           This option may not be used in conjunction with the -standby option
           and has no effect if used with the -expedite option.

           The  administratively  imposed  limits  from  which  a  job  may be
           exempted are:

           CPUS>MAX  job requires more nodes than is permitted
           CPU&TIME  job requires more node time than is permitted
           JRESLIM   maximum allowable jobs for the user or bank
                     are already running
           NRESLIM   maximum allowable nodes for the user or bank
                     are already in use
           NTRESLIM  maximum node-time limit for the user or bank
                     has been reached
           QTOTLIM   host is running as many jobs as is permitted
           QTOTLIMU  host is running as many jobs for user as is
                     permitted
           TOOLONG   job resuires more time than is permitted
           WMEML     host load within target parameters

       -expedite
           Causes the job to be placed into the expedited class. To expedite a
           job, you must have been granted the expedite permission to the bank
           the job will use for its resources. All expedited jobs  are  exempt
           from  administratively  imposed  limits  and are priority scheduled
           ahead of jobs in either the  normal  or  the  standby  class.  This
           option  may not be used in conjunction with the -standby option. If
           used, then use of the -exempt option adds no advantage for the job.

       -g [tasks][switch][@layout]
           This option is only useful on IBM SP hosts running LoadLeveler. The
           geometry option is used in  conjunction  with  the  -ln  option  to
           determine  how  tasks will be assigned to nodes. The tasks value is
           the total number of parallel tasks that will  be  started  for  the
           job. If a variable number of nodes is specified for the job via the
           -ln option, then the tasks value must not be specified. The  switch
           value  is  the communications switch type and may be either "ip" or
           "us". layout may be either "tpn<number>" where <number> is the num-
           ber  of  tasks per node or "dist" which specifies that tasks should
           be distributed as evenly as  possible  across  nodes.  The  default
           switch  is  "us"  and the default layout is "dist". If tasks is not
           specified, then if the layout is "dist", the  number  of  tasks  is
           equal  to  the number of nodes. But if the layout is "tpn<number>",
           then the number of tasks is the multiple of  the  number  of  nodes
           (specified  with the -ln option) and the <number> of tasks per node
           specified. If both tasks is specified and the layout  is  specified
           as  "tpn<number>",  then the number of tasks per node multiplied by
           the number of nodes requested must equal the number of tasks speci-
           fied.

       -i command
           A  shortcut  method to indicate a simple command directive to LCRM.
           This eliminates the need to have a script or use stdin when  a  job
           requires  only  a short, single line of input commands. When typing
           an immediate command, take care to isolate shell interpreted  char-
           acters by quoting the command.

       -ke Leaves  the stderr file on the execution host. Normally, the stderr
           file is returned to the originating host. This option  may  not  be
           used in conjunction with the -eo option.

       -ko Leaves  the stdout file on the execution host. Normally, the stdout
           file is returned to the originating host.

       -lc limit
           Set a per-process maximum core file size limit for the  batch  job.
           See the Limits subsection for more information.

       -ld limit
           Set  a  per-process  maximum  data-segment size limit for the batch
           job. See the Limits subsection for more information.

       -lF limit
           Sets the maximum per-job permanent-file space limit for  the  batch
           job. See the Limits subsection for more information.

       -lf limit
           Sets  a per-process maximum file size limit for the batch job. If a
           process attempts to write to a file that exceeds the process's file
           size  limit, the write fails with the [EFBIG] error. See the Limits
           subsection for more information.

       -lM jsize
           Advises the scheduler as to the expected maximum resident set  size
           to  be  attained by the job during its execution. The maximum resi-
           dent set size of jobs that have terminated as well as  the  maximum
           resident  set  size reported so far by currently executing jobs may
           be obtained through the pstat utility.  (See pstat.)  By  providing
           accurate  estimates,  the scheduler can more appropriately schedule
           jobs to avoid memory contention by jobs on a host.   The  scheduler
           will  not use memory advisories for users who consistently underes-
           timate their jobs' sizes.

       -ln nodedist
           This parameter is ignored on all but node scheduled (or multi-node)
           hosts.  (See "Scheduling Modes" under the NOTES section below for a
           way to discover the host's  scheduling  type.)  This  parameter  is
           required on RMS hosts. If not used on IBM SP hosts, the job will be
           executed as a single node serial job. If "-ln 1" is specified,  the
           job  will  be  executed  as a single node parallel job. This option
           specifies the distribution of nodes required for a job. The  format
           of the nodedist is:

           node_range[(node_distribution)] or
           (node_distribution) or
           node_distribution or
           node_range:tlist

           node_range := minnodes[-maxnodes]
           minnodes   := a number of nodes
           maxnodes   := a number of nodes >= minnodes
           node_distribution :=
                         nodes[:tlist][,nodes:tlist]* where
                         sum(nodes) <= minnodes
           tlist      := trait | (trait[,trait]*)
           trait      := [quantity]attribute
           attribute  := an abstract or predefined attribute

           Predefined  attributes  are memory and cpus per node. Cpus per node
           is expressed as <q>cpn where <q> is a nonnegative  integer.  Memory
           is expressed as <q><u> where <q> is a nonnegative integer and

           <u> := GB | Gb | gb | MB | Mb | mb | KB | Kb | kb
                  GB = Gb = gb = 1000MB
                  MB = Mb = mb = 1000KB
                  KB = Kb = kb = 1024 bytes

           Several  examples  of  the  -ln option can be found in the EXAMPLES
           section.  (Note: due to the terseness involved in the extended uses
           of  this option, it is highly advised to view the examples if using
           the advanced capabilities.)  the examples before extended use.)

       -lo limit
           Set a per-process open file count limit for the batch job.  See the
           Limits subsection for more information.

       -lr limit
           Set  a  per-process  maximum  resident set size limit for the batch
           job.  See the Limits subsection for more information.

       -ls limit
           Set a per-process maximum stack size limit for the batch job.   See
           the Limits subsection for more information.

       -lt limit
           Specifies  the  CPU  time limit for each process of a batch job. If
           the batch system supports this option, then the job will be  killed
           when any process in the job accumulates the specified amount of CPU
           time. If this option is not used or is not supported by the  native
           batch  system,  there  is  no process-specific limit imposed on CPU
           time usage.

           The default units when using the nnnn[dhms] format is minutes.  See
           the Limits subsection for more information on the implementation of
           batch job limits and for a description of the syntax of a  per-pro-
           cess CPU time limit.

       -mb Request that the underlying batch system send mail to the user when
           the job begins execution.

       -me Request that the underlying batch system send mail to the user when
           the job has ended execution.

       -mn Request  that  the  underlying batch system never send mail for any
           reason. This option does not prohibit the LCRM system from  sending
           mail, only the underlying batch system. This option may not be used
           in conjunction with any other mail option.

           (Note that using this option will stop the batch system from  send-
           ing  all  mail,  even  mail about critical errors. Please use judi-
           ciously.)

       -nc (LCRM supports no system that supports job checkpointing.  Documen-
           tation  for this option is moot for now.) Specifies that the job is
           not checkpointable.  This option has no effect on systems  that  do
           not  support  job  checkpointing. On systems that do support check-
           pointing, the job will be killed if LCRM determines  its  execution
           must  be stopped for any reason. On checkpointing systems, jobs are
           expected to be checkpointable in the absence of this option.

           When a batch system that supports checkpointing is shutdown  grace-
           fully, a checkpoint of all running, checkpointable jobs is taken at
           the time of shutdown. When a batch system  that  does  not  support
           checkpointing  is shutdown, then obviously no checkpoint of any job
           is taken. When  a  batch  system  that  supports  checkpointing  is
           restarted, it will attempt to resume processing of each job running
           at the time of the last shutdown at the point it was  when  it  was
           last  checkpointed.  This is called a batch job restart. If a batch
           job restart fails, the job is treated as if it had not been  check-
           pointed.

           When  a  batch  system first initializes, typically after rebooting
           (whether or not it supports checkpointing), all the jobs running at
           the  time  of  shutdown that cannot be restarted are rerun from the
           beginning. (Except see the -nr option.)

           Jobs that are not running at the time of a batch  system  interrup-
           tion or shutdown are not effected by the -nc option.

           See  the  "Batch  Job  Restart" subsection of the NOTES section for
           more discussion concerning the checkpoint/restart of batch jobs.

       -net protocol
           Specifies that the network protocol type that is  to  be  used  for
           this job.  (Ignored on all but IBM SP hosts.)

       -nettype type
           Specifies  that  the network communications type that is to be used
           for this job.  (Ignored on all but IBM SP hosts.)

       -nobulkxfer
           Specifies that block transfer is to be turned off, the default set-
           ting is on.  (Ignored on all but IBM SP hosts.)

       -np cpus_per_node
           Identical  to the -cpn option above. This option exists as a conve-
           nience for those used to using -np to specify cpus per node.

       -nr Specifies that the batch job is nonrerunnable. The job will not  be
           rerun  at  a  batch  system initialization if it is not rerunnable.
           Note that rerunning a job is not the same  thing  as  restarting  a
           job. To rerun a job means to begin its execution from the top again
           while restarting a job means to continue  its  execution  from  the
           point of the last checkpoint. A job may be restartable (but see the
           -nc option) but not rerunnable.

           In the absence of -nr, the batch system attempts to rerun a job  on
           batch system initialization.

           Jobs  that  are not running at the time of a batch system interrup-
           tion or shutdown are not effected by the -nr option.

       -o file
           Directs the standard output file (stdout) produced by the batch job
           to  the  specified  destination. This string may not contain a ':',
           this character has a special meaning  to  some  of  the  underlying
           batch systems.

       -p priority
           Set  the  scheduling priority for the selected job to the specified
           value. The value specified  with  this  option  must  be  a  number
           between  0.0  and  1.0  inclusive or the special value "float". The
           user must have been granted the  permission  to  use  this  option.
           After setting the priority of a job to a numerical value, LCRM will
           not reevaluate its priority. To permit LCRM to  calculate  a  job's
           priority again, use palter -p float.

       -r jobname
           Assigns  the  specified  jobname  to the job. If this option is not
           used, then the job's jobname is the basename of the script file  or
           the  owner's  user  name  if a script file is not specified. If the
           jobname contains a '%' sign, an error message is displayed and  the
           job  is  not submitted. If it starts with a digit, an uppercase 'R'
           will be inserted as the first character of the  name.  The  jobname
           will be truncated to 15 characters if necessary.

       -re Specifies  that the stderr output produced by the job is written to
           the final destination file as output is generated. This option  may
           not be used in conjunction with the -eo option.

           By default, all standard error output generated by a job is spooled
           to a temporary file in a protected directory which the batch system
           maintains  on the executing host. When the batch job completes exe-
           cution, this file is routed to its final destination.

       -ro Specifies that the standard output produced by the job  is  written
           to the destination file as output is generated.

           By  default, all standard output generated by a job is spooled to a
           temporary file in a protected  directory  which  the  batch  system
           maintains  on the executing host. When the batch job completes exe-
           cution, this file is routed to its final destination.

       -s shell-name
           Specifies the absolute path name of the shell used to interpret the
           batch job shell script. (Warning: there have been problems reported
           with using this option. You are encouraged to  allow  your  default
           login  shell  to interpret the script or to use the "#!" convention
           to specify the shell that is to interpret the script.)

           In the absence of this option, and if the "#!"  convention  is  not
           used  to specify the shell within the script, your login shell will
           be used to run the job.

       -standby
           The job will be submitted as a standby job. This job will be exempt
           from the following administratively imposed limits.

           JRESLIM   maximum allowable jobs for the user or bank
                     are already running
           NRESLIM   maximum allowable nodes for the user or bank
                     are already in use
           NTRESLIM  maximum node-time limit for the user or bank
                     has been reached

           However, the job will be removed (or otherwise stopped) when a non-
           standby job is eligible to run on the host and is in  need  of  the
           resources  being  used by the standby job. If the job is registered
           to receive a signal, it will be signaled prior  to  removal  within
           the  configured  grace  time  limit. The -standby option may not be
           used in conjunction  with  either  the  -expedite  or  the  -exempt
           options.

       -tM time
           Maximum  CPU  time per task. The job will be killed or checkpointed
           and not allowed to run any more after it accumulates the  specified
           (or  defaulted)  amount of CPU time per task. The default is a con-
           figuration parameter. (See lrmmgr or plim). The default units  when
           using  the nnnn[dhms] format is minutes.  The resulting interpreta-
           tion of this value must be greater than or equal to 1 minute.

           Note: On SMP hosts, there is only one task per  job.  Consequently,
           the  CPU  time per task is the sum of the CPU times of all the con-
           current processes or threads that make up the  job.  On  multi-node
           hosts, the CPU time represents the average time per task within the
           job. So, to specify a CPU time limit per task for a job running  on
           a SMP host that manages to keep 4 CPUs busy and needs 1 hour to run
           in a dedicated environment, the -tM option value should be  4h.  If
           the  same  job is to run on a multi-node host, the -tM limit should
           be 1h to limit the average time per task to one hour. On multi-node
           hosts,  it  is not possible to have a number of tasks less than the
           number of CPUs/node times the number of  nodes  on  which  the  job
           runs.

       -tW time
           Maximum  elapsed  run  time (aka, wall clock time). The job will be
           killed or checkpointed and not allowed to run any  more  after  its
           elapsed  run  time  exceeds  the  specified  or defaulted time. The
           default is a configuration parameter. The default units when  using
           the  nnnn[dhms]  format is minutes. The resulting interpretation of
           this value must be greater than or equal to 1 minute. The time dur-
           ing which a job is preempted or checkpointed does not count against
           its elapsed run time limit.

       -v  Enables psub's verbose mode.  Under certain circumstances, psub may
           encounter  non-fatal  errors  (such as using the -p option when you
           are not authorized to do so) which do not prevent a job from  being
           submitted  to LCRM, but may result in job attributes different than
           expected.  The default behavior is to silently  ignore  these  non-
           fatal errors.  If, however, the verbose mode is enabled, these non-
           fatal errors will generate  warning  messages  to  be  be  sent  to
           stderr.

       -x  Exports all environment variables (except ENVIRONMENT).

           LCRM  always  defines and initializes several environment variables
           before committing your job to the batch system.  These  environment
           variables  are  available  to  your job during execution. They, and
           their values, are:

           PCS_REQID    the LCRM job id (for backward
                        compatibility)
           PCS_TMPDIR   only if LCRM has created a temporary
                        directory for the job, in which case
                        the name of the directory.  This
                        directory will be exempt from any file
                        system purges during the course of the
                        job, but will still be subject to file
                        system quotas.  When the job terminates,
                        this directory, and all files within it,
                        will be removed.
           PSUB_HOME    your home directory on the submission host.
           PSUB_HOST    the submission host name.
           PSUB_JOBID   the LCRM job identifier.
           PSUB_LOGNAME your UNIX login name on the submission
                        host.
           PSUB_MAIL    your mail repository on the execution host.
           PSUB_PATH    the value of the environment variable PATH
                        at the time of job submission.
           PSUB_REQNAME the job's specified or implied jobname.
           PSUB_SHELL   your login shell on the submission host.
           PSUB_SUBDIR  your current directory on the submission
                        host.
           PSUB_TZ      the time zone in effect on the submission
                        host.
           PSUB_USER    your UNIX login name on the submission
                        host.
           PSUB_WORKDIR the same value as PCS_TMPDIR if that
                        environment variable is set. Otherwise,
                        your home directory on the execution host.
           HOME         your home directory on the executing host.
           SHELL        the path name of the shell that interprets
                        the job script.
           PATH         "/bin:/usr/bin"

       script [session-args]
           If provided, this file is expected to be a  command  script  to  be
           interpreted  by  a  standard  shell.  This option cannot be used in
           conjunction with the -i option.  If this option is not used and the
           -i  option  is  not  used,  then psub will read shell commands from
           standard in and construct a script from the input. If  standard  in
           is  the  terminal, then type ^D (Control-D) to signify that you are
           finished with input commands.

           Any characters following the script  name  (session-args)  will  be
           interpreted  by  psub  as arguments for the job. They are stored in
           the environment variable SESSARGS.  SESSARGS will be exported  with
           the  script.   SESSARGS  can  be  retrieved at run-time. It will be
           returned as a simple string.  It is up to the user to  process  the
           string.

           For example, if the following script is submitted,

           #!/bin/csh
           #
           echo $SESSARGS

           with the execute line
           psub <script name> 1 2 3 4 abc def ghi

           the output file will contain the line

           1 2 3 4 abc def ghi

           as a result of the echo $SESSARGS command in the script.

           This technique works with csh, sh, ksh and PERL shell scripts.

CONSTRAINTS
       Constraints are used to constrain jobs to run on particular hosts. LCRM
       matches specified constraints against features assigned  to  hosts  and
       only permits the job to run on the set of hosts with features that sat-
       isfy the specified constraints. (The constraint name matching algorithm
       is case insensitive.)

       The  structure  of  a constraint is "<token>[<operator><token>]*" where
       <token> is a simple constraint and <operator> is any one of the charac-
       ters ",", "&" or "|". Blanks are not permitted. The "," and "&" charac-
       ters have the same meaning which is "and" while the "|"  character  has
       the  meaning "or". Note that operator characters are significant to the
       shell. Consequently, complex constraints should be  surrounded  by  the
       "'"  character.  The "and" operator is evaluated before the "or" opera-
       tor. For example, the constraint 'a|b&c" means "any host with the  fea-
       ture  "a" or any host with both the feature "b" and the feature "c". If
       the constraint implied by '(a|b)&c' is intended, it must  be  expressed
       as 'a&c|b&c'.

       If  you  do  not  use a constraint, then LCRM looks for a "default con-
       straint" that may have been set for the resource partition of which the
       host where the job is submitted is a member. If a default constraint is
       found, it is used to determine the hosts that may run  the  job.  If  a
       default  constraint  is not found, then LCRM looks for the setting of a
       "localization" configuration parameter.  If the localization  parameter
       is GLOBAL, then any host may run the job. If the localization paramater
       is PARTITION, then only hosts in the same  resource  partition  as  the
       host  where  the  job is submitted can run the job. If the localization
       parameter is LOCAL, then the job can run only on the host where  it  is
       submitted.

       To  find  out what features are defined and assigned to a host, use the
       lrmmgr command and show  the  attributes  of  the  host.  The  features
       assigned  are  listed under "assigned features". The resource partition
       of which the host is a member  is  also  displayed.  If  you  show  the
       attributes  of the resource partition, it will display any default con-
       straint an administrator may have set.

JOB CLASSES (Delayed, Normal, Standby, Expedited)
       Jobs may be one of several classes: delayed, normal, standby  or  expe-
       dited.   Delayed  jobs will become normal, standby or expedited in FIFO
       order as the owner's jobs are scheduled. A normal job is  one  that  is
       not delayed, not standby and not expedited.

       Expedited  jobs  are  automatically  exempt  from  all administratively
       imposed limits. In addition,  they  are  scheduled  in  priority  order
       before all other classes of jobs.

       Normal jobs may be manually exempted from administratively imposed lim-
       its.  They are scheduled in priority order after expedited jobs.

       Standby jobs are automatically exempt from a subset of administratively
       imposed  limits.  Consequently, they may run when other classes of jobs
       cannot. Standby jobs are scheduled in priority order after  normal  and
       expedited jobs. In addition, they may be prematurely terminated to make
       room for a normal or expedited job.

       If the expedite class or the -exempt option is requested and  there  is
       an  error,  or you don't have permission, the job will be submitted and
       an error will be sent to stderr.

NOTES
   Batch Job Submission Events
       LCRM creates a spooled file that will contain all the commands  of  the
       batch job.

       LCRM commits the job to a batch system as a part of scheduling the job.

       The batch system sets the real and effective user IDs  of  the  session
       leader process to those of the submitter.

       The  batch  system sets the real and effective group IDs of the session
       leader process to the default (login) group IDs of the submitter.

       The batch system sets the user file creation mask to 077. (That is,  by
       default  files created from within the job will have only owner access.

       If the -s option is present, the batch  system  selects  the  specified
       shell  to  interpret  the script. Otherwise, the user's preferred login
       shell at the execution host is selected to interpret the script.

       All batch systems except loadleveler adds the environment string  ENVI-
       RONMENT=BATCH to the environment so that shell scripts (including .pro-
       file, Since loadLeveler does not add the  environment  string  ENVIRON-
       MENT=BATCH, system administrators are encouraged to set the environment
       variable within the lower level login shell scripts so that this  envi-
       ronment variable is consistent between platforms.

       The  batch system sets the current working directory to the home direc-
       tory on the executing host.

       As a final step, the chosen shell then executes  the  batch  job  shell
       script as though it is the login shell with the environment constructed
       as outlined in the preceding steps. If the Bourne or  Korn  shell  exe-
       cutes  the  script, the .profile file is read. If the C-shell executes,
       the .cshrc and .login scripts are read.

       Only the first 1Mb of the standard error and standard output files gen-
       erated  by batch jobs will be routed from one host to another. If these
       files are not routed between hosts, LCRM does  not  limit  their  size.
       (However, there may be file system quotas in effect.)

   Scheduling Modes
       LCRM has three scheduling modes:  memory, cluster, and node scheduling.
       In memory scheduling, LCRM schedules jobs on a single-node  host  based
       on the amount of memory available on that host and the memory limit the
       job requests.  In cluster scheduling, LCRM schedules jobs on a  single-
       node  host  based  on the number of cpus available on that node and the
       cpu count the job requests.  In node scheduling, LCRM schedules jobs to
       a  multi-node  host based on the number of nodes available on that host
       and the node count the job requests.

       Invoke phstat to see which scheduling mode is in effect  on  any  given
       host.  The  first column under the AVAILABILITY header will be either C
       indicating cluster scheduling, M indicating  memory  scheduling,  or  N
       indicating node scheduling.


   MPI Runs
       LCRM  permits MPI and multi-threaded jobs to run on SMP hosts. However,
       these jobs will typically get better performance if  they  are  run  on
       hosts  that are scheduled by the LCRM cluster scheduler. (See "Schedul-
       ing Modes" above for how to discover the host's  scheduling  type.)  To
       specify  that you want several CPUs reserved for your job when it is to
       run in a LCRM cluster scheduled host, use the -cpn <n> option.

   Limits
       LCRM allows the user to set resource limits within  which  a  job  must
       execute. Host, user and bank specific job resource limits can be set by
       system administrators to constrain the  scheduling  of  batch  jobs  to
       assure that limits are not exceeded.

       Finite  limits  are  expressed  as a time, a quota or an integer value.
       Finite time limits have the following format:

           nnnn[dhms]
             or
           hh:mm

       Example time limit-values are as follows:

           12d  - means 12 days
           13h  - means 13 hours
           200m - means 200 minutes
           5000s     - means 5000 seconds

              or

           5:00 - means 5 hours
           0:30 - means 30 minutes

       Finite quotas use the following syntax:

           .fraction [units]
       or
           integer [.fraction] [units]

       where the integer and fraction tokens represent strings of up to 8 dec-
       imal  digits,  denoting the obvious values. In both cases, the units of
       allocation may also  be  specified  as  one  of  the  case  insensitive
       strings:

           b          bytes
           kb         kilobytes = 1024b
           mb         megabytes = 1024kb
           gb         gigabytes = 1024mb

       In  the  absence  of  any  unit specification, the unit of kilobytes is
       assumed.

       Finite integer values are expressed as an optionally signed value  with
       no units.

       An  unlimited  value  can be specified by the string "unlimited" or any
       initial substring thereof, such as "unl".

       If a batch job specifies a limit that cannot be enforced by the  under-
       lying  operating  system,  batch controller or LCRM implementation, the
       limit is ignored, and the batch job operates  as  though  no  limit  is
       specified.

   Default Limits
       Per-process corefile size limit (-lc)
           No LCRM limit, limits may be imposed by the batch system.

       Per-process data-segment size limit (-ld)
           No LCRM limit, limits may be imposed by the batch system.

       Per-job permanent-file space limit (-lF)
           No LCRM limit, limits may be imposed by the batch system.

       Per-process permanent-file size limit (-lf)
           No LCRM limit, limits may be imposed by the batch system.

       Per-process CPU time limit (-lt)
           No LCRM limit, limits may be imposed by the batch system.

       Per-process stack size limit (-ls)
           No LCRM limit, limits may be imposed by the batch system.

       Per-process resident set size limit (-lr)
           No LCRM limit, limits may be imposed by the batch system.

       Per-process open file count limit (-lo)
           No LCRM limit, limits may be imposed by the batch system or operat-
           ing system.

   Job Submission Limits
       A user is allowed to submit maxjobsperuser jobs to the LCRM system. The
       maxjobsperuser  variable is a LCRM global variable. Jobs a user submits
       in excess of the maxjobsperuser limit are added to a FIFO  delayed  job
       class.  Jobs in the delayed job class are not considered for scheduling
       on any host. The delayed job class is simply  a  convenient  means  for
       accepting  a large number of job submissions from a user which are then
       submitted on behalf of the user as the user's submitted jobs are sched-
       uled.

       Only  delayedjoblimit jobs per user may be in the delayed job class. If
       this limit is exceeded, the psub command will fail. Use 'lrmmgr -r' and
       show  the  global parameters to determine values for the maxjobsperuser
       and delayedjoblimit values.

   Signal Handling
       If you use prm(1) to remove a running job  from  LCRM,  the  underlying
       batch  system is the factor that will determine which signal(s) will be
       sent to the job's processes to achieve its removal from the system.

EXAMPLES
       In the examples below, do not assume that any  of  the  specified  con-
       straints  actually  will match any features assigned to any host in the
       LCRM domain.  These features are dynamic and may  vary  over  time  and
       domain  to  domain.  Use lrmmgr to find out which features exist, which
       hosts exist and which features are assigned to which hosts.

       Following are typical uses of  command  line  options.  These  examples
       assume no embedded options.

       psub script
              Generally,  this  job  will  run  on  any  host  within the same
              resource partition as the resource partition of which  the  host
              where  the job is submitted is a member. For instance, say there
              are 4 hosts a, b, c, and d. and that a, b and c are  members  of
              one  resource  partition  while d is the only member of a second
              resource partition. Then if a job is submitted without using the
              -c option at host d, the job will run at host d. But if a job is
              submitted without using the -c option from host b, then  it  may
              run at any one of hosts a, b or c.

              However,  an  administrator  may  set a default constraint for a
              resource partition.  If so, then  the  default  constraint  will
              apply  if  you  do  not  use the -c option.  For instance, if an
              administrator sets the default constraint for the resource  par-
              tition  of  which  host  d  is  a member to -c 'a|b|c', then the
              absence of the -c option for a job submitted from d  would  have
              the same effect as the absence of the -c option for jobs submit-
              ted from hosts a, b or c.

       psub -c 'red|green' script
              Under the assumption that there are  two  hosts  named  red  and
              green,  this  job  is  constrained  to run on either of them. It
              would be an error to specify -c red,green since there should  be
              no host that had both features.

       psub -c pbatch script
              Under  the  assumption  that the feature name pbatch refers to a
              pool of nodes on multi-node hosts, there may be several hosts to
              which  this  feature  name  will have been assigned (i.e., has a
              pool of nodes with the name  pbatch).  This  job  will  be  con-
              strained to run in the pbatch pool on ANY of those hosts regard-
              less of where the job had been submitted.

       psub -c 'pdebug&red' script
              Under the assumption that there is a host named red and the fea-
              ture  name pdebug refers to a pool of nodes on multi-node hosts,
              there may be several hosts to which this feature name will  have
              been  assigned (i.e., has a pool of nodes with the name pdebug).
              This job will be constrained to run in the pdebug  pool  on  the
              host named red.

       psub -c '2000Mb,Alicense' script
              Under the assumption that Mb refers to megabytes of memory, this
              constraint indicates that the job is to be run on any host  that
              has at least 2000 megabytes of memory and can run processes that
              require the "Alicense".

       psub -ln 48 script
              This job requires 48 nodes and does  not  specify  any  required
              attributes for the nodes assigned.

       psub -ln 48-64 script
              This  job  requires at least 48 nodes, but no more than 64 nodes
              and does not specify  any  required  attributes  for  the  nodes
              assigned.

       psub -ln '48(24,24:16Gb)' script
              This  job  requires 48 nodes. Of those 24 must run on nodes with
              at least 16Gb of memory. There is no requirement  specified  for
              the other 24 nodes.

       psub -cpn 8 -ln 48 script
       psub -ln '48:8cpn' script
       psub -ln '48(48:8cpn)' script
           These  examples  are  identical.  The jobs require 48 nodes, all of
           which must have at least 8 cpus per node.

       psub -ln '8-48(2:(8cpn,4Gb),3:(4cpn,8Gb),1:gcard)' script
           This example is meant to show the flexibility of node distribution.
           The  job  requires  at  least  8 nodes, but no more than 48. Of all
           nodes assigned to the job, 2 nodes must have at least  8  cpus  per
           node  and  4  Gigabytes  of memory, another 3 nodes must at least 4
           cpus per node and 8 Gigabytes of memory and 1 node must have a fea-
           ture  called  "gcard".  Note that only 5 nodes are required to have
           specified attributes. However, all  nodes  assigned  must  have  at
           least one of the sets of attributes specified.

       psub -ln '8-48(1,2:(8cpn,4Gb),3:(4cpn,8Gb),1:gcard)' script
           This  example is the same as the above except that 1 node is speci-
           fied as not having any requirements. This means that no requirement
           is specified for any nodes assigned in excess of the 5 that do have
           specified requirements.

       psub -i "hostname"
           This job will show the host at which it ran in its standard  output
           file. Note that the 'script' option is not specified.

       psub -lc 1 script
           This  job will create core files no larger than 1 byte. This option
           is useful in preserving file system space on  multi-node  hosts  on
           which  many  hundreds  of core files might be created if a parallel
           process core dumps.

       psub -b batchbank script
           Assuming the caller had access to the bank named batchbank and that
           the  bank  permitted  batch jobs to run against its resources, this
           option is useful for users whose default bank does not permit batch
           jobs  to run. This option, if necessary, would usually be embedded.

       psub -A tomorrow script
           This job will start no earlier than midnight of the day after it is
           submitted.

       psub -cpn 4 -c smp1 script
           Assuming that smp1 is a LCRM cluster scheduled host and that it has
           4 cpus on it, this job specifies that it intends to use all 4  cpus
           of  the  host and will be scheduled alone on the host (without con-
           tention from other batch jobs).

       The following is an example of a typical batch job  file  submitted  to
       LCRM:
           #!/bin/sh
           # PSUB -r bigjob
           # PSUB -eo
           # PSUB -tM 20:00          # Job is to run for a maximum of 20 hours
           # PSUB -o list/bigjob.l
           cft77 -eL for.f
           segldr for.o
           date
           a.out
           date

       This batch job will have the name of "bigjob". Its  stderr  and  stdout
       files  will be merged and returned to file "list/bigjob.1", relative to
       the submitter's current directory. It requires  20  CPU  hours  of  run
       time. Note that the psub directives are specified in the same format as
       psub command options.

       You may choose to set up different batch and  interactive  environments
       in your .profile and .login files.

       The  following are examples of .profile and .login which enable command
       echo and command echo time stamping. See sh(1), ksh(1), and csh(1)  for
       additional information about these options.

           #
           #         sh .profile script
           #
           if test "$ENVIRONMENT" = "BATCH"
           then
           set -xS

           #
           #         csh .login script
           #
           if ("$ENVIRONMENT" = "BATCH") then
           set timestamp
           set echo
           endif

CAVEATS
       Because psub is a setuid root program, the user's job script must exist
       in a file system that allows read access by root.

RETURN VALUE
       An exit status of 0 indicates normal completion; an exit  status  of  1
       indicates a user error.

MESSAGES
       Explanatory  messages are given in the standard error file (stderr) for
       all irregularities.

AUTHOR
       Philip D Eckert, Lawrence Livermore National Laboratory,
            pdesr@llnl.gov

SEE ALSO
       palter(1), lrmmgr(1), pexp(1),  phold(1),  prel(1),  prm(1),  pstat(1),
       liblrm(1)



                                                                       PSUB(l)