Option(s) define multiple jobs in a co-scheduled heterogeneous job.
For more details about heterogeneous jobs see the document
.br
https://slurm.schedmd.com/heterogeneous_jobs.html

.SH "DESCRIPTION"
sbatch submits a batch script to Slurm.  The batch script may be given to
sbatch through a file name on the command line, or if no file name is specified,
sbatch will read in a script from standard input. The batch script may contain
options preceded with "#SBATCH" before any executable commands in the script.
sbatch will stop processing further #SBATCH directives once the first
non-comment non-whitespace line has been reached in the script.

sbatch exits immediately after the script is successfully transferred to the
Slurm controller and assigned a Slurm job ID.  The batch script is not
necessarily granted resources immediately, it may sit in the queue of pending
jobs for some time before its required resources become available.

By default both standard output and standard error are directed to a file of
the name "slurm\-%j.out", where the "%j" is replaced with the job allocation
number. The file will be generated on the first node of the job allocation.
Other than the batch script itself, Slurm does no movement of user files.

When the job allocation is finally granted for the batch script, Slurm
runs a single copy of the batch script on the first node in the set of
allocated nodes.

The following document describes the influence of various options on the
allocation of cpus to jobs and tasks.
.br
https://slurm.schedmd.com/cpu_management.html

.SH "RETURN VALUE"
sbatch will return 0 on success or error code on failure.

.SH "SCRIPT PATH RESOLUTION"

The batch script is resolved in the following order:
.br

1. If script starts with ".", then path is constructed as:
current working directory / script
.br
2. If script starts with a "/", then path is considered absolute.
.br
3. If script is in current working directory.
.br
4. If script can be resolved through PATH. See \fBpath_resolution\fR(7).
.br
.P
Current working directory is the calling process working directory unless the
number. For example, "\-\-array=0\-15:4" is equivalent to "\-\-array=0,4,8,12".
A maximum number of simultaneously running tasks from the job array may be
specified using a "%" separator.
For example "\-\-array=0\-15%4" will limit the number of simultaneously
running tasks from this job array to 4.
The minimum index value is 0.
the maximum value is one less than the configuration parameter MaxArraySize.
NOTE: currently, federated job arrays only run on the local cluster.

.TP
\fB\-A\fR, \fB\-\-account\fR=<\fIaccount\fR>
Charge resources used by this job to specified account.
The \fIaccount\fR is an arbitrary string. The account name may
be changed after job submission using the \fBscontrol\fR
command.

.TP
\fB\-\-acctg\-freq\fR
Define the job accounting and profiling sampling intervals.
This can be used to override the \fIJobAcctGatherFrequency\fR parameter in Slurm's
configuration file, \fIslurm.conf\fR.
The supported format is as follows:
.RS
.TP 12
\fB\-\-acctg\-freq=\fR\fI<datatype>\fR\fB=\fR\fI<interval>\fR
where \fI<datatype>\fR=\fI<interval>\fR specifies the task sampling
interval for the jobacct_gather plugin or a
sampling interval for a profiling type by the
acct_gather_profile plugin. Multiple,
comma-separated \fI<datatype>\fR=\fI<interval>\fR intervals
may be specified. Supported datatypes are as follows:
.RS
.TP
\fBtask=\fI<interval>\fR
where \fI<interval>\fR is the task sampling interval in seconds
for the jobacct_gather plugins and for task
profiling by the acct_gather_profile plugin.
NOTE: This frequency is used to monitor memory usage. If memory limits
are enforced the highest frequency a user can request is what is configured in
the slurm.conf file.  They can not turn it off (=0) either.
.TP
\fBenergy=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for energy profiling using the acct_gather_energy plugin
.TP
\fBnetwork=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for infiniband profiling using the acct_gather_interconnect
plugin.
.TP
\fBfilesystem=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
Smaller (non\-zero) values have a greater impact upon job performance,
but a value of 30 seconds is not likely to be noticeable for
applications having less than 10,000 tasks.
.RE

.TP
\fB\-B\fR \fB\-\-extra\-node\-info\fR=<\fIsockets\fR[:\fIcores\fR[:\fIthreads\fR]]>
Restrict node selection to nodes with at least the specified number of
sockets, cores per socket and/or threads per core.
NOTE: These options do not specify the resource allocation size.
Each value specified is considered a minimum.
An asterisk (*) can be used as a placeholder indicating that all available
resources of that type are to be utilized. Values can also be specified as
min-max. The individual levels can also be specified in separate options if
desired:
.nf
    \fB\-\-sockets\-per\-node\fR=<\fIsockets\fR>
    \fB\-\-cores\-per\-socket\fR=<\fIcores\fR>
    \fB\-\-threads\-per\-core\fR=<\fIthreads\fR>
.fi
If task/affinity plugin is enabled, then specifying an allocation in this
manner also results in subsequently launched tasks being bound to threads
if the \fB\-B\fR option specifies a thread count, otherwise an option of
\fIcores\fR if a core count is specified, otherwise an option of \fIsockets\fR.
If SelectType is configured to select/cons_res, it must have a parameter of
CR_Core, CR_Core_Memory, CR_Socket, or CR_Socket_Memory for this option
to be honored.
If not specified, the scontrol show job will display 'ReqS:C:T=*:*:*'. This
option applies to job allocations.

.TP
\fB\-\-batch\fR=<\fIlist\fR>
Nodes can have \fBfeatures\fR assigned to them by the Slurm administrator.
Users can specify which of these \fBfeatures\fR are required by their batch
script using this options.
For example a job's allocation may include both Intel Haswell and KNL nodes
with features "haswell" and "knl" respectively.
On such a configuration the batch script would normally benefit by executing
on a faster Haswell node.
This would be specified using the option "\-\-batch=haswell".
The specification can include AND and OR operators using the ampersand and
vertical bar separators. For example:
"\-\-batch=haswell|broadwell" or "\-\-batch=haswell|big_memory".
The \-\-batch argument must be a subset of the job's
\fB\-\-constraint\fR=<\fIlist\fR> argument (i.e. the job can not request only
KNL nodes, but require the script to execute on a Haswell node).
If the request can not be satisfied from the resources allocated to the job,
the batch script will execute on the first node of the job allocation.

.TP
\fB\-\-bb\fR=<\fIspec\fR>
Burst buffer specification. The form of the specification is system dependent.
tell the controller to defer the allocation of the job until the specified time.

Time may be of the form \fIHH:MM:SS\fR to run a job at
a specific time of day (seconds are optional).
(If that time is already past, the next day is assumed.)
You may also specify \fImidnight\fR, \fInoon\fR, \fIfika\fR (3 PM) or
\fIteatime\fR (4 PM) and you can have a time\-of\-day suffixed
with \fIAM\fR or \fIPM\fR for running in the morning or the evening.
You can also say what day the job will be run, by specifying
a date of the form \fIMMDDYY\fR or \fIMM/DD/YY\fR
\fIYYYY\-MM\-DD\fR. Combine date and time using the following
format \fIYYYY\-MM\-DD[THH:MM[:SS]]\fR. You can also
give times like \fInow + count time\-units\fR, where the time\-units
can be \fIseconds\fR (default), \fIminutes\fR, \fIhours\fR,
\fIdays\fR, or \fIweeks\fR and you can tell Slurm to run
the job today with the keyword \fItoday\fR and to run the
job tomorrow with the keyword \fItomorrow\fR.
The value may be changed after job submission using the
\fBscontrol\fR command.
For example:
.nf
   \-\-begin=16:00
   \-\-begin=now+1hour
   \-\-begin=now+60           (seconds by default)
   \-\-begin=2010\-01\-20T12:34:00
.fi

.RS
.PP
Notes on date/time specifications:
 \- Although the 'seconds' field of the HH:MM:SS time specification is
allowed by the code, note that the poll time of the Slurm scheduler
is not precise enough to guarantee dispatch of the job on the exact
second.  The job will be eligible to start on the next poll
following the specified time. The exact poll interval depends on the
Slurm scheduler (e.g., 60 seconds with the default sched/builtin).
 \- If no time (HH:MM:SS) is specified, the default is (00:00:00).
 \- If a date is specified without a year (e.g., MM/DD) then the current
year is assumed, unless the combination of MM/DD and HH:MM:SS has
already passed for that year, in which case the next year is used.
.RE

.TP
\fB\-\-cluster\-constraint\fR=[!]<\fIlist\fR>
Specifies features that a federated cluster must have to have a sibling job
submitted to it. Slurm will attempt to submit a sibling job to a cluster if it
has at least one of the specified features. If the "!" option is included, Slurm
will attempt to submit a sibling job to a cluster that has none of the specified
features.

.TP
\fB\-\-comment\fR=<\fIstring\fR>
.RS
.TP
\fBSingle Name\fR
Only nodes which have the specified feature will be used.
For example, \fB\-\-constraint="intel"\fR
.TP
\fBNode Count\fR
A request can specify the number of nodes needed with some feature
by appending an asterisk and count after the feature name.
For example "\fB\-\-nodes=16 \-\-constraint=graphics*4 ..."\fR
indicates that the job requires 16 nodes and that at least four of those
nodes must have the feature "graphics."
.TP
\fBAND\fR
If only nodes with all of specified features will be used.
The ampersand is used for an AND operator.
For example, \fB\-\-constraint="intel&gpu"\fR
.TP
\fBOR\fR
If only nodes with at least one of specified features will be used.
The vertical bar is used for an OR operator.
For example, \fB\-\-constraint="intel|amd"\fR
.TP
\fBMatching OR\fR
If only one of a set of possible options should be used for all allocated
nodes, then use the OR operator and enclose the options within square brackets.
For example: "\fB\-\-constraint=[rack1|rack2|rack3|rack4]"\fR might
be used to specify that all nodes must be allocated on a single rack of
the cluster, but any of those four racks can be used.
.TP
\fBMultiple Counts\fR
Specific counts of multiple resources may be specified by using the AND
operator and enclosing the options within square brackets.
For example: "\fB\-\-constraint=[rack1*2&rack2*4]"\fR might
be used to specify that two nodes must be allocated from nodes with the feature
of "rack1" and four nodes must be allocated from nodes with the feature
"rack2".

\fBNOTE:\fR This construct does not support multiple Intel KNL NUMA or MCDRAM
modes. For example, while "\fB\-\-constraint=[(knl&quad)*2&(knl&hemi)*4]"\fR is
not supported, "\fB\-\-constraint=[haswell*2&(knl&hemi)*4]"\fR is supported.
Specification of multiple KNL modes requires the use of a heterogeneous job.

.TP
\fBParenthesis\fR
Parenthesis can be used to group like node features together. For example
"\fB\-\-constraint=[(knl&snc4&flat)*4&haswell*1]"\fR might be used to specify
that four nodes with the features "knl", "snc4" and "flat" plus one node with
the feature "haswell" are required. All options within parenthesis should be
grouped with AND (e.g. "&") operands.
.RE


Request that job steps initiated by srun commands inside this sbatch script
be run at some requested frequency if possible, on the CPUs selected
for the step on the compute node(s).

\fBp1\fR can be  [#### | low | medium | high | highm1] which will set the
frequency scaling_speed to the corresponding value, and set the frequency
scaling_governor to UserSpace. See below for definition of the values.

\fBp1\fR can be [Conservative | OnDemand | Performance | PowerSave] which
will set the scaling_governor to the corresponding value. The governor has to be
in the list set by the slurm.conf option CpuFreqGovernors.

When \fBp2\fR is present, p1 will be the minimum scaling frequency and
p2 will be the maximum scaling frequency.

\fBp2\fR can be  [#### | medium | high | highm1] p2 must be greater than p1.

\fBp3\fR can be [Conservative | OnDemand | Performance | PowerSave | UserSpace]
which will set the governor to the corresponding value.

If \fBp3\fR is UserSpace, the frequency scaling_speed will be set by a power
or energy aware scheduling strategy to a value between p1 and p2 that lets the
job run within the site's power goal. The job may be delayed if p1 is higher
than a frequency that allows the job to run within the goal.

If the current frequency is < min, it will be set to min. Likewise,
if the current frequency is > max, it will be set to max.

Acceptable values at present include:
.RS
.TP 14
\fB####\fR
frequency in kilohertz
.TP
\fBLow\fR
the lowest available frequency
.TP
\fBHigh\fR
the highest available frequency
.TP
\fBHighM1\fR
(high minus one) will select the next highest available frequency
.TP
\fBMedium\fR
attempts to set a frequency in the middle of the available range
.TP
\fBConservative\fR
attempts to use the Conservative CPU governor
.TP
\fBOnDemand\fR
attempts to use the OnDemand CPU governor (the default value)
.nf
        SLURM_CPU_FREQ_REQ
.fi

This environment variable can also be used to supply the value for the
CPU frequency request if it is set when the 'srun' command is issued.
The \fB\-\-cpu\-freq\fR on the command line will override the
environment variable value.  The form on the environment variable is
the same as the command line.
See the \fBENVIRONMENT VARIABLES\fR
section for a description of the SLURM_CPU_FREQ_REQ variable.

\fBNOTE\fR: This parameter is treated as a request, not a requirement.
If the job step's node does not support setting the CPU frequency, or
the requested value is outside the bounds of the legal frequencies, an
error is logged, but the job step is allowed to continue.

\fBNOTE\fR: Setting the frequency for just the CPUs of the job step
implies that the tasks are confined to those CPUs.  If task
confinement (i.e., TaskPlugin=task/affinity or
TaskPlugin=task/cgroup with the "ConstrainCores" option) is not
configured, this parameter is ignored.

\fBNOTE\fR: When the step completes, the frequency and governor of each
selected CPU is reset to the previous values.

\fBNOTE\fR: When submitting jobs with  the \fB\-\-cpu\-freq\fR option
with linuxproc as the ProctrackType can cause jobs to run too quickly before
Accounting is able to poll for job information. As a result not all of
accounting information will be present.
.RE

.TP
\fB\-\-cpus\-per\-gpu\fR=<\fIncpus\fR>
Advise Slurm that ensuing job steps will require \fIncpus\fR processors per
allocated GPU.
Not compatible with the \fB\-\-cpus\-per\-task\fR option.

.TP
\fB\-c\fR, \fB\-\-cpus\-per\-task\fR=<\fIncpus\fR>
Advise the Slurm controller that ensuing job steps will require \fIncpus\fR
number of processors per task.  Without this option, the controller will
just try to allocate one processor per task.

For instance,
consider an application that has 4 tasks, each requiring 3 processors.  If our
cluster is comprised of quad\-processors nodes and we simply ask for
12 processors, the controller might give us only 3 nodes.  However, by using
the \-\-cpus\-per\-task=3 options, the controller knows that each task requires
3 processors on the same node, and the controller will grant an allocation
of 4 nodes, one for each of the 4 tasks.

.TP
\fB\-\-delay\-boot\fR=<\fIminutes\fR>
Do not reboot nodes in order to satisfied this job's feature specification if
the job has been eligible to run for less than this time period.
If the job has waited for less than the specified period, it will use only
nodes which already have the specified features.
The argument is in units of minutes.
A default value may be set by a system administrator using the \fBdelay_boot\fR
option of the \fBSchedulerParameters\fR configuration parameter in the
slurm.conf file, otherwise the default value is zero (no delay).

.TP
\fB\-d\fR, \fB\-\-dependency\fR=<\fIdependency_list\fR>
Defer the start of this job until the specified dependencies have been
satisfied completed.
<\fIdependency_list\fR> is of the form
<\fItype:job_id[:job_id][,type:job_id[:job_id]]\fR> or
<\fItype:job_id[:job_id][?type:job_id[:job_id]]\fR>.
All dependencies must be satisfied if the "," separator is used.
Any dependency may be satisfied if the "?" separator is used.
Only one separator may be used.
Many jobs can share the same dependency and these jobs may even belong to
different  users. The  value may be changed after job submission using the
scontrol command.
Dependencies on remote jobs are allowed in a federation.
Once a job dependency fails due to the termination state of a preceding job,
the dependent job will never be run, even if the preceding job is requeued and
has a different termination state in a subsequent execution.
.PD
.RS
.TP
\fBafter:job_id[[+time][:jobid[+time]...]]\fR
After the specified jobs start or are cancelled and 'time' in minutes from job
start or cancellation happens, this
job can begin execution. If no 'time' is given then then there is no delay after
start or cancellation.
.TP
\fBafterany:job_id[:jobid...]\fR
This job can begin execution after the specified jobs have terminated.
.TP
\fBafterburstbuffer:job_id[:jobid...]\fR
This job can begin execution after the specified jobs have terminated and
any associated burst buffer stage out operations have completed.
.TP
\fBaftercorr:job_id[:jobid...]\fR
A task of this job array can begin execution after the corresponding task ID
in the specified job has completed successfully (ran to completion with an
exit code of zero).
.TP
\fBafternotok:job_id[:jobid...]\fR
This job can begin execution after the specified jobs have terminated
in some failed state (non-zero exit code, node failure, timed out, etc).
sharing the same job name and user have terminated.
In other words, only one job by that name and owned by that user can be running
or suspended at any point in time.
In a federation, a singleton dependency must be fulfilled on all clusters
unless DependencyParameters=disable_remote_singleton is used in slurm.conf.
.RE

.TP
\fB\-D\fR, \fB\-\-chdir\fR=<\fIdirectory\fR>
Set the working directory of the batch script to \fIdirectory\fR before
it is executed. The path can be specified as full path or relative path
to the directory where the command is executed.

.TP
\fB\-e\fR, \fB\-\-error\fR=<\fIfilename pattern\fR>
Instruct Slurm to connect the batch script's standard error directly to the
file name specified in the "\fIfilename pattern\fR".
By default both standard output and standard error are directed to the same file.
For job arrays, the default file name is "slurm-%A_%a.out", "%A" is replaced
by the job ID and "%a" with the array index.
For other jobs, the default file name is "slurm-%j.out", where the "%j" is
replaced by the job ID.
See the \fBfilename pattern\fR section below for filename specification options.

.TP
\fB\-\-exclusive[=user|mcs]\fR
The job allocation can not share nodes with other running jobs (or just other
users with the "=user" option or with the "=mcs" option).
The default shared/exclusive behavior depends on system configuration and the
partition's \fBOverSubscribe\fR option takes precedence over the job's option.

.TP
\fB\-\-export\fR=<\fI[ALL,]environment variables\fR|\fIALL\fR|\fINONE\fR>
Identify which environment variables from the submission environment are
propagated to the launched application.
.RS
.TP 10
\fB\-\-export\fR=\fIALL\fR
.br
Default mode if \fB\-\-export\fR is not specified. All of the users environment
will be loaded (either from callers environment or clean environment if
\fI\-\-get\-user\-env\fR is specified).
.TP
\fB\-\-export\fR=\fINONE\fR
.br
None of the user environment will be defined. User must use absolute path to
the binary to be executed that will define the environment. User can not specify
explicit environment variables with \fINONE\fR. \fI\-\-get\-user\-env\fR will
be ignored.
.br
This option is particularly important for jobs that are submitted on one cluster
and execute on a different cluster (e.g. with different paths).
.TP 5
Example: \fB\-\-export\fR=EDITOR,ARG1=test
.br
In this example, the propagated environment will only contain the
variable \fIEDITOR\fR from the user's environment, \fISLURM_*\fR environment
variables, and \fIARG1\fR=test.
.TP
Example: \fB\-\-export\fR=ALL,EDITOR=/bin/emacs
.br
There are two possible outcomes for this example. If the caller has the
\fIEDITOR\fR environment variable defined, then the job's environment will
inherit the variable from the caller's environment.  If the caller doesn't
have an environment variable defined for \fIEDITOR\fR, then the job's
environment will use the value given by \fB--export\fR.
.RE
.RE

.TP
\fB\-\-export\-file\fR=<\fIfilename\fR | \fIfd\fR>
If a number between 3 and OPEN_MAX is specified as the argument to
this option, a readable file descriptor will be assumed (STDIN and
STDOUT are not supported as valid arguments).  Otherwise a filename is
assumed.  Export environment variables defined in <\fIfilename\fR> or
read from <\fIfd\fR> to the job's execution environment. The
content is one or more environment variable definitions of the form
NAME=value, each separated by a null character.  This allows the use
of special characters in environment definitions.

.TP
\fB\-F\fR, \fB\-\-nodefile\fR=<\fInode file\fR>
Much like \-\-nodelist, but the list is contained in a file of name
\fInode file\fR.  The node names of the list may also span multiple lines
in the file.    Duplicate node names in the file will be ignored.
The order of the node names in the list is not important; the node names
will be sorted by Slurm.

.TP
\fB\-\-get\-user\-env\fR[=\fItimeout\fR][\fImode\fR]
This option will tell sbatch to retrieve the
login environment variables for the user specified in the \fB\-\-uid\fR option.
The environment variables are retrieved by running something of this sort
"su \- <username> \-c /usr/bin/env" and parsing the output.
Be aware that any environment variables already set in sbatch's environment
will take precedence over any environment variables in the user's
login environment. Clear any environment variables before calling sbatch
that you do not want propagated to the spawned program.
The optional \fItimeout\fR value is in seconds. Default value is 8 seconds.
The optional \fImode\fR value control the "su" options.
With a \fImode\fR value of "S", "su" is executed without the "\-" option.
With a \fImode\fR value of "L", "su" is executed with the "\-" option,
replicating the login environment.
If \fImode\fR not specified, the mode established at Slurm build time
For example "\-\-gpus=volta:3".
Multiple options can be requested in a comma separated list, for example:
"\-\-gpus=volta:3,kepler:1".
See also the \fB\-\-gpus\-per\-node\fR, \fB\-\-gpus\-per\-socket\fR and
\fB\-\-gpus\-per\-task\fR options.

.TP
\fB\-\-gpu\-bind\fR=<\fItype\fR>
Bind tasks to specific GPUs.
By default every spawned task can access every GPU allocated to the job.

Supported \fItype\fR options:
.RS
.TP 10
\fBclosest\fR
Bind each task to the GPU(s) which are closest.
In a NUMA environment, each task may be bound to more than one GPU (i.e.
all GPUs in that NUMA environment).
.TP
\fBmap_gpu:<list>\fR
Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
 #!/bin/sh
.br
> srun hostname |sort
.br
> EOF
.br
sbatch: Submitted batch job 65541
.br

.br
$ cat slurm\-65541.out
.br
host1
.br
host2
.br
host3
.br
host4

.LP
To create a heterogeneous job with 3 components, each allocating a unique set of nodes:
.IP
sbatch -w node[2-3] : -w node4 : -w node[5-7] work.bash
.br
Submitted batch job 34987

.SH "COPYING"
Copyright (C) 2006\-2007 The Regents of the University of California.
Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
details.

.SH "SEE ALSO"
.LP
\fBsinfo\fR(1), \fBsattach\fR(1), \fBsalloc\fR(1), \fBsqueue\fR(1), \fBscancel\fR(1), \fBscontrol\fR(1),
\fBslurm.conf\fR(5), \fBsched_setaffinity\fR (2), \fBnuma\fR (3)

Man(1) output converted with man2html