configuration information, the nodes to be managed, information about
how those nodes are grouped into partitions, and various scheduling
parameters associated with those partitions. This file should be
consistent across all nodes in the cluster.
.LP
The file location can be modified at system build time using the
DEFAULT_SLURM_CONF parameter or at execution time by setting the SLURM_CONF
environment variable. The Slurm daemons also allow you to override
both the built\-in and environment\-provided location using the "\-f"
option on the command line.
.LP
The contents of the file are case insensitive except for the names of nodes
and partitions. Any text following a "#" in the configuration file is treated
as a comment through the end of that line.
Changes to the configuration file take effect upon restart of
Slurm daemons, daemon receipt of the SIGHUP signal, or execution
of the command "scontrol reconfigure" unless otherwise noted.
.LP
If a line begins with the word "Include" followed by whitespace
and then a file name, that file will be included inline with the current
configuration file. For large or complex systems, multiple configuration files
may prove easier to manage and enable reuse of some files (See INCLUDE
MODIFIERS for more details).
.LP
Note on file permissions:
.LP
The \fIslurm.conf\fR file must be readable by all users of Slurm, since it
is used by many of the Slurm commands.  Other files that are defined
in the \fIslurm.conf\fR file, such as log files and job accounting files,
may need to be created/owned by the user "SlurmUser" to be successfully
accessed.  Use the "chown" and "chmod" commands to set the ownership
and permissions appropriately.
See the section \fBFILE AND DIRECTORY PERMISSIONS\fR for information
about the various files and directories used by Slurm.

.SH "PARAMETERS"
.LP
The overall configuration parameters available include:

.TP
\fBAccountingStorageBackupHost\fR
The name of the backup machine hosting the accounting storage database.
If used with the accounting_storage/slurmdbd plugin, this is where the backup
slurmdbd would be running.
Only used with systems using SlurmDBD, ignored otherwise.

.TP
\fBAccountingStorageEnforce\fR
This controls what level of association\-based enforcement to impose
on job submissions.  Valid options are any combination of
\fIassociations\fR, \fIlimits\fR, \fInojobs\fR, \fInosteps\fR, \fIqos\fR, \fIsafe\fR, and \fIwckeys\fR, or
\fIall\fR for all things (expect nojobs and nosteps, they must be requested as well).
like wise if nosteps is set Slurm will not account for any steps ran limits
will still be enforced.

If safe is enforced, a job will only be launched against an association or qos
that has a GrpTRESMins limit set if the job will be able to run to completion.
Without this option set, jobs will be launched as long as their usage
hasn't reached the cpu-minutes limit which can lead to jobs being
launched but then killed when the limit is reached.

With qos and/or wckeys enforced jobs will not be scheduled unless a valid qos
and/or workload characterization key is specified.

When \fBAccountingStorageEnforce\fR is changed, a restart of the slurmctld
daemon is required (not just a "scontrol reconfig").

.TP
\fBAccountingStorageExternalHost\fR
A comma separated list of external slurmdbds (<host/ip>[:port][,...]) to
register with. If no port is given, the \fBAccountingStoragePort\fR will be
used.

This allows clusters registered with the external slurmdbd to communicate with
each other using the \fI--cluster/-M\fR client command options.

The cluster will add itself to the external slurmdbd if it doesn't exist. If a
non-external cluster already exists on the external slurmdbd, the slurmctld
will ignore registering to the external slurmdbd.

.TP
\fBAccountingStorageHost\fR
The name of the machine hosting the accounting storage database.
Only used with systems using SlurmDBD, ignored otherwise.
Also see \fBDefaultStorageHost\fR.

.TP
\fBAccountingStorageLoc\fR
The fully qualified file name where accounting records are written
when the \fBAccountingStorageType\fR is "accounting_storage/filetxt".
Also see \fBDefaultStorageLoc\fR.

.TP
\fBAccountingStoragePass\fR
The password used to gain access to the database to store the
accounting data.  Only used for database type storage plugins, ignored
otherwise.  In the case of Slurm DBD (Database Daemon) with MUNGE
authentication this can be configured to use a MUNGE daemon
specifically configured to provide authentication between clusters
while the default MUNGE daemon provides authentication within a
cluster.  In that case, \fBAccountingStoragePass\fR should specify the
named port to be used for communications with the alternate MUNGE
daemon (e.g.  "/var/run/munge/global.socket.2"). The default value is
NULL.  Also see \fBDefaultStoragePass\fR.
These are the resources requested by the sbatch/srun job when it
is submitted. Currently this consists of any GRES, BB (burst buffer) or
license along with CPU, Memory, Node, Energy, FS/[Disk|Lustre], IC/OFED, Pages,
and VMem. By default Billing, CPU, Energy, Memory, Node, FS/Disk, Pages and VMem
are tracked. These default TRES cannot be disabled, but only appended to.
AccountingStorageTRES=gres/craynetwork,license/iop1
will track billing, cpu, energy, memory, nodes, fs/disk, pages and vmem along
with a gres called craynetwork as well as a license called iop1. Whenever these
resources are used on the cluster they are recorded. The TRES are automatically
set up in the database on the start of the slurmctld.

If multiple GRES of different types are tracked (e.g. GPUs of different types),
then job requests with matching type specifications will be recorded.
Given a configuration of
"AccountingStorageTRES=gres/gpu,gres/gpu:tesla,gres/gpu:volta"
Then "gres/gpu:tesla" and "gres/gpu:volta" will track only jobs that explicitly
request those two GPU types, while "gres/gpu" will track allocated GPUs of any
type ("tesla", "volta" or any other GPU type).

Given a configuration of
"AccountingStorageTRES=gres/gpu:tesla,gres/gpu:volta"
Then "gres/gpu:tesla" and "gres/gpu:volta" will track jobs that explicitly
request those GPU types.
If a job requests GPUs, but does not explicitly specify the GPU type, then
its resource allocation will be accounted for as either "gres/gpu:tesla" or
"gres/gpu:volta", although the accounting may not match the actual GPU type
allocated to the job and the GPUs allocated to the job could be heterogeneous.
In an environment containing various GPU types, use of a job_submit plugin
may be desired in order to force jobs to explicitly specify some GPU type.

.TP
\fBAccountingStorageType\fR
The accounting storage mechanism type.  Acceptable values at
present include "accounting_storage/filetxt", "accounting_storage/none"
and "accounting_storage/slurmdbd".  The
"accounting_storage/filetxt" value indicates that accounting records
will be written to the file specified by the
\fBAccountingStorageLoc\fR parameter.
The "accounting_storage/slurmdbd" value indicates that accounting records
will be written to the Slurm DBD, which manages an underlying MySQL
database. See "man slurmdbd" for more information.  The
default value is "accounting_storage/none" and indicates that account
records are not maintained.
Note: The filetxt plugin records only a limited subset of accounting
information and will prevent some sacct options from proper operation.
Also see \fBDefaultStorageType\fR.

.TP
\fBAccountingStorageUser\fR
The user account for accessing the accounting storage database.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageUser\fR.
acct_gather_energy/rapl plugin, set a value less
than 300 because the counters may overflow beyond this rate.
The default value is zero. This value disables accounting sampling
for nodes. Note: The accounting sampling interval for jobs is
determined by the value of \fBJobAcctGatherFrequency\fR.

.TP
\fBAcctGatherEnergyType\fR
Identifies the plugin to be used for energy consumption accounting.
The jobacct_gather plugin and slurmd daemon call this plugin to collect
energy consumption data for jobs and nodes. The collection of energy
consumption data takes place on the node level, hence only in case of exclusive
job allocation the energy consumption measurements will reflect the job's
real consumption. In case of node sharing between jobs the reported consumed
energy per job (through sstat or sacct) will not reflect the real energy
consumed by the jobs.

Configurable values at present are:
.RS
.TP 20
\fBacct_gather_energy/none\fR
No energy consumption data is collected.
.TP
\fBacct_gather_energy/ipmi\fR
Energy consumption data is collected from the Baseboard Management Controller
(BMC) using the Intelligent Platform Management Interface (IPMI).
.TP
\fBacct_gather_energy/rapl\fR
Energy consumption data is collected from hardware sensors using the Running
Average Power Limit (RAPL) mechanism. Note that enabling RAPL may require the
execution of the command "sudo modprobe msr".
.RE

.TP
\fBAcctGatherInfinibandType\fR
Identifies the plugin to be used for infiniband network traffic accounting.
The jobacct_gather plugin and slurmd daemon call this plugin to collect
network traffic data for jobs and nodes.
The collection of network traffic data takes place on the node level,
hence only in case of exclusive job allocation the collected values will
reflect the job's real traffic. In case of node sharing between jobs the reported
network traffic per job (through sstat or sacct) will not reflect the real
network traffic by the jobs.

Configurable values at present are:
.RS
.TP 20
\fBacct_gather_interconnect/none\fR
No infiniband network data are collected.
.TP
\fBacct_gather_interconnect/ofed\fR
Infiniband network traffic data are collected from the hardware monitoring
filesystem traffic by the jobs.


Configurable values at present are:
.RS
.TP 20
\fBacct_gather_filesystem/none\fR
No filesystem data are collected.
.TP
\fBacct_gather_filesystem/lustre\fR
Lustre filesystem traffic data are collected from the counters found in
/proc/fs/lustre/.
In order to account for per job lustre traffic, add the "fs/lustre" TRES to
\fIAccountingStorageTRES\fR.
.RE

.TP
\fBAcctGatherProfileType\fR
Identifies the plugin to be used for detailed job profiling.
The jobacct_gather plugin and slurmd daemon call this plugin to collect
detailed data such as I/O counts, memory usage, or energy consumption for jobs
and nodes. There are interfaces in this plugin to collect data as step start
and completion, task start and completion, and at the account gather
frequency. The data collected at the node level is related to jobs only in
case of exclusive job allocation.

Configurable values at present are:
.RS
.TP 20
\fBacct_gather_profile/none\fR
No profile data is collected.
.TP
\fBacct_gather_profile/hdf5\fR
This enables the HDF5 plugin. The directory where the profile files
are stored and which values are collected are configured in the
acct_gather.conf file.
.TP
\fBacct_gather_profile/influxdb\fR
This enables the influxdb plugin. The influxdb instance host, port, database,
retention policy and which values are collected are configured in the
acct_gather.conf file.
.RE

.TP
\fBAllowSpecResourcesUsage\fR
If set to "YES", Slurm allows individual jobs to override node's configured
CoreSpecCount value. For a job to take advantage of this feature,
a command line option of \-\-core\-spec must be specified.  The default
value for this option is "YES" for Cray systems and "NO" for other system types.

.TP
\fBAuthAltTypes\fR
Default job step credential lifetime, in seconds (e.g. "cred_expire=1200").
It must be sufficiently long enough to load user environment, run prolog,
deal with the slurmd getting paged out of memory, etc.
This also controls how long a requeued job must wait before starting again.
The default value is 120 seconds.
.TP
\fBsocket\fR
Path name to a MUNGE daemon socket to use
(e.g. "socket=/var/run/munge/munge.socket.2").
The default value is "/var/run/munge/munge.socket.2".
Used by \fIauth/munge\fR and \fIcred/munge\fR.
.TP
\fBttl\fR
Credential lifetime, in seconds (e.g. "ttl=300").
The default value is dependent upon the MUNGE installation, but is typically
300 seconds.
.RE

.TP
\fBAuthType\fR
The authentication method for communications between Slurm
components.
Acceptable values at present include "auth/munge" and "auth/none".
The default value is "auth/munge".
"auth/none" includes the UID in each communication, but it is not verified.
This may be fine for testing purposes, but
\fBdo not use "auth/none" if you desire any security\fR.
"auth/munge" indicates that MUNGE is to be used.
(See "https://dun.github.io/munge/" for more information).
All Slurm daemons and commands must be terminated prior to changing
the value of \fBAuthType\fR and later restarted.

.TP
\fBBackupAddr\fR
Defunct option, see \fBSlurmctldHost\fR.

.TP
\fBBackupController\fR
Defunct option, see \fBSlurmctldHost\fR.

The backup controller recovers state information from the
\fBStateSaveLocation\fR directory, which must be readable and writable from both
the primary and backup controllers.
While not essential, it is recommended that you specify a backup controller.
See  the \fBRELOCATING CONTROLLERS\fR section if you change this.

.TP
\fBBatchStartTimeout\fR
The maximum time (in seconds) that a batch job is permitted for
launching before being considered missing and releasing the
allocation. The default value is 10 (seconds). Larger values may be
required if more time is required to execute the \fBProlog\fR, load
.RS
.TP
\fBburst_buffer/datawrap\fR
Use Cray DataWarp API to provide burst buffer functionality.
.TP
\fBburst_buffer/none\fR
.RE

.TP
\fBCliFilterPlugins\fR
A comma delimited list of command line interface option filter/modification
plugins. The specified plugins will be executed in the order listed.
These are intended to be site\-specific plugins which can be used to set
default job parameters and/or logging events.
No cli_filter plugins are used by default.

.TP
\fBClusterName\fR
The name by which this Slurm managed cluster is known in the
accounting database.  This is needed distinguish accounting records
when multiple clusters report to the same database. Because of limitations
in some databases, any upper case letters in the name will be silently mapped
to lower case. In order to avoid confusion, it is recommended that the name
be lower case.

.TP
\fBCommunicationParameters\fR
Comma separated options identifying communication options.
.RS
.TP 15
\fBCheckGhalQuiesce\fR
Used specifically on a Cray using an Aries Ghal interconnect.  This will check
to see if the system is quiescing when sending a message, and if so, we wait
until it is done before sending.
.TP
\fBNoAddrCache\fR By default, Slurm will cache a node's network address after
successfully establishing the node's network address. This option disables the
cache and Slurm will look up the node's network address each time a connection
is made. This is useful, for example, in a cloud environment where the node
addresses come and go out of DNS.
.TP
\fBNoCtldInAddrAny\fR
Used to directly bind to the address of what the node resolves to running
the slurmctld instead of binding messages to any address on the node,
which is the default.
.TP
\fBNoInAddrAny\fR
Used to directly bind to the address of what the node resolves to instead
of binding messages to any address on the node which is the default.
This option is for all daemons/clients except for the slurmctld.
.RE

The value may not exceed 65533.

.TP
\fBControlAddr\fR
Defunct option, see \fBSlurmctldHost\fR.

.TP
\fBControlMachine\fR
Defunct option, see \fBSlurmctldHost\fR.

.TP
\fBCoreSpecPlugin\fR
Identifies the plugins to be used for enforcement of core specialization.
The slurmd daemon must be restarted for a change in CoreSpecPlugin
to take effect.
Acceptable values at present include:
.RS
.TP 20
\fBcore_spec/cray_aries\fR
used only for Cray systems
.TP
\fBcore_spec/none\fR
used for all other system types
.RE

.TP
\fBCpuFreqDef\fR
Default CPU frequency value or frequency governor to use when running a
job step if it has not been explicitly set with the \-\-cpu\-freq option.
Acceptable values at present include a numeric value (frequency in kilohertz)
or one of the following governors:
.RS
.TP 14
\fBConservative\fR
attempts to use the Conservative CPU governor
.TP
\fBOnDemand\fR
attempts to use the OnDemand CPU governor
.TP
\fBPerformance\fR
attempts to use the Performance CPU governor
.TP
\fBPowerSave\fR
attempts to use the PowerSave CPU governor
.RE
There is no default value. If unset, no attempt to set the governor is
made if the \-\-cpu\-freq option has not been set.

.TP
\fBCpuFreqGovernors\fR
List of CPU frequency governors allowed to be set with the salloc, sbatch, or
srun option  \-\-cpu\-freq.
.TP
\fBUserSpace\fR
attempts to use the UserSpace CPU governor (a default value)
.RE
The default is OnDemand, Performance and UserSpace.
.TP
\fBCredType\fR
The cryptographic signature tool to be used in the creation of
job step credentials.
The slurmctld daemon must be restarted for a change in \fBCredType\fR
to take effect.
Acceptable values at present include "cred/munge".
The default value is "cred/munge" and is the recommended.

.TP
\fBDebugFlags\fR
Defines specific subsystems which should provide more detailed event logging.
Multiple subsystems can be specified with comma separators.
Most DebugFlags will result in verbose logging for the identified subsystems
and could impact performance.
Valid subsystems available today (with more to come) include:
.RS
.TP 17
\fBAccrue\fR
Accrue counters accounting details
.TP
\fBAgent\fR
RPC agents (outgoing RPCs from Slurm daemons)
.TP
\fBBackfill\fR
Backfill scheduler details
.TP
\fBBackfillMap\fR
Backfill scheduler to log a very verbose map of reserved resources through
time. Combine with \fBBackfill\fR for a verbose and complete view of the
backfill scheduler's work.
.TP
\fBBurstBuffer\fR
Burst Buffer plugin
.TP
\fBCPU_Bind\fR
CPU binding details for jobs and steps
.TP
\fBCpuFrequency\fR
Cpu frequency details for jobs and steps using the \-\-cpu\-freq option.
.TP
\fBData\fR
Generic data structure details.
.TP
\fBDependency\fR
Job dependency debug info
.TP
.TP
\fBGres\fR
Generic resource details
.TP
\fBHetjob\fR
Heterogeneous job details
.TP
\fBGang\fR
Gang scheduling details
.TP
\fBJobContainer\fR
Job container plugin details
.TP
\fBLicense\fR
License management details
.TP
\fBNetwork\fR
Network details
.TP
\fBNodeFeatures\fR
Node Features plugin debug info
.TP
\fBNO_CONF_HASH\fR
Do not log when the slurm.conf files differs between Slurm daemons
.TP
\fBPower\fR
Power management plugin
.TP
\fBPowerSave\fR
Power save (suspend/resume programs) details
.TP
\fBPriority\fR
Job prioritization
.TP
\fBProfile\fR
AcctGatherProfile plugins details
.TP
\fBProtocol\fR
Communication protocol details
.TP
\fBReservation\fR
Advanced reservations
.TP
\fBRoute\fR
Message forwarding and message aggregation debug info
.TP
\fBSelectType\fR
Resource selection plugin
.TP
\fBSteps\fR
Slurmctld resource allocation for job steps
.TP
Slurmctld triggers
.TP
\fBWorkQueue\fR
Work Queue details
.RE

.TP
\fBDefCpuPerGPU\fR
Default count of CPUs allocated per allocated GPU.

.TP
\fBDefMemPerCPU\fR
Default real memory size available per allocated CPU in megabytes.
Used to avoid over\-subscribing memory and causing paging.
\fBDefMemPerCPU\fR would generally be used if individual processors
are allocated to jobs (\fBSelectType=select/cons_res\fR or
\fBSelectType=select/cons_tres\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerGPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
mutually exclusive.

.TP
\fBDefMemPerGPU\fR
Default real memory size available per allocated GPU in megabytes.
The default value is 0 (unlimited).
Also see \fBDefMemPerCPU\fR and \fBDefMemPerNode\fR.
\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
mutually exclusive.

.TP
\fBDefMemPerNode\fR
Default real memory size available per allocated node in megabytes.
Used to avoid over\-subscribing memory and causing paging.
\fBDefMemPerNode\fR would generally be used if whole nodes
are allocated to jobs (\fBSelectType=select/linear\fR) and
resources are over\-subscribed (\fBOverSubscribe=yes\fR or
\fBOverSubscribe=force\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerCPU\fR.
\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
mutually exclusive.

.TP
\fBDefaultStorageHost\fR
The default name of the machine hosting the accounting storage and
job completion databases.
Only used for database type storage plugins and when the
\fBAccountingStorageHost\fR and \fBJobCompHost\fR have not been
defined.

.TP
\fBDefaultStoragePort\fR
The listening port of the accounting storage and/or job completion
database server.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStoragePort\fR and \fBJobCompPort\fR.

.TP
\fBDefaultStorageType\fR
The accounting and job completion storage mechanism type.  Acceptable
values at present include "filetxt", "mysql" and "none".
The value "filetxt" indicates that records will be written to a file.
The value "mysql" indicates that accounting records will be written to a MySQL
or MariaDB database.
The default value is "none", which means that records are not maintained.
Also see \fBAccountingStorageType\fR and \fBJobCompType\fR.

.TP
\fBDefaultStorageUser\fR
The user account for accessing the accounting storage and/or job
completion database.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStorageUser\fR and \fBJobCompUser\fR.

.TP
\fBDependencyParameters\fR
Multiple options may be comma-separated.

.RS
.TP
\fBdisable_remote_singleton\fR
By default, when a federated job has a singleton dependeny, each cluster in the
federation must clear the singleton dependency before the job's singleton
dependency is considered satisfied. Enabling this option means that only the
origin cluster must clear the singleton dependency. This option must be set
in every cluster in the federation.
.TP
\fBkill_invalid_depend\fR
If a job has an invalid dependency and it can never run terminate it
and set its state to be JOB_CANCELLED. By default the job stays pending
with reason DependencyNeverSatisfied.
\fBmax_depend_depth=#\fR
Maximum number of jobs to test for a circular job dependency. Stop testing
after this number of job dependencies have been tested. The default value is
10 jobs.
.RE

.TP
\fBDisableRootJobs\fR
If set to "YES" then user root will be prevented from running any jobs.
The default value is "NO", meaning user root will be able to execute jobs.
\fBDisableRootJobs\fR may also be set by partition.

If set to "ANY" a job must satisfy any of the requested partitions
to be submitted. The default value is "NO".
NOTE: If set, then a job's QOS can not be used to exceed partition limits.
NOTE: The partition limits being considered are its configured MaxMemPerCPU,
MaxMemPerNode, MinNodes, MaxNodes, MaxTime, AllocNodes, AllowAccounts,
AllowGroups, AllowQOS, and QOS usage threshold.

.TP
\fBEpilog\fR
Fully qualified pathname of a script to execute as user root on every
node when a user's job completes (e.g. "/usr/local/slurm/epilog"). A
glob pattern (See \fBglob\fR (7)) may also be used to run more than
one epilog script (e.g. "/etc/slurm/epilog.d/*"). The Epilog script
or scripts may be used to purge files, disable user login, etc.
By default there is no epilog.
See \fBProlog and Epilog Scripts\fR for more information.

.TP
\fBEpilogMsgTime\fR
The number of microseconds that the slurmctld daemon requires to process
an epilog completion message from the slurmd daemons. This parameter can
be used to prevent a burst of epilog completion messages from being sent
at the same time which should help prevent lost messages and improve
throughput for large jobs.
The default value is 2000 microseconds.
For a 1000 node job, this spreads the epilog completion messages out over
two seconds.

.TP
\fBEpilogSlurmctld\fR
Fully qualified pathname of a program for the slurmctld to execute
upon termination of a job allocation (e.g.
"/usr/local/slurm/epilog_controller").
The program executes as SlurmUser, which gives it permission to drain
nodes and requeue the job if a failure occurs (See scontrol(1)).
Exactly what the program does and how it accomplishes this is completely at
the discretion of the system administrator.
Information about the job being initiated, its allocated nodes, etc. are
passed to the program using environment variables.
See \fBProlog and Epilog Scripts\fR for more information.

.TP
\fBExtSensorsFreq\fR
The external sensors plugin sampling interval.
If \fBExtSensorsType=ext_sensors/none\fR, this parameter is ignored.
For all other values of \fBExtSensorsType\fR, this parameter is the number
of seconds between external sensors samples for hardware components (nodes,
switches, etc.) The default value is zero. This value disables external
sensors sampling. Note: This parameter does not affect external sensors
data collection for jobs/steps.

.TP
External sensors data is collected from the RRD database.
.RE

.TP
\fBFairShareDampeningFactor\fR
Dampen the effect of exceeding a user or group's fair share of allocated
resources. Higher values will provides greater ability to differentiate
between exceeding the fair share at high levels (e.g. a value of 1 results
in almost no difference between overconsumption by a factor of 10 and 100,
while a value of 5 will result in a significant difference in priority).
The default value is 1.

.TP
\fBFederationParameters\fR
Used to define federation options. Multiple options may be comma separated.

.RS
.TP
\fBfed_display\fR
If set, then the client status commands (e.g. squeue, sinfo, sprio, etc.) will
display information in a federated view by default. This option is functionally
equivalent to using the \-\-federation options on each command. Use the client's
\-\-local option to override the federated view and get a local view of the
given cluster.
.RE

.TP
\fBFirstJobId\fR
The job id to be used for the first submitted to Slurm without a
specific requested value. Job id values generated will incremented by 1
for each subsequent job. This may be used to provide a meta\-scheduler
with a job id space which is disjoint from the interactive jobs.
The default value is 1.
Also see \fBMaxJobId\fR

.TP
\fBGetEnvTimeout\fR
Controls how long the job should wait (in seconds) to load the user's
environment before attempting to load it from a cache file.
Applies when the salloc or sbatch \fI\-\-get\-user\-env\fR option is used.
If set to 0 then always load the user's environment from the cache file.
The default value is 2 seconds.

.TP
\fBGresTypes\fR
A comma delimited list of generic resources to be managed (e.g.
\fIGresTypes=gpu,mps\fR).
These resources may have an associated GRES plugin of the same name providing
additional functionality.
No generic resources are managed by default.
Ensure this parameter is consistent across all nodes in the cluster for
proper operation.
\fBGroupUpdateTime\fR
Controls how frequently information about which users are members of
groups allowed to use a partition will be updated, and how long user
group membership lists will be cached.
The time interval is given in seconds with a default value of 600 seconds.
A value of zero will prevent periodic updating of group membership information.
Also see the \fBGroupUpdateForce\fR parameter.

.TP
\fBGpuFreqDef\fR=[<\fItype\fR]=\fIvalue\fR>[,<\fItype\fR=\fIvalue\fR>]
Default GPU frequency to use when running a job step if it
has not been explicitly set using the \-\-gpu\-freq option.
This option can be used to independently configure the GPU and its memory
frequencies. Defaults to "high,memory=high".
After the job is completed, the frequencies of all affected GPUs will be reset
to the highest possible values.
In some cases, system power caps may override the requested values.
The field \fItype\fR can be "memory".
If \fItype\fR is not specified, the GPU frequency is implied.
The \fIvalue\fR field can either be "low", "medium", "high", "highm1" or
a numeric value in megahertz (MHz).
If the specified numeric value is not possible, a value as close as
possible will be used.
See below for definition of the values.
Examples of use include "GpuFreqDef=medium,memory=high and "GpuFreqDef=450".

Supported \fIvalue\fR definitions:
.RS
.TP 10
\fBlow\fR
the lowest available frequency.
.TP
\fBmedium\fR
attempts to set a frequency in the middle of the available range.
.TP
\fBhigh\fR
the highest available frequency.
.TP
\fBhighm1\fR
(high minus one) will select the next highest available frequency.
.RE

.TP
\fBHealthCheckInterval\fR
The interval in seconds between executions of \fBHealthCheckProgram\fR.
The default value is zero, which disables execution.

.TP
\fBHealthCheckNodeState\fR
Identify what node states should execute the \fBHealthCheckProgram\fR.
Multiple state values may be specified with a comma separator.
The default value is ANY to execute on nodes in any state.
\fBIDLE\fR
Run on nodes in the IDLE state.
.TP
\fBMIXED\fR
Run on nodes in the MIXED state (some CPUs idle and other CPUs allocated).
.RE

.TP
\fBHealthCheckProgram\fR
Fully qualified pathname of a script to execute as user root periodically
on all compute nodes that are \fBnot\fR in the NOT_RESPONDING state. This
program may be used to verify the node is fully operational and DRAIN the node
or send email if a problem is detected.
Any action to be taken must be explicitly performed by the program
(e.g. execute
"scontrol update NodeName=foo State=drain Reason=tmp_file_system_full"
to drain a node).
The execution interval is controlled using the \fBHealthCheckInterval\fR
parameter.
Note that the \fBHealthCheckProgram\fR will be executed at the same time
on all nodes to minimize its impact upon parallel programs.
This program is will be killed if it does not terminate normally within
60 seconds.
This program will also be executed when the slurmd daemon is first started and
before it registers with the slurmctld daemon.
By default, no program will be executed.

.TP
\fBInactiveLimit\fR
The interval, in seconds, after which a non\-responsive job allocation
command (e.g. \fBsrun\fR or \fBsalloc\fR) will result in the job being
terminated. If the node on which the command is executed fails or the
command abnormally terminates, this will terminate its job allocation.
This option has no effect upon batch jobs.
When setting a value, take into consideration that a debugger using \fBsrun\fR
to launch an application may leave the \fBsrun\fR command in a stopped state
for extended periods of time.
This limit is ignored for jobs running in partitions with the
\fBRootOnly\fR flag set (the scheduler running as root will be
responsible for the job).
The default value is unlimited (zero) and may not exceed 65533 seconds.

.TP
\fBJobAcctGatherType\fR
The job accounting mechanism type.
Acceptable values at present include "jobacct_gather/linux" (for Linux
systems) and is the recommended one, "jobacct_gather/cgroup" and
"jobacct_gather/none" (no accounting data collected).
The default value is "jobacct_gather/none".
"jobacct_gather/cgroup" is a plugin for the Linux operating system
that uses cgroups to collect accounting statistics. The plugin collects the
following statistics: From the cgroup memory subsystem: memory.usage_in_bytes
The job accounting and profiling sampling intervals.
The supported format is follows:
.RS
.TP 12
\fBJobAcctGatherFrequency=\fR\fI<datatype>\fR\fB=\fR\fI<interval>\fR
where \fI<datatype>\fR=\fI<interval>\fR specifies the task sampling
interval for the jobacct_gather plugin or a
sampling interval for a profiling type by the
acct_gather_profile plugin. Multiple,
comma-separated \fI<datatype>\fR=\fI<interval>\fR intervals
may be specified. Supported datatypes are as follows:
.RS
.TP
\fBtask=\fI<interval>\fR
where \fI<interval>\fR is the task sampling interval in seconds
for the jobacct_gather plugins and for task
profiling by the acct_gather_profile plugin.
.TP
\fBenergy=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for energy profiling using the acct_gather_energy plugin
.TP
\fBnetwork=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for infiniband profiling using the acct_gather_interconnect
plugin.
.TP
\fBfilesystem=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for filesystem profiling using the acct_gather_filesystem
plugin.
.TP
.RE
.RE
The default value for task sampling interval
is 30 seconds. The default value for all other intervals is 0.
An interval of 0 disables sampling of the specified type.
If the task sampling interval is 0, accounting
information is collected only at job termination (reducing Slurm
interference with the job).
.br
.br
Smaller (non\-zero) values have a greater impact upon job performance,
but a value of 30 seconds is not likely to be noticeable for
applications having less than 10,000 tasks.
.br
.br
Users can independently override each interval on a per job basis using the
\fB\-\-acctg\-freq\fR option when submitting the job.
.RE

.TP
every time accounting information is gathered by the JobAcctGather plugin.
This parameter should be used with caution because a job exceeding its memory
allocation may affect other processes and/or machine health.
NOTE: If available, it is recommended to limit memory by enabling task/cgroup
in TaskPlugin and making use of ConstrainRAMSpace=yes cgroup.conf instead
of using this JobAcctGather mechanism for memory enforcement, since the former
has a lower resolution (JobAcctGatherFreq) and OOMs could happen at some point.
.RE

.TP
\fBJobCompHost\fR
The name of the machine hosting the job completion database.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageHost\fR.

.TP
\fBJobCompLoc\fR
The fully qualified file name where job completion records are written
when the \fBJobCompType\fR is "jobcomp/filetxt" or the database where
job completion records are stored when the \fBJobCompType\fR is a
database, or an url with format http://yourelasticserver:port when
\fBJobCompType\fR is "jobcomp/elasticsearch".
NOTE: when you specify a URL for Elasticsearch, Slurm will remove any trailing
slashes "/" from the configured URL and append "/slurm/jobcomp", which are the
Elasticsearch index name (slurm) and mapping (jobcomp).
NOTE: More information is available at the Slurm web site
( https://slurm.schedmd.com/elasticsearch.html ).
Also see \fBDefaultStorageLoc\fR.

.TP
\fBJobCompParams\fR
Pass arbitrary text string to job completion plugin.
Also see \fBJobCompType\fR.

.TP
\fBJobCompPass\fR
The password used to gain access to the database to store the job
completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStoragePass\fR.

.TP
\fBJobCompPort\fR
The listening port of the job completion database server.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStoragePort\fR.

.TP
\fBJobCompType\fR
The job completion logging mechanism type.
Acceptable values at present include "jobcomp/none", "jobcomp/elasticsearch",
"jobcomp/filetxt", "jobcomp/lua", "jobcomp/mysql" and "jobcomp/script".
The value "jobcomp/mysql" indicates that a record of the job should be
written to a MySQL or MariaDB database specified by the \fBJobCompLoc\fR
parameter.
The value "jobcomp/script" indicates that a script specified by the
\fBJobCompLoc\fR parameter is to be executed with environment variables
indicating the job information.
.TP
\fBJobCompUser\fR
The user account for accessing the job completion database.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageUser\fR.

.TP
\fBJobContainerType\fR
Identifies the plugin to be used for job tracking.
The slurmd daemon must be restarted for a change in JobContainerType
to take effect.
NOTE: The \fBJobContainerType\fR applies to a job allocation, while
\fBProctrackType\fR applies to job steps.
Acceptable values at present include:
.RS
.TP 20
\fBjob_container/cncu\fR
used only for Cray systems (CNCU = Compute Node Clean Up)
.TP
\fBjob_container/none\fR
used for all other system types
.RE

.TP
\fBJobFileAppend\fR
This option controls what to do if a job's output or error file
exist when the job is started.
If \fBJobFileAppend\fR is set to a value of 1, then append to
the existing file.
By default, any existing file is truncated.

.TP
\fBJobRequeue\fR
This option controls the default ability for batch jobs to be requeued.
Jobs may be requeued explicitly by a system administrator, after node
failure, or upon preemption by a higher priority job.
If \fBJobRequeue\fR is set to a value of 1, then batch job may be requeued
unless explicitly disabled by the user.
If \fBJobRequeue\fR is set to a value of 0, then batch job will not be requeued
unless explicitly enabled by the user.
Use the \fBsbatch\fR \fI\-\-no\-requeue\fR or \fI\-\-requeue\fR
option to change the default behavior for individual jobs.
The default value is 1.

.TP
\fBJobSubmitPlugins\fR
.TP
\fBKeepAliveTime\fR
Specifies how long sockets communications used between the srun command and its
slurmstepd process are kept alive after disconnect.
Longer values can be used to improve reliability of communications in the
event of network failures.
The default value leaves the system default value.
The value may not exceed 65533.

.TP
\fBKillOnBadExit\fR
If set to 1, a step will be terminated immediately if any task is
crashed or aborted, as indicated by a non-zero exit code.
With the default value of 0, if one of the processes is crashed or aborted
the other processes will continue to run while the crashed or aborted process
waits. The user can override this configuration parameter by using srun's
\fB\-K\fR, \fB\-\-kill\-on\-bad\-exit\fR.

.TP
\fBKillWait\fR
The interval, in seconds, given to a job's processes between the
SIGTERM and SIGKILL signals upon reaching its time limit.
If the job fails to terminate gracefully in the interval specified,
it will be forcibly terminated.
The default value is 30 seconds.
The value may not exceed 65533.

.TP
\fBNodeFeaturesPlugins\fR
Identifies the plugins to be used for support of node features which can
change through time. For example, a node which might be booted with various
BIOS setting. This is supported through the use of a node's active_features
and available_features information.
Acceptable values at present include:
.RS
.TP 20
\fBnode_features/knl_cray\fR
used only for Intel Knights Landing processors (KNL) on Cray systems
.TP 20
\fBnode_features/knl_generic\fR
used for Intel Knights Landing processors (KNL) on a generic Linux system
.RE

.TP
\fBLaunchParameters\fR
Identifies options to the job launch plugin.
Acceptable values include:
.RS
.TP 24
\fBbatch_step_set_cpu_freq\fR
Set the cpu frequency for the batch step from given \-\-cpu\-freq, or
slurm.conf CpuFreqDef, option.  By default only steps started with srun will
Permits passwd and group resolution for a job to be serviced by slurmstepd rather
than requiring a lookup from a network based service. See
https://slurm.schedmd.com/nss_slurm.html for more information.
.TP 24
\fBlustre_no_flush\fR
If set on a Cray Native cluster, then do not flush the Lustre cache on job step
completion. This setting will only take effect after reconfiguring, and will
only take effect for newly launched jobs.
.TP 24
\fBmem_sort\fR
Sort NUMA memory at step start. User can override this default with
SLURM_MEM_BIND environment variable or \-\-mem\-bind=nosort command line option.
.TP
\fBdisable_send_gids\fR
By default the slurmctld will lookup and send the user_name and extended gids
for a job, rather than individual on each node as part of each task launch. Which avoids issues around name service scalability when launching jobs involving
many nodes.  Using this option will reverse this functionality.
.TP 24
\fBslurmstepd_memlock\fR
Lock the slurmstepd process's current memory in RAM.
.TP
\fBslurmstepd_memlock_all\fR
Lock the slurmstepd process's current and future memory in RAM.
.TP
\fBtest_exec\fR
Have srun verify existence of the executable program along with user
execute permission on the node where srun was called before attempting to
launch it on nodes in the step.
.RE

.TP
\fBLaunchType\fR
Identifies the mechanism to be used to launch application tasks.
Acceptable values include:
.RS
.TP
\fBlaunch/slurm\fR
The default value.
.RE

.TP
\fBLicenses\fR
Specification of licenses (or other resources available on all
nodes of the cluster) which can be allocated to jobs.
License names can optionally be followed by a colon
and count with a default count of one.
Multiple license names should be comma separated (e.g.
"Licenses=foo:4,bar").
Note that Slurm prevents jobs from being scheduled if their
required license specification is not available.
Slurm does not prevent jobs from using licenses that are
not explicitly listed in the job submission specification.
and ID.

.TP
\fBMailDomain\fR
Domain name to qualify usernames if email address is not explicitly given
with the "--mail-user" option. If unset, the local MTA will need to qualify
local address itself. Changes to MailDomain will only affect new jobs.

.TP
\fBMailProg\fR
Fully qualified pathname to the program used to send email per user request.
The default value is "/bin/mail" (or "/usr/bin/mail" if "/bin/mail" does not
exist but "/usr/bin/mail" does exist).

.TP
\fBMaxArraySize\fR
The maximum job array size.
The maximum job array task index value will be one less than MaxArraySize
to allow for an index value of zero.
Configure MaxArraySize to 0 in order to disable job array use.
The value may not exceed 4000001.
The value of \fBMaxJobCount\fR should be much larger than \fBMaxArraySize\fR.
The default value is 1001.

.TP
\fBMaxDBDMsgs\fR
When communication to the SlurmDBD is not possible the slurmctld will queue messages meant to processed when the the SlurmDBD is available again.
In order to avoid running out of memory the slurmctld will only queue so many messages.
The default value is 10000, or \fBMaxJobCount\fR * 2 + Node Count * 4, whichever is greater.  The value can not be less than 10000.

.TP
\fBMaxJobCount\fR
The maximum number of jobs Slurm can have in its active database
at one time. Set the values of \fBMaxJobCount\fR and \fBMinJobAge\fR
to ensure the slurmctld daemon does not exhaust its memory or other
resources. Once this limit is reached, requests to submit additional
jobs will fail. The default value is 10000 jobs.
NOTE: Each task of a job array counts as one job even though they will not
occupy separate job records until modified or initiated.
Performance can suffer with more than a few hundred thousand jobs.
Setting per MaxSubmitJobs per user is generally valuable to prevent a single
user from filling the system with jobs.
This is accomplished using Slurm's database and configuring enforcement of
resource limits.
This value may not be reset via "scontrol reconfig".
It only takes effect upon restart of the slurmctld daemon.

.TP
\fBMaxJobId\fR
The maximum job id to be used for jobs submitted to Slurm without a specific
requested value. Job ids are unsigned 32bit integers with the first 26 bits
reserved for local job ids and the remaining 6 bits reserved for a cluster id
\fBSelectType=select/cons_tres\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerNode\fR.
\fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.

NOTE: If a job specifies a memory per CPU limit that exceeds this system limit,
that job's count of CPUs per task will automatically be increased. This may
result in the job failing due to CPU count limits.

.TP
\fBMaxMemPerNode\fR
Maximum real memory size available per allocated node in megabytes.
Used to avoid over\-subscribing memory and causing paging.
\fBMaxMemPerNode\fR would generally be used if whole nodes
are allocated to jobs (\fBSelectType=select/linear\fR) and
resources are over\-subscribed (\fBOverSubscribe=yes\fR or
\fBOverSubscribe=force\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
\fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.

.TP
\fBMaxStepCount\fR
The maximum number of steps that any job can initiate. This parameter
is intended to limit the effect of bad batch scripts.
The default value is 40000 steps.

.TP
\fBMaxTasksPerNode\fR
Maximum number of tasks Slurm will allow a job step to spawn
on a single node. The default \fBMaxTasksPerNode\fR is 512.
May not exceed 65533.

.TP
\fBMCSParameters\fR
MCS = Multi-Category Security
MCS Plugin Parameters.
The supported parameters are specific to the \fBMCSPlugin\fR.
Changes to this value take effect when the Slurm daemons are reconfigured.
More information about MCS is available here

Man(1) output converted with man2html