Resource Limits (QoS)¶

Concept¶

Use QoS (Quality of Service) to limit job submissions per partition. Each partition gets its own QoS that restricts accounts to 1 job per partition.

Groups/accounts are structured as seen in Account Structure.

Implementation¶

1. Create a QoS for Each Partition¶

sacctmgr create qos name=PARTITION_NAME_qos maxjobspa=1

2. Assign the QoS to the Partition¶

sudo scontrol update partition=PARTITION_NAME qos=PARTITION_NAME_qos

Result¶

Each account can submit 1 job per partition.
Multiple partitions = multiple possible jobs per account.
Fair resource sharing across all accounts.

Example¶

# Create QoS
sacctmgr create qos name=normal_partition maxjobspa=1
sacctmgr create qos name=rtx3090_partition maxjobspa=1

# Apply to partitions
sudo scontrol update partition=Normal qos=normal_partition
sudo scontrol update partition=RTX3090 qos=rtx3090_partition

Make it permanent

These changes are active now but temporary. To make them permanent, add the QoS lines to /etc/slurm/slurm.conf and run sudo scontrol reconfigure.

See also Deadline Budgets & QoS for a more advanced, per-group GPU-hour budget built on top of this same QoS mechanism.