SLURM¶

Set Up SLURM User and Group¶

Same slurm user/group, same UID/GID as every other node:

sudo addgroup --gid 1121 slurm

sudo adduser --uid 1121 --gid 1121 --no-create-home --disabled-password --gecos "" slurm

Install Built SLURM¶

Install dependencies:

sudo apt-get update
sudo apt-get install -y libnuma-dev
sudo apt-get install -y libpam0g-dev
sudo apt-get install -y libhdf5-dev
sudo apt-get install -y liblz4-dev libhwloc-dev
sudo apt-get install -y libgtk2.0-dev libglib2.0-dev
sudo apt-get install -y librdkafka-dev
sudo apt-get install -y libdbus-1-dev
sudo apt-get install -y check
sudo apt-get install -y liblua5.3-dev
sudo apt-get install -y libreadline-dev
sudo apt-get install -y linux-headers-$(uname -r)
sudo apt-get install -y freeipmi-tools libfreeipmi-dev
sudo apt-get install -y rrdtool librrd-dev
sudo apt-get install -y libjson-c-dev libjansson-dev
sudo apt-get install -y libjwt-dev
sudo apt-get install -y libhttp-parser-dev
sudo apt-get install -y libyaml-dev
sudo apt-get install -y man2html
sudo apt-get install -y mailutils
sudo apt-get install -y libhdf5-dev
sudo apt-get install -y libmysqlclient-dev
sudo apt-get install -y lua5.3 liblua5.3-0 lua-posix lua-filesystem

Install SLURM that was already built during controller setup — it should exist in the shared NFS dir:

sudo dpkg -i slurm-23.11.0_1.0_amd64.deb

sudo cp ubuntu-slurm/slurmd.service /etc/systemd/system/

SLURM Configuration¶

Make the directories needed:

sudo mkdir -p /etc/slurm /etc/slurm/prolog.d /etc/slurm/epilog.d /var/spool/slurm/ctld /var/spool/slurm/d /var/log/slurm
sudo chown slurm /var/spool/slurm/ctld /var/spool/slurm/d /var/log/slurm

This directory should also be created:

sudo mkdir -p /var/spool/slurm/d
sudo chown slurm /var/spool/slurm/d

And this one:

sudo mkdir -p /run/slurmd
sudo chown slurm:slurm /run/slurmd

slurm.conf¶

slurm.conf is edited once on the controller (see Controller: slurm.conf for the canonical copy) and copied to every node, including this one. The login node is deliberately kept out of real scheduling — it has a minimal NodeName= entry (no Gres=/CPUs=/RealMemory=) and its own hidden, down partition:

NodeName=jrcai23 State=UNKNOWN
PartitionName=LoginNode Nodes=jrcai23 State=DOWN Hidden=YES

Copy the file here once the controller's copy includes this node:

sudo cp ubuntu-slurm/slurm.conf /etc/slurm/
sudo chown slurm:slurm /etc/slurm/slurm.conf

gres.conf¶

The login node has no GPUs for SLURM to manage, so there's nothing to add here — just copy the default file as-is:

sudo cp ubuntu-slurm/gres.conf /etc/slurm/gres.conf

cgroup.conf¶

Edit the cgroup file:

sudo nano ubuntu-slurm/cgroup.conf

Comment out some lines, as follows:

#CgroupAutomount=yes
#CgroupReleaseAgentDir="/etc/slurm/cgroup"

ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes
#TaskAffinity=ye

Then copy it to the system:

sudo cp ubuntu-slurm/cgroup.conf /etc/slurm/cgroup.conf

cgroup_allowed_devices_file.conf¶

Copy it as-is:

sudo cp ubuntu-slurm/cgroup_allowed_devices_file.conf /etc/slurm/cgroup_allowed_devices_file.conf

Configure cgroups (GRUB)¶

sudo nano /etc/default/grub

Add cgroup and swap to GRUB_CMDLINE_LINUX:

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

sudo update-grub

Start SLURM¶

sudo systemctl enable slurmd
sudo systemctl start slurmd

Check status:

sudo systemctl status slurmd

Note

slurmd still runs here so squeue/sinfo/sbatch work for interactive users — the node just never appears as schedulable, since its only partition (LoginNode) is State=DOWN Hidden=YES.