Node State & Conf Redeploy¶

Info

Quick-access versions of these two also live under Troubleshooting: Change Node State, Redeploy slurm.conf.

Change State of a Node¶

sudo scontrol update NodeName=jrcai07 State=DOWN Reason="Maintenance"

sudo scontrol update NodeName=jrcai08 State=RESUME

Edit slurm.conf¶

You can edit the default slurm.conf (the compute node list, SelectType, and ControlMachine) — see Controller: slurm.conf for the canonical reference copy:

sudo nano ubuntu-slurm/slurm.conf

Copy it out after editing, then refresh:

sudo cp ubuntu-slurm/slurm.conf /etc/slurm/
sudo chown slurm:slurm /etc/slurm/slurm.conf

Or push it to every node at once with clush:

sudo clush -a "cp /raid_storage/SLURM/home/slurm-admin/ubuntu-slurm/slurm.conf /etc/slurm/slurm.conf"
sudo clush -a "chown slurm:slurm /etc/slurm/slurm.conf"

Then restart:

sudo systemctl restart slurmd

On the controller/master, also restart the accounting/control daemons:

sudo systemctl restart slurmctld
sudo systemctl restart slurmdbd
sudo systemctl restart slurmd

Quick version for this cluster specifically:

cd /raid_storage/SLURM/home/slurm-admin
sudo cp ubuntu-slurm/slurm.conf /etc/slurm/
sudo chown slurm:slurm /etc/slurm/slurm.conf
sudo systemctl restart slurmd