Skip to content

Cluster Conventions

Rules for where things live and how they reach users. Established September 2026.

Principle

Anything user-facing lives once on the shared NetApp volume (/SLURM, mounted on every node), not per node. Per-node changes are kept to what can't be shared. This avoids drift between nodes and nodes that were down missing changes.

Layout under /SLURM/public

Path Holds Owner / mode
/SLURM/public/apps/<tool>/<version>/ Installed software (one folder per version when versions matter) root, read-only for users
/SLURM/public/apps/<tool>/ Single-script tools with no versions (e.g. jupyter-start) root
/SLURM/public/modulefiles/<tool>/<version>.lua One Lmod modulefile per software version root
/SLURM/public/etc/cluster-env.sh Shell-settings loader; defines cluster_env root, 644
/SLURM/public/etc/cluster-env.d/NN-<name>.sh One file per shell feature root, 644
/SLURM/public/guide/<topic>/ README + example scripts users run or copy root, read-only
/SLURM/public/<tool>/... Shared data for a tool (e.g. /SLURM/public/ollama/models) root, read-only for users
/SLURM/public/models/ HuggingFace model zoo read-only for users

No secrets anywhere under /SLURM/public

Everything there is readable by all users. Root-only material goes elsewhere on the volume (e.g. a mode-700 directory) or local on the host that needs it.

Standard permissions after any change:

sudo chown -R root:root <path>
sudo chmod -R a+rX,go-w <path>

How Things Reach Users

Kind Mechanism User action
Software Lmod module in /SLURM/public/modulefiles module load <tool>
Shell settings / helper commands cluster_env <name> feature One line in ~/.bashrc; comment it to turn off
Examples /SLURM/public/guide, linked as ~/guide in every home Run directly or copy to edit
  • No forced/"core" features: every feature is a visible line in the user's .bashrc.
  • Software is not added to PATH globally; users load modules.
  • Prefer plain example scripts in ~/guide over custom wrapper tools.

The User .bashrc

  • Template: /etc/skel/.bashrc on server02 (accounts are created there).
  • JRCAI block at the top, before the interactive check, so batch jobs and remote commands get it:
# >>> JRCAI cluster settings (comment a line to turn it off) >>>
source /SLURM/public/etc/cluster-env.sh
cluster_env module        # module command (Lmod)
cluster_env slurm         # SLURM defaults
cluster_env jupyter       # jupyter-start command
cluster_env conda         # Anaconda
cluster_env nexus         # JRCAI package cache
# module load ollama      # Ollama
# <<< JRCAI cluster settings <<<
  • Personal lines go at the end under # ---- Your own settings below ----.
  • Batch examples start with source ~/.bashrc (absolute path).

cluster_env Features

See cluster-env under Operations for the full reference — the loader itself, each feature file, admin tasks (add/remove/rename a feature), checks, and troubleshooting.

Node-Level Rules

Kept per node only because they can't be shared:

Item Setting
Packages Lmod (apt), Apptainer (official PPA ppa:apptainer/ppa), NVIDIA driver, Anaconda in /opt/anaconda3
/etc/lmod/modulespath First line /SLURM/public/modulefiles
/etc/cron.allow root only
sshd_config (compute nodes) DenyGroups slurmUsers, placed above any Match block
server02 sshd_config Match Group slurmUsers → SFTP only, chroot /raid_storage/SLURM/home, no TTY/forwarding
Services No user-facing services (e.g. Ollama) outside SLURM on compute nodes

SLURM settings relied on: ProctrackType=proctrack/cgroup, LaunchParameters=use_interactive_step.

Admin Working Habits

  • Read-only check → change → verify. One step at a time.
  • Back up before editing (<file>.bak-<date>); write commands so rerunning is safe (skip if already done).
  • With clush, sudo goes inside the remote command: sudo clush -w <nodes> 'sudo <cmd>'. Plain ssh node 'sudo ...' fails (no TTY for the password).
  • Pipe files through clush instead of remote sudo over ssh: cat file | sudo clush -w <node> 'sudo tee <dest> >/dev/null'.
  • Long files: transfer (scp, or via clush/ssh pipe), not terminal paste; check with bash -n / md5sum.
  • sshd: validate with sshd -t before systemctl try-reload-or-restart ssh; confirm with sshd -T.
  • Mask secrets before sharing output.
  • Match blocks in sshd_config run to the next Match or end of file — never append global rules at the end.