clush¶

Parallel command execution across cluster nodes, gated behind sudo via a dedicated cluster_admin key and sudoers rule — copied from clush, kept in sync there. This page covers the server02-only side: installing clush, generating the key, locking the binary down, and defining node groups. Every other node also needs a cluster_admin account before clush can reach it — see the matching clush page on the Login Node and Worker Nodes.

Install clush¶

sudo apt install clustershell

Generate cluster_admin Key¶

sudo ssh-keygen -t ed25519 -f /root/.ssh/cluster_admin_key -N ""

Info

The public half of this key gets distributed to every other node's cluster_admin account — see Login Node and Worker Nodes.

Lock Down clush Binary¶

sudo mv /home/jrcai_admin/.ssh/cluster_admin_key /root/.ssh/cluster_admin_key
sudo mv /home/jrcai_admin/.ssh/cluster_admin_key.pub /root/.ssh/cluster_admin_key.pub
sudo chown root:root /root/.ssh/cluster_admin_key*
sudo chmod 600 /root/.ssh/cluster_admin_key

sudo tee /root/.ssh/config << 'EOF'
Host jrcai* login01
    User cluster_admin
    IdentityFile /root/.ssh/cluster_admin_key
    StrictHostKeyChecking accept-new
EOF
sudo chmod 600 /root/.ssh/config

sudo chmod 750 /usr/bin/clush

Create Node Groups¶

sudo mkdir -p /etc/clustershell/groups.d

sudo tee /etc/clustershell/groups.d/local.cfg << 'EOF'
A100: server02
A4500: jrcai[14-16]
RTX3090: jrcai[01-02,06-10]
A5000: jrcai[12-13]
A6000: jrcai[17-19],server01
all: server02,jrcai[01-02,06-10,12-19]
EOF

sudo tee /etc/clustershell/groups.conf << 'EOF'
[Main]
default: local

[local]
map: sed -n 's/^$GROUP:\(.*\)/\1/p' /etc/clustershell/groups.d/local.cfg
list: sed -n 's/^\(.*\):.*/\1/p' /etc/clustershell/groups.d/local.cfg
EOF

Verification¶

Only works once the target nodes have their cluster_admin account set up (see Login Node and Worker Nodes):

# Plain clush fails (permission denied)
clush -g RTX3090 -b 'whoami'

# sudo clush works
sudo clush -g RTX3090 -b 'whoami'

# Should return cluster_admin on all nodes
sudo clush -g RTX3090 -b 'sudo -n whoami'
# Should return root

For day-to-day usage (common commands, adding a node later) see clush under Operations.