clush¶

Parallel command execution across cluster nodes, gated behind sudo via a dedicated cluster_admin key and sudoers rule.

Info

The one-time setup below is also folded into the per-role installation pages: Controller, Login Node, Worker Nodes.

Setup¶

clush runs commands in parallel across nodes. It's gated behind sudo to require your password before accessing the cluster.

Step 0: Install clush (on server02)¶

sudo apt install clustershell

Step 1: Generate cluster_admin Key (on server02)¶

sudo ssh-keygen -t ed25519 -f /root/.ssh/cluster_admin_key -N ""

Step 2: Create cluster_admin on All Nodes¶

for node in jrcai01 jrcai02 jrcai06 jrcai07 jrcai08 jrcai09 jrcai10 jrcai12 jrcai13 jrcai14 jrcai15 jrcai16 jrcai17 jrcai18 jrcai19 login01; do
  ssh jrcai_admin@$node 'sudo useradd -M -s /bin/bash cluster_admin'
done

Step 3: SSH Key Directory for cluster_admin¶

for node in jrcai01 jrcai02 jrcai06 jrcai07 jrcai08 jrcai09 jrcai10 jrcai12 jrcai13 jrcai14 jrcai15 jrcai16 jrcai17 jrcai18 jrcai19 login01; do
  ssh jrcai_admin@$node 'sudo mkdir -p /etc/ssh-keys/cluster_admin && sudo chmod 700 /etc/ssh-keys/cluster_admin && sudo chown cluster_admin:cluster_admin /etc/ssh-keys/cluster_admin'
done

Step 4: Add sshd Match Block to Each Node¶

for node in jrcai01 jrcai02 jrcai06 jrcai07 jrcai08 jrcai09 jrcai10 jrcai12 jrcai13 jrcai14 jrcai15 jrcai16 jrcai17 jrcai18 jrcai19 login01; do
  ssh jrcai_admin@$node 'echo -e "\nMatch User cluster_admin\n    AuthorizedKeysFile /etc/ssh-keys/cluster_admin/authorized_keys" | sudo tee -a /etc/ssh/sshd_config && sudo sshd -t && sudo systemctl restart sshd'
done

(If the service is ssh not sshd, adjust the systemctl restart line.)

Step 5: Distribute cluster_admin Public Key¶

KEY=$(sudo cat /root/.ssh/cluster_admin_key.pub)
for node in jrcai01 jrcai02 jrcai06 jrcai07 jrcai08 jrcai09 jrcai10 jrcai12 jrcai13 jrcai14 jrcai15 jrcai16 jrcai17 jrcai18 jrcai19 login01; do
  ssh -t jrcai_admin@$node "echo 'from=\"10.22.154.100\" $KEY' | sudo tee /etc/ssh-keys/cluster_admin/authorized_keys && sudo chmod 600 /etc/ssh-keys/cluster_admin/authorized_keys && sudo chown cluster_admin:cluster_admin /etc/ssh-keys/cluster_admin/authorized_keys"
done

Step 6: NOPASSWD Sudoers for cluster_admin¶

for node in jrcai01 jrcai02 jrcai06 jrcai07 jrcai08 jrcai09 jrcai10 jrcai12 jrcai13 jrcai14 jrcai15 jrcai16 jrcai17 jrcai18 jrcai19 login01; do
  echo "--- $node ---"
  ssh -t jrcai_admin@$node 'echo "cluster_admin ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cluster_admin && sudo chmod 440 /etc/sudoers.d/cluster_admin'
done

Step 7: Lock Down clush Binary on server02¶

sudo mv /home/jrcai_admin/.ssh/cluster_admin_key /root/.ssh/cluster_admin_key
sudo mv /home/jrcai_admin/.ssh/cluster_admin_key.pub /root/.ssh/cluster_admin_key.pub
sudo chown root:root /root/.ssh/cluster_admin_key*
sudo chmod 600 /root/.ssh/cluster_admin_key

sudo tee /root/.ssh/config << 'EOF'
Host jrcai* login01
    User cluster_admin
    IdentityFile /root/.ssh/cluster_admin_key
    StrictHostKeyChecking accept-new
EOF
sudo chmod 600 /root/.ssh/config

sudo chmod 750 /usr/bin/clush

Step 8: Create Node Groups¶

sudo mkdir -p /etc/clustershell/groups.d

sudo tee /etc/clustershell/groups.d/local.cfg << 'EOF'
A100: server02
A4500: jrcai[14-16]
RTX3090: jrcai[01-02,06-10]
A5000: jrcai[12-13]
A6000: jrcai[17-19],server01
all: server02,jrcai[01-02,06-10,12-19]
EOF

sudo tee /etc/clustershell/groups.conf << 'EOF'
[Main]
default: local

[local]
map: sed -n 's/^$GROUP:\(.*\)/\1/p' /etc/clustershell/groups.d/local.cfg
list: sed -n 's/^\(.*\):.*/\1/p' /etc/clustershell/groups.d/local.cfg
EOF

Verification¶

# Plain clush fails (permission denied)
clush -g RTX3090 -b 'whoami'

# sudo clush works
sudo clush -g RTX3090 -b 'whoami'

# Should return cluster_admin on all nodes
sudo clush -g RTX3090 -b 'sudo -n whoami'
# Should return root

Day-to-Day Operations¶

Node Groups¶

@RTX3090: jrcai[01-02,06-10]
@A5000: jrcai[12-13]
@A4500: jrcai[14-16]
@A6000: jrcai[17-19],server01
@all: server02,jrcai[01-02,06-10,12-19]

List groups:

nodeset -l

Common Commands¶

# Read-only check
sudo clush -g RTX3090 -b 'uptime'

# Privileged operation
sudo clush -g A5000 -b 'sudo -n apt update'

# Specific nodes
sudo clush -w jrcai01,jrcai02,jrcai06 -b 'whoami'

Adding a Node¶

Runbook for adding a new node (example: jrcai21) to an existing clush setup. See Setup if clush itself isn't configured yet.

# 1. Create cluster_admin
ssh jrcai_admin@jrcai21 'sudo useradd -M -s /bin/bash cluster_admin'

# 2. SSH key directory
ssh jrcai_admin@jrcai21 'sudo mkdir -p /etc/ssh-keys/cluster_admin && sudo chmod 700 /etc/ssh-keys/cluster_admin && sudo chown cluster_admin:cluster_admin /etc/ssh-keys/cluster_admin'

# 3. sshd Match block
ssh jrcai_admin@jrcai21 'echo -e "\nMatch User cluster_admin\n    AuthorizedKeysFile /etc/ssh-keys/cluster_admin/authorized_keys" | sudo tee -a /etc/ssh/sshd_config && sudo sshd -t && sudo systemctl restart sshd'

# 4. Distribute cluster_admin public key
KEY=$(sudo cat /root/.ssh/cluster_admin_key.pub)
ssh -t jrcai_admin@jrcai21 "echo 'from=\"10.22.154.100\" $KEY' | sudo tee /etc/ssh-keys/cluster_admin/authorized_keys && sudo chmod 600 /etc/ssh-keys/cluster_admin/authorized_keys && sudo chown cluster_admin:cluster_admin /etc/ssh-keys/cluster_admin/authorized_keys"

# 5. NOPASSWD sudoers
ssh -t jrcai_admin@jrcai21 'echo "cluster_admin ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cluster_admin && sudo chmod 440 /etc/sudoers.d/cluster_admin'

# 6. Home directory (optional)
ssh -t jrcai_admin@jrcai21 'sudo mkdir -p /home/cluster_admin && sudo chown cluster_admin:cluster_admin /home/cluster_admin'

# 7. Test
sudo clush -w jrcai21 -b 'whoami'
# Should return: cluster_admin

# 8. Add to node groups
sudo nano /etc/clustershell/groups.d/local.cfg
# Add jrcai21 to the appropriate group (RTX3090, A5000, A4500, or A6000)