stack.basicstack.de/infrastructure/ADD_WORKER_NODE.md

Adding a Worker Node to k3s Cluster - Quick Instruction

Purpose: Step-by-step checklist for adding a new worker node to the BasicStack k3s cluster.

Audience: CTO agent, infrastructure operators

Prerequisites:

• Hetzner Cloud access (console or hcloud CLI with credentials)
• kubectl access to the cluster
• SSH access to control plane (178.105.17.239)

---

Step 1: Create Hetzner Cloud VM

Via Hetzner Cloud Console

1. Navigate to Hetzner Cloud Console
2. Select the BasicStack project
3. Click "Add Server"
4. Configure:
   • Name: k3s-worker-N (use next available number)
   • Location: Match existing cluster (check with kubectl get nodes -o wide)
   • Image: Ubuntu 24.04 LTS
   • Type: CX32 or larger (minimum 4 vCPU, 8 GB RAM)
   • Network: Attach to existing private network (10.42.1.0/24)
   • SSH Keys: Select both cto-paperclip AND silentmaxx
   • Firewall: Attach fw-k3s
5. Click "Create & Buy now"
6. Record the IPs:
   • Private IP: ________________ (from Hetzner network tab)
   • Public IP: ________________ (from server overview)

Via hcloud CLI

# List available networks
hcloud network list

# List available SSH keys (verify both keys exist)
hcloud ssh-key list | grep -E 'cto-paperclip|silentmaxx'

# Create the server (adjust location as needed)
hcloud server create \
  --name k3s-worker-4 \
  --type cx32 \
  --image ubuntu-24.04 \
  --ssh-key cto-paperclip \
  --ssh-key silentmaxx \
  --network k3s-network \
  --firewall fw-k3s \
  --location nbg1

# Record the IPs from output

---

Step 2: Verify Firewall Configuration

Why: Flannel VXLAN uses public IPs for pod networking tunnels. Without the firewall rule, pods cannot communicate.

# Check if public IP is in fw-k3s
hcloud firewall describe fw-k3s | grep <public-ip>

# If missing, add the rule:
hcloud firewall add-rule fw-k3s \
  --direction in \
  --protocol tcp \
  --source-ips <public-ip>/32 \
  --port any

Checklist:

• Firewall rule exists for node's public IP

---

Step 3: Verify SSH Access

Test connectivity before proceeding:

# Test public IP
ssh root@<public-ip> 'hostname && date'

# Test private IP (if accessible)
ssh root@<private-ip> 'hostname && date'

Checklist:

• SSH access confirmed to public IP
• SSH access confirmed to private IP

---

Step 4: Run Provisioning Script

Location: infrastructure/scripts/provision-k3s-worker.sh

What it does:

• Detects current k3s cluster version
• Prompts for firewall confirmation
• Retrieves join token from control plane
• Installs k3s with version pinning
• Verifies successful cluster join

Execution:

cd /path/to/stack.basicstack.de/infrastructure/scripts

# Syntax: ./provision-k3s-worker.sh <private-ip> <public-ip>
./provision-k3s-worker.sh <PRIVATE-IP> <PUBLIC-IP>

# Example:
# ./provision-k3s-worker.sh 10.42.1.6 167.233.121.122

During execution:

• Confirm firewall update when prompted (press Enter after verifying Step 2)
• Wait for k3s installation (may take 5-10 minutes)
• Review the final node list output

Checklist:

• Script executed successfully
• No error messages in output
• Node appears in final kubectl get nodes output

---

Step 5: Verify Node Health

Run these verification commands:

# 1. Check node status (should be Ready)
kubectl get nodes -o wide

# 2. Verify internal IP matches Hetzner private IP
kubectl get node k3s-worker-N -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'
# Expected: 10.42.1.X (the private IP from Step 1)

# 3. Check k3s version consistency
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
# All nodes should have the same version

# 4. Verify flannel configuration
kubectl get node k3s-worker-N -o yaml | grep flannel
# Should see flannel.alpha.coreos.com annotations

# 5. Check pod CIDR assignment
kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}'
# Should return something like 10.244.X.0/24

Expected Results:

• Node status: Ready
• Internal IP: Matches private IP from Hetzner network
• k3s version: Matches existing nodes
• Flannel annotations: Present
• Pod CIDR: Assigned (10.244.X.0/24)

Checklist:

• Node status is Ready
• Internal IP is correct (Hetzner private IP)
• k3s version matches cluster
• Flannel annotations present
• Pod CIDR assigned

---

Step 6: Label the Node (Optional)

Add standard worker label:

kubectl label nodes k3s-worker-N node-role.kubernetes.io/worker=worker

Checklist:

• Node labeled

---

Step 7: Monitor Pod Scheduling

Watch for pods to be scheduled on the new node:

# Check overall pod distribution
kubectl get pods --all-namespaces -o wide | grep k3s-worker-N

# If adding capacity for specific workload (e.g., Stalwart):
kubectl get pod -n stalwart -l app=stalwart -o wide

# Watch for scheduling events
kubectl get events --sort-by='.lastTimestamp' | grep k3s-worker-N

Checklist:

• Pods are being scheduled on new node
• Target workload (if specified) successfully scheduled

---

Step 8: Update Documentation

Update the cluster architecture table in infrastructure/K3S_OPERATIONS.md:

1. Get the assigned pod CIDR:

   kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}'
   

2. Edit infrastructure/K3S_OPERATIONS.md and add to the "Node CIDRs" table:

   | k3s-worker-N | 10.244.X.0/24 | 10.42.1.Y | <public-ip> |
   

3. Commit the change:

   cd infrastructure
   git add K3S_OPERATIONS.md
   git commit -m "docs: add k3s-worker-N to cluster architecture table
   
   Added new worker node with Pod CIDR 10.244.X.0/24.
   Node added to resolve resource constraints for workload scheduling.
   
   Co-Authored-By: Paperclip <noreply@paperclip.ing>"
   git push
   

Checklist:

• K3S_OPERATIONS.md updated with new node
• Changes committed and pushed

---

Success Criteria

All checkboxes above should be completed:

• VM created with correct specifications
• Both SSH keys (cto-paperclip, silentmaxx) added
• Firewall rule configured
• SSH access verified
• Provisioning script executed successfully
• Node status: Ready
• Internal IP matches Hetzner private IP
• k3s version matches cluster
• Flannel annotations present
• Pod CIDR assigned
• Pods scheduling on new node
• Documentation updated

---

Rollback Procedure

If issues occur and the node needs to be removed:

# 1. Drain the node (move all pods off)
kubectl drain k3s-worker-N --ignore-daemonsets --delete-emptydir-data --timeout=5m

# 2. Delete from cluster
kubectl delete node k3s-worker-N

# 3. Remove firewall rule (get rule ID first)
hcloud firewall describe fw-k3s
hcloud firewall delete-rule fw-k3s <rule-id>

# 4. Delete Hetzner VM
hcloud server delete k3s-worker-N

---

Troubleshooting

Node Not Appearing in kubectl get nodes

Check:

1. Firewall rule is configured (Step 2)
2. k3s-agent service is running on the node:

   ssh root@<node-ip> 'systemctl status k3s-agent'
   

3. Connectivity to control plane:

   ssh root@<node-ip> 'curl -k https://10.42.1.1:6443'
   

Internal IP Shows Public IP Instead of Private IP

Cause: Node not properly connected to Hetzner private network.

Fix:

1. Verify network attachment in Hetzner Cloud Console
2. Ensure private network interface is up:

   ssh root@<node-ip> 'ip addr show'
   

3. May need to recreate the VM with network properly attached

k3s Version Mismatch

Cause: Provisioning script couldn't detect cluster version, or manual installation was attempted.

Fix:

1. Cordon the node:

   kubectl cordon k3s-worker-N
   

2. Get correct version:

   CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | sed 's/v//')
   

3. Reinstall with correct version:

   K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token')
   ssh root@<node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
     --server https://10.42.1.1:6443 \
     --token '${K3S_TOKEN}'"
   

4. Uncordon:

   kubectl uncordon k3s-worker-N
   

---

Reference Links

• K3S_OPERATIONS.md - Full operational guide
• CLUSTER_ACCESS.md - Access and authentication
• Hetzner Cloud Docs
• k3s Documentation

---

Last Updated: 2026-07-26 Maintained By: CTO Agent