From ae1573d6e8b25b2fedb5eac48193f7d79ca434a0 Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 26 Jul 2026 16:07:40 +0000 Subject: [PATCH] docs: add standalone worker node addition instruction Created ADD_WORKER_NODE.md as a step-by-step checklist for adding new worker nodes to the k3s cluster. This complements the existing K3S_OPERATIONS.md with a focused, actionable guide. Key features: - Pre-flight checklist format - Hetzner Cloud VM creation steps (console and CLI) - SSH key requirements (cto-paperclip, silentmaxx) - Firewall configuration verification - Provisioning script usage - Health verification steps - Documentation update procedure - Rollback instructions - Troubleshooting guide Created for DEV-408 (adding worker node to resolve Stalwart scheduling). Co-Authored-By: Paperclip --- infrastructure/ADD_WORKER_NODE.md | 345 ++++++++++++++++++++++++++++++ 1 file changed, 345 insertions(+) create mode 100644 infrastructure/ADD_WORKER_NODE.md diff --git a/infrastructure/ADD_WORKER_NODE.md b/infrastructure/ADD_WORKER_NODE.md new file mode 100644 index 0000000..1b94239 --- /dev/null +++ b/infrastructure/ADD_WORKER_NODE.md @@ -0,0 +1,345 @@ +# Adding a Worker Node to k3s Cluster - Quick Instruction + +**Purpose:** Step-by-step checklist for adding a new worker node to the BasicStack k3s cluster. + +**Audience:** CTO agent, infrastructure operators + +**Prerequisites:** +- Hetzner Cloud access (console or hcloud CLI with credentials) +- kubectl access to the cluster +- SSH access to control plane (178.105.17.239) + +--- + +## Step 1: Create Hetzner Cloud VM + +### Via Hetzner Cloud Console + +1. Navigate to [Hetzner Cloud Console](https://console.hetzner.cloud/) +2. Select the BasicStack project +3. Click "Add Server" +4. Configure: + - **Name:** `k3s-worker-N` (use next available number) + - **Location:** Match existing cluster (check with `kubectl get nodes -o wide`) + - **Image:** Ubuntu 24.04 LTS + - **Type:** CX32 or larger (minimum 4 vCPU, 8 GB RAM) + - **Network:** Attach to existing private network (`10.42.1.0/24`) + - **SSH Keys:** Select both `cto-paperclip` AND `silentmaxx` + - **Firewall:** Attach `fw-k3s` +5. Click "Create & Buy now" +6. **Record the IPs:** + - Private IP: `________________` (from Hetzner network tab) + - Public IP: `________________` (from server overview) + +### Via hcloud CLI + +```bash +# List available networks +hcloud network list + +# List available SSH keys (verify both keys exist) +hcloud ssh-key list | grep -E 'cto-paperclip|silentmaxx' + +# Create the server (adjust location as needed) +hcloud server create \ + --name k3s-worker-4 \ + --type cx32 \ + --image ubuntu-24.04 \ + --ssh-key cto-paperclip \ + --ssh-key silentmaxx \ + --network k3s-network \ + --firewall fw-k3s \ + --location nbg1 + +# Record the IPs from output +``` + +--- + +## Step 2: Verify Firewall Configuration + +**Why:** Flannel VXLAN uses public IPs for pod networking tunnels. Without the firewall rule, pods cannot communicate. + +```bash +# Check if public IP is in fw-k3s +hcloud firewall describe fw-k3s | grep + +# If missing, add the rule: +hcloud firewall add-rule fw-k3s \ + --direction in \ + --protocol tcp \ + --source-ips /32 \ + --port any +``` + +**Checklist:** +- [ ] Firewall rule exists for node's public IP + +--- + +## Step 3: Verify SSH Access + +Test connectivity before proceeding: + +```bash +# Test public IP +ssh root@ 'hostname && date' + +# Test private IP (if accessible) +ssh root@ 'hostname && date' +``` + +**Checklist:** +- [ ] SSH access confirmed to public IP +- [ ] SSH access confirmed to private IP + +--- + +## Step 4: Run Provisioning Script + +**Location:** `infrastructure/scripts/provision-k3s-worker.sh` + +**What it does:** +- Detects current k3s cluster version +- Prompts for firewall confirmation +- Retrieves join token from control plane +- Installs k3s with version pinning +- Verifies successful cluster join + +**Execution:** + +```bash +cd /path/to/stack.basicstack.de/infrastructure/scripts + +# Syntax: ./provision-k3s-worker.sh +./provision-k3s-worker.sh + +# Example: +# ./provision-k3s-worker.sh 10.42.1.6 167.233.121.122 +``` + +**During execution:** +- Confirm firewall update when prompted (press Enter after verifying Step 2) +- Wait for k3s installation (may take 5-10 minutes) +- Review the final node list output + +**Checklist:** +- [ ] Script executed successfully +- [ ] No error messages in output +- [ ] Node appears in final `kubectl get nodes` output + +--- + +## Step 5: Verify Node Health + +Run these verification commands: + +```bash +# 1. Check node status (should be Ready) +kubectl get nodes -o wide + +# 2. Verify internal IP matches Hetzner private IP +kubectl get node k3s-worker-N -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}' +# Expected: 10.42.1.X (the private IP from Step 1) + +# 3. Check k3s version consistency +kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion +# All nodes should have the same version + +# 4. Verify flannel configuration +kubectl get node k3s-worker-N -o yaml | grep flannel +# Should see flannel.alpha.coreos.com annotations + +# 5. Check pod CIDR assignment +kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}' +# Should return something like 10.244.X.0/24 +``` + +**Expected Results:** +- Node status: `Ready` +- Internal IP: Matches private IP from Hetzner network +- k3s version: Matches existing nodes +- Flannel annotations: Present +- Pod CIDR: Assigned (10.244.X.0/24) + +**Checklist:** +- [ ] Node status is Ready +- [ ] Internal IP is correct (Hetzner private IP) +- [ ] k3s version matches cluster +- [ ] Flannel annotations present +- [ ] Pod CIDR assigned + +--- + +## Step 6: Label the Node (Optional) + +Add standard worker label: + +```bash +kubectl label nodes k3s-worker-N node-role.kubernetes.io/worker=worker +``` + +**Checklist:** +- [ ] Node labeled + +--- + +## Step 7: Monitor Pod Scheduling + +Watch for pods to be scheduled on the new node: + +```bash +# Check overall pod distribution +kubectl get pods --all-namespaces -o wide | grep k3s-worker-N + +# If adding capacity for specific workload (e.g., Stalwart): +kubectl get pod -n stalwart -l app=stalwart -o wide + +# Watch for scheduling events +kubectl get events --sort-by='.lastTimestamp' | grep k3s-worker-N +``` + +**Checklist:** +- [ ] Pods are being scheduled on new node +- [ ] Target workload (if specified) successfully scheduled + +--- + +## Step 8: Update Documentation + +Update the cluster architecture table in `infrastructure/K3S_OPERATIONS.md`: + +1. Get the assigned pod CIDR: + ```bash + kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}' + ``` + +2. Edit `infrastructure/K3S_OPERATIONS.md` and add to the "Node CIDRs" table: + ```markdown + | k3s-worker-N | 10.244.X.0/24 | 10.42.1.Y | | + ``` + +3. Commit the change: + ```bash + cd infrastructure + git add K3S_OPERATIONS.md + git commit -m "docs: add k3s-worker-N to cluster architecture table + + Added new worker node with Pod CIDR 10.244.X.0/24. + Node added to resolve resource constraints for workload scheduling. + + Co-Authored-By: Paperclip " + git push + ``` + +**Checklist:** +- [ ] K3S_OPERATIONS.md updated with new node +- [ ] Changes committed and pushed + +--- + +## Success Criteria + +All checkboxes above should be completed: + +- [ ] VM created with correct specifications +- [ ] Both SSH keys (cto-paperclip, silentmaxx) added +- [ ] Firewall rule configured +- [ ] SSH access verified +- [ ] Provisioning script executed successfully +- [ ] Node status: Ready +- [ ] Internal IP matches Hetzner private IP +- [ ] k3s version matches cluster +- [ ] Flannel annotations present +- [ ] Pod CIDR assigned +- [ ] Pods scheduling on new node +- [ ] Documentation updated + +--- + +## Rollback Procedure + +If issues occur and the node needs to be removed: + +```bash +# 1. Drain the node (move all pods off) +kubectl drain k3s-worker-N --ignore-daemonsets --delete-emptydir-data --timeout=5m + +# 2. Delete from cluster +kubectl delete node k3s-worker-N + +# 3. Remove firewall rule (get rule ID first) +hcloud firewall describe fw-k3s +hcloud firewall delete-rule fw-k3s + +# 4. Delete Hetzner VM +hcloud server delete k3s-worker-N +``` + +--- + +## Troubleshooting + +### Node Not Appearing in kubectl get nodes + +**Check:** +1. Firewall rule is configured (Step 2) +2. k3s-agent service is running on the node: + ```bash + ssh root@ 'systemctl status k3s-agent' + ``` +3. Connectivity to control plane: + ```bash + ssh root@ 'curl -k https://10.42.1.1:6443' + ``` + +### Internal IP Shows Public IP Instead of Private IP + +**Cause:** Node not properly connected to Hetzner private network. + +**Fix:** +1. Verify network attachment in Hetzner Cloud Console +2. Ensure private network interface is up: + ```bash + ssh root@ 'ip addr show' + ``` +3. May need to recreate the VM with network properly attached + +### k3s Version Mismatch + +**Cause:** Provisioning script couldn't detect cluster version, or manual installation was attempted. + +**Fix:** +1. Cordon the node: + ```bash + kubectl cordon k3s-worker-N + ``` +2. Get correct version: + ```bash + CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | sed 's/v//') + ``` +3. Reinstall with correct version: + ```bash + K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token') + ssh root@ "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \ + --server https://10.42.1.1:6443 \ + --token '${K3S_TOKEN}'" + ``` +4. Uncordon: + ```bash + kubectl uncordon k3s-worker-N + ``` + +--- + +## Reference Links + +- [K3S_OPERATIONS.md](./K3S_OPERATIONS.md) - Full operational guide +- [CLUSTER_ACCESS.md](./CLUSTER_ACCESS.md) - Access and authentication +- [Hetzner Cloud Docs](https://docs.hetzner.com/cloud/) +- [k3s Documentation](https://docs.k3s.io/) + +--- + +**Last Updated:** 2026-07-26 +**Maintained By:** CTO Agent