# Adding a Worker Node to k3s Cluster - Quick Instruction

**Purpose:** Step-by-step checklist for adding a new worker node to the BasicStack k3s cluster.

**Audience:** CTO agent, infrastructure operators

**Prerequisites:**
- Hetzner Cloud access (console or hcloud CLI with credentials)
- kubectl access to the cluster
- SSH access to control plane (178.105.17.239)

---

## Step 1: Create Hetzner Cloud VM

### Via Hetzner Cloud Console

1. Navigate to [Hetzner Cloud Console](https://console.hetzner.cloud/)
2. Select the BasicStack project
3. Click "Add Server"
4. Configure:
   - **Name:** `k3s-worker-N` (use next available number)
   - **Location:** Match existing cluster (check with `kubectl get nodes -o wide`)
   - **Image:** Ubuntu 24.04 LTS
   - **Type:** CX32 or larger (minimum 4 vCPU, 8 GB RAM)
   - **Network:** Attach to existing private network (`10.42.1.0/24`)
   - **SSH Keys:** Select both `cto-paperclip` AND `silentmaxx`
   - **Firewall:** Attach `fw-k3s`
5. Click "Create & Buy now"
6. **Record the IPs:**
   - Private IP: `________________` (from Hetzner network tab)
   - Public IP: `________________` (from server overview)

### Via hcloud CLI

```bash
# List available networks
hcloud network list

# List available SSH keys (verify both keys exist)
hcloud ssh-key list | grep -E 'cto-paperclip|silentmaxx'

# Create the server (adjust location as needed)
hcloud server create \
  --name k3s-worker-4 \
  --type cx32 \
  --image ubuntu-24.04 \
  --ssh-key cto-paperclip \
  --ssh-key silentmaxx \
  --network k3s-network \
  --firewall fw-k3s \
  --location nbg1

# Record the IPs from output
```

---

## Step 2: Verify Firewall Configuration

**Why:** Flannel VXLAN uses public IPs for pod networking tunnels. Without the firewall rule, pods cannot communicate.

```bash
# Check if public IP is in fw-k3s
hcloud firewall describe fw-k3s | grep <public-ip>

# If missing, add the rule:
hcloud firewall add-rule fw-k3s \
  --direction in \
  --protocol tcp \
  --source-ips <public-ip>/32 \
  --port any
```

**Checklist:**
- [ ] Firewall rule exists for node's public IP

---

## Step 3: Verify SSH Access

Test connectivity before proceeding:

```bash
# Test public IP
ssh root@<public-ip> 'hostname && date'

# Test private IP (if accessible)
ssh root@<private-ip> 'hostname && date'
```

**Checklist:**
- [ ] SSH access confirmed to public IP
- [ ] SSH access confirmed to private IP

---

## Step 4: Run Provisioning Script

**Location:** `infrastructure/scripts/provision-k3s-worker.sh`

**What it does:**
- Detects current k3s cluster version
- Prompts for firewall confirmation
- Retrieves join token from control plane
- Installs k3s with version pinning
- Verifies successful cluster join

**Execution:**

```bash
cd /path/to/stack.basicstack.de/infrastructure/scripts

# Syntax: ./provision-k3s-worker.sh <private-ip> <public-ip>
./provision-k3s-worker.sh <PRIVATE-IP> <PUBLIC-IP>

# Example:
# ./provision-k3s-worker.sh 10.42.1.6 167.233.121.122
```

**During execution:**
- Confirm firewall update when prompted (press Enter after verifying Step 2)
- Wait for k3s installation (may take 5-10 minutes)
- Review the final node list output

**Checklist:**
- [ ] Script executed successfully
- [ ] No error messages in output
- [ ] Node appears in final `kubectl get nodes` output

---

## Step 5: Verify Node Health

Run these verification commands:

```bash
# 1. Check node status (should be Ready)
kubectl get nodes -o wide

# 2. Verify internal IP matches Hetzner private IP
kubectl get node k3s-worker-N -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'
# Expected: 10.42.1.X (the private IP from Step 1)

# 3. Check k3s version consistency
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
# All nodes should have the same version

# 4. Verify flannel configuration
kubectl get node k3s-worker-N -o yaml | grep flannel
# Should see flannel.alpha.coreos.com annotations

# 5. Check pod CIDR assignment
kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}'
# Should return something like 10.244.X.0/24
```

**Expected Results:**
- Node status: `Ready`
- Internal IP: Matches private IP from Hetzner network
- k3s version: Matches existing nodes
- Flannel annotations: Present
- Pod CIDR: Assigned (10.244.X.0/24)

**Checklist:**
- [ ] Node status is Ready
- [ ] Internal IP is correct (Hetzner private IP)
- [ ] k3s version matches cluster
- [ ] Flannel annotations present
- [ ] Pod CIDR assigned

---

## Step 6: Label the Node (Optional)

Add standard worker label:

```bash
kubectl label nodes k3s-worker-N node-role.kubernetes.io/worker=worker
```

**Checklist:**
- [ ] Node labeled

---

## Step 7: Monitor Pod Scheduling

Watch for pods to be scheduled on the new node:

```bash
# Check overall pod distribution
kubectl get pods --all-namespaces -o wide | grep k3s-worker-N

# If adding capacity for specific workload (e.g., Stalwart):
kubectl get pod -n stalwart -l app=stalwart -o wide

# Watch for scheduling events
kubectl get events --sort-by='.lastTimestamp' | grep k3s-worker-N
```

**Checklist:**
- [ ] Pods are being scheduled on new node
- [ ] Target workload (if specified) successfully scheduled

---

## Step 8: Update Documentation

Update the cluster architecture table in `infrastructure/K3S_OPERATIONS.md`:

1. Get the assigned pod CIDR:
   ```bash
   kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}'
   ```

2. Edit `infrastructure/K3S_OPERATIONS.md` and add to the "Node CIDRs" table:
   ```markdown
   | k3s-worker-N | 10.244.X.0/24 | 10.42.1.Y | <public-ip> |
   ```

3. Commit the change:
   ```bash
   cd infrastructure
   git add K3S_OPERATIONS.md
   git commit -m "docs: add k3s-worker-N to cluster architecture table

   Added new worker node with Pod CIDR 10.244.X.0/24.
   Node added to resolve resource constraints for workload scheduling.

   Co-Authored-By: Paperclip <noreply@paperclip.ing>"
   git push
   ```

**Checklist:**
- [ ] K3S_OPERATIONS.md updated with new node
- [ ] Changes committed and pushed

---

## Success Criteria

All checkboxes above should be completed:

- [ ] VM created with correct specifications
- [ ] Both SSH keys (cto-paperclip, silentmaxx) added
- [ ] Firewall rule configured
- [ ] SSH access verified
- [ ] Provisioning script executed successfully
- [ ] Node status: Ready
- [ ] Internal IP matches Hetzner private IP
- [ ] k3s version matches cluster
- [ ] Flannel annotations present
- [ ] Pod CIDR assigned
- [ ] Pods scheduling on new node
- [ ] Documentation updated

---

## Rollback Procedure

If issues occur and the node needs to be removed:

```bash
# 1. Drain the node (move all pods off)
kubectl drain k3s-worker-N --ignore-daemonsets --delete-emptydir-data --timeout=5m

# 2. Delete from cluster
kubectl delete node k3s-worker-N

# 3. Remove firewall rule (get rule ID first)
hcloud firewall describe fw-k3s
hcloud firewall delete-rule fw-k3s <rule-id>

# 4. Delete Hetzner VM
hcloud server delete k3s-worker-N
```

---

## Troubleshooting

### Node Not Appearing in kubectl get nodes

**Check:**
1. Firewall rule is configured (Step 2)
2. k3s-agent service is running on the node:
   ```bash
   ssh root@<node-ip> 'systemctl status k3s-agent'
   ```
3. Connectivity to control plane:
   ```bash
   ssh root@<node-ip> 'curl -k https://10.42.1.1:6443'
   ```

### Internal IP Shows Public IP Instead of Private IP

**Cause:** Node not properly connected to Hetzner private network.

**Fix:**
1. Verify network attachment in Hetzner Cloud Console
2. Ensure private network interface is up:
   ```bash
   ssh root@<node-ip> 'ip addr show'
   ```
3. May need to recreate the VM with network properly attached

### k3s Version Mismatch

**Cause:** Provisioning script couldn't detect cluster version, or manual installation was attempted.

**Fix:**
1. Cordon the node:
   ```bash
   kubectl cordon k3s-worker-N
   ```
2. Get correct version:
   ```bash
   CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | sed 's/v//')
   ```
3. Reinstall with correct version:
   ```bash
   K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token')
   ssh root@<node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
     --server https://10.42.1.1:6443 \
     --token '${K3S_TOKEN}'"
   ```
4. Uncordon:
   ```bash
   kubectl uncordon k3s-worker-N
   ```

---

## Reference Links

- [K3S_OPERATIONS.md](./K3S_OPERATIONS.md) - Full operational guide
- [CLUSTER_ACCESS.md](./CLUSTER_ACCESS.md) - Access and authentication
- [Hetzner Cloud Docs](https://docs.hetzner.com/cloud/)
- [k3s Documentation](https://docs.k3s.io/)

---

**Last Updated:** 2026-07-26
**Maintained By:** CTO Agent