docs: add standalone worker node addition instruction
Created ADD_WORKER_NODE.md as a step-by-step checklist for adding new worker nodes to the k3s cluster. This complements the existing K3S_OPERATIONS.md with a focused, actionable guide. Key features: - Pre-flight checklist format - Hetzner Cloud VM creation steps (console and CLI) - SSH key requirements (cto-paperclip, silentmaxx) - Firewall configuration verification - Provisioning script usage - Health verification steps - Documentation update procedure - Rollback instructions - Troubleshooting guide Created for DEV-408 (adding worker node to resolve Stalwart scheduling). Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
2a89be6707
commit
ae1573d6e8
1 changed files with 345 additions and 0 deletions
345
infrastructure/ADD_WORKER_NODE.md
Normal file
345
infrastructure/ADD_WORKER_NODE.md
Normal file
|
|
@ -0,0 +1,345 @@
|
|||
# Adding a Worker Node to k3s Cluster - Quick Instruction
|
||||
|
||||
**Purpose:** Step-by-step checklist for adding a new worker node to the BasicStack k3s cluster.
|
||||
|
||||
**Audience:** CTO agent, infrastructure operators
|
||||
|
||||
**Prerequisites:**
|
||||
- Hetzner Cloud access (console or hcloud CLI with credentials)
|
||||
- kubectl access to the cluster
|
||||
- SSH access to control plane (178.105.17.239)
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Create Hetzner Cloud VM
|
||||
|
||||
### Via Hetzner Cloud Console
|
||||
|
||||
1. Navigate to [Hetzner Cloud Console](https://console.hetzner.cloud/)
|
||||
2. Select the BasicStack project
|
||||
3. Click "Add Server"
|
||||
4. Configure:
|
||||
- **Name:** `k3s-worker-N` (use next available number)
|
||||
- **Location:** Match existing cluster (check with `kubectl get nodes -o wide`)
|
||||
- **Image:** Ubuntu 24.04 LTS
|
||||
- **Type:** CX32 or larger (minimum 4 vCPU, 8 GB RAM)
|
||||
- **Network:** Attach to existing private network (`10.42.1.0/24`)
|
||||
- **SSH Keys:** Select both `cto-paperclip` AND `silentmaxx`
|
||||
- **Firewall:** Attach `fw-k3s`
|
||||
5. Click "Create & Buy now"
|
||||
6. **Record the IPs:**
|
||||
- Private IP: `________________` (from Hetzner network tab)
|
||||
- Public IP: `________________` (from server overview)
|
||||
|
||||
### Via hcloud CLI
|
||||
|
||||
```bash
|
||||
# List available networks
|
||||
hcloud network list
|
||||
|
||||
# List available SSH keys (verify both keys exist)
|
||||
hcloud ssh-key list | grep -E 'cto-paperclip|silentmaxx'
|
||||
|
||||
# Create the server (adjust location as needed)
|
||||
hcloud server create \
|
||||
--name k3s-worker-4 \
|
||||
--type cx32 \
|
||||
--image ubuntu-24.04 \
|
||||
--ssh-key cto-paperclip \
|
||||
--ssh-key silentmaxx \
|
||||
--network k3s-network \
|
||||
--firewall fw-k3s \
|
||||
--location nbg1
|
||||
|
||||
# Record the IPs from output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Verify Firewall Configuration
|
||||
|
||||
**Why:** Flannel VXLAN uses public IPs for pod networking tunnels. Without the firewall rule, pods cannot communicate.
|
||||
|
||||
```bash
|
||||
# Check if public IP is in fw-k3s
|
||||
hcloud firewall describe fw-k3s | grep <public-ip>
|
||||
|
||||
# If missing, add the rule:
|
||||
hcloud firewall add-rule fw-k3s \
|
||||
--direction in \
|
||||
--protocol tcp \
|
||||
--source-ips <public-ip>/32 \
|
||||
--port any
|
||||
```
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Firewall rule exists for node's public IP
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Verify SSH Access
|
||||
|
||||
Test connectivity before proceeding:
|
||||
|
||||
```bash
|
||||
# Test public IP
|
||||
ssh root@<public-ip> 'hostname && date'
|
||||
|
||||
# Test private IP (if accessible)
|
||||
ssh root@<private-ip> 'hostname && date'
|
||||
```
|
||||
|
||||
**Checklist:**
|
||||
- [ ] SSH access confirmed to public IP
|
||||
- [ ] SSH access confirmed to private IP
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Run Provisioning Script
|
||||
|
||||
**Location:** `infrastructure/scripts/provision-k3s-worker.sh`
|
||||
|
||||
**What it does:**
|
||||
- Detects current k3s cluster version
|
||||
- Prompts for firewall confirmation
|
||||
- Retrieves join token from control plane
|
||||
- Installs k3s with version pinning
|
||||
- Verifies successful cluster join
|
||||
|
||||
**Execution:**
|
||||
|
||||
```bash
|
||||
cd /path/to/stack.basicstack.de/infrastructure/scripts
|
||||
|
||||
# Syntax: ./provision-k3s-worker.sh <private-ip> <public-ip>
|
||||
./provision-k3s-worker.sh <PRIVATE-IP> <PUBLIC-IP>
|
||||
|
||||
# Example:
|
||||
# ./provision-k3s-worker.sh 10.42.1.6 167.233.121.122
|
||||
```
|
||||
|
||||
**During execution:**
|
||||
- Confirm firewall update when prompted (press Enter after verifying Step 2)
|
||||
- Wait for k3s installation (may take 5-10 minutes)
|
||||
- Review the final node list output
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Script executed successfully
|
||||
- [ ] No error messages in output
|
||||
- [ ] Node appears in final `kubectl get nodes` output
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Verify Node Health
|
||||
|
||||
Run these verification commands:
|
||||
|
||||
```bash
|
||||
# 1. Check node status (should be Ready)
|
||||
kubectl get nodes -o wide
|
||||
|
||||
# 2. Verify internal IP matches Hetzner private IP
|
||||
kubectl get node k3s-worker-N -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'
|
||||
# Expected: 10.42.1.X (the private IP from Step 1)
|
||||
|
||||
# 3. Check k3s version consistency
|
||||
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
|
||||
# All nodes should have the same version
|
||||
|
||||
# 4. Verify flannel configuration
|
||||
kubectl get node k3s-worker-N -o yaml | grep flannel
|
||||
# Should see flannel.alpha.coreos.com annotations
|
||||
|
||||
# 5. Check pod CIDR assignment
|
||||
kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}'
|
||||
# Should return something like 10.244.X.0/24
|
||||
```
|
||||
|
||||
**Expected Results:**
|
||||
- Node status: `Ready`
|
||||
- Internal IP: Matches private IP from Hetzner network
|
||||
- k3s version: Matches existing nodes
|
||||
- Flannel annotations: Present
|
||||
- Pod CIDR: Assigned (10.244.X.0/24)
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Node status is Ready
|
||||
- [ ] Internal IP is correct (Hetzner private IP)
|
||||
- [ ] k3s version matches cluster
|
||||
- [ ] Flannel annotations present
|
||||
- [ ] Pod CIDR assigned
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Label the Node (Optional)
|
||||
|
||||
Add standard worker label:
|
||||
|
||||
```bash
|
||||
kubectl label nodes k3s-worker-N node-role.kubernetes.io/worker=worker
|
||||
```
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Node labeled
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Monitor Pod Scheduling
|
||||
|
||||
Watch for pods to be scheduled on the new node:
|
||||
|
||||
```bash
|
||||
# Check overall pod distribution
|
||||
kubectl get pods --all-namespaces -o wide | grep k3s-worker-N
|
||||
|
||||
# If adding capacity for specific workload (e.g., Stalwart):
|
||||
kubectl get pod -n stalwart -l app=stalwart -o wide
|
||||
|
||||
# Watch for scheduling events
|
||||
kubectl get events --sort-by='.lastTimestamp' | grep k3s-worker-N
|
||||
```
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Pods are being scheduled on new node
|
||||
- [ ] Target workload (if specified) successfully scheduled
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Update Documentation
|
||||
|
||||
Update the cluster architecture table in `infrastructure/K3S_OPERATIONS.md`:
|
||||
|
||||
1. Get the assigned pod CIDR:
|
||||
```bash
|
||||
kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}'
|
||||
```
|
||||
|
||||
2. Edit `infrastructure/K3S_OPERATIONS.md` and add to the "Node CIDRs" table:
|
||||
```markdown
|
||||
| k3s-worker-N | 10.244.X.0/24 | 10.42.1.Y | <public-ip> |
|
||||
```
|
||||
|
||||
3. Commit the change:
|
||||
```bash
|
||||
cd infrastructure
|
||||
git add K3S_OPERATIONS.md
|
||||
git commit -m "docs: add k3s-worker-N to cluster architecture table
|
||||
|
||||
Added new worker node with Pod CIDR 10.244.X.0/24.
|
||||
Node added to resolve resource constraints for workload scheduling.
|
||||
|
||||
Co-Authored-By: Paperclip <noreply@paperclip.ing>"
|
||||
git push
|
||||
```
|
||||
|
||||
**Checklist:**
|
||||
- [ ] K3S_OPERATIONS.md updated with new node
|
||||
- [ ] Changes committed and pushed
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
All checkboxes above should be completed:
|
||||
|
||||
- [ ] VM created with correct specifications
|
||||
- [ ] Both SSH keys (cto-paperclip, silentmaxx) added
|
||||
- [ ] Firewall rule configured
|
||||
- [ ] SSH access verified
|
||||
- [ ] Provisioning script executed successfully
|
||||
- [ ] Node status: Ready
|
||||
- [ ] Internal IP matches Hetzner private IP
|
||||
- [ ] k3s version matches cluster
|
||||
- [ ] Flannel annotations present
|
||||
- [ ] Pod CIDR assigned
|
||||
- [ ] Pods scheduling on new node
|
||||
- [ ] Documentation updated
|
||||
|
||||
---
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If issues occur and the node needs to be removed:
|
||||
|
||||
```bash
|
||||
# 1. Drain the node (move all pods off)
|
||||
kubectl drain k3s-worker-N --ignore-daemonsets --delete-emptydir-data --timeout=5m
|
||||
|
||||
# 2. Delete from cluster
|
||||
kubectl delete node k3s-worker-N
|
||||
|
||||
# 3. Remove firewall rule (get rule ID first)
|
||||
hcloud firewall describe fw-k3s
|
||||
hcloud firewall delete-rule fw-k3s <rule-id>
|
||||
|
||||
# 4. Delete Hetzner VM
|
||||
hcloud server delete k3s-worker-N
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Node Not Appearing in kubectl get nodes
|
||||
|
||||
**Check:**
|
||||
1. Firewall rule is configured (Step 2)
|
||||
2. k3s-agent service is running on the node:
|
||||
```bash
|
||||
ssh root@<node-ip> 'systemctl status k3s-agent'
|
||||
```
|
||||
3. Connectivity to control plane:
|
||||
```bash
|
||||
ssh root@<node-ip> 'curl -k https://10.42.1.1:6443'
|
||||
```
|
||||
|
||||
### Internal IP Shows Public IP Instead of Private IP
|
||||
|
||||
**Cause:** Node not properly connected to Hetzner private network.
|
||||
|
||||
**Fix:**
|
||||
1. Verify network attachment in Hetzner Cloud Console
|
||||
2. Ensure private network interface is up:
|
||||
```bash
|
||||
ssh root@<node-ip> 'ip addr show'
|
||||
```
|
||||
3. May need to recreate the VM with network properly attached
|
||||
|
||||
### k3s Version Mismatch
|
||||
|
||||
**Cause:** Provisioning script couldn't detect cluster version, or manual installation was attempted.
|
||||
|
||||
**Fix:**
|
||||
1. Cordon the node:
|
||||
```bash
|
||||
kubectl cordon k3s-worker-N
|
||||
```
|
||||
2. Get correct version:
|
||||
```bash
|
||||
CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | sed 's/v//')
|
||||
```
|
||||
3. Reinstall with correct version:
|
||||
```bash
|
||||
K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token')
|
||||
ssh root@<node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
|
||||
--server https://10.42.1.1:6443 \
|
||||
--token '${K3S_TOKEN}'"
|
||||
```
|
||||
4. Uncordon:
|
||||
```bash
|
||||
kubectl uncordon k3s-worker-N
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Reference Links
|
||||
|
||||
- [K3S_OPERATIONS.md](./K3S_OPERATIONS.md) - Full operational guide
|
||||
- [CLUSTER_ACCESS.md](./CLUSTER_ACCESS.md) - Access and authentication
|
||||
- [Hetzner Cloud Docs](https://docs.hetzner.com/cloud/)
|
||||
- [k3s Documentation](https://docs.k3s.io/)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-07-26
|
||||
**Maintained By:** CTO Agent
|
||||
Loading…
Add table
Reference in a new issue