basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs: add standalone worker node addition instruction

Browse source 
Created ADD_WORKER_NODE.md as a step-by-step checklist for adding
new worker nodes to the k3s cluster. This complements the existing
K3S_OPERATIONS.md with a focused, actionable guide.

Key features:
- Pre-flight checklist format
- Hetzner Cloud VM creation steps (console and CLI)
- SSH key requirements (cto-paperclip, silentmaxx)
- Firewall configuration verification
- Provisioning script usage
- Health verification steps
- Documentation update procedure
- Rollback instructions
- Troubleshooting guide

Created for DEV-408 (adding worker node to resolve Stalwart scheduling).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-07-26 16:07:40 +00:00
parent 2a89be6707
commit ae1573d6e8
1 changed files with 345 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

345
infrastructure/ADD_WORKER_NODE.md Normal file
View file

@ -0,0 +1,345 @@
# Adding a Worker Node to k3s Cluster - Quick Instruction
**Purpose:** Step-by-step checklist for adding a new worker node to the BasicStack k3s cluster.
**Audience:** CTO agent, infrastructure operators
**Prerequisites:**
- Hetzner Cloud access (console or hcloud CLI with credentials)
- kubectl access to the cluster
- SSH access to control plane (178.105.17.239)
---
## Step 1: Create Hetzner Cloud VM
### Via Hetzner Cloud Console
1. Navigate to [Hetzner Cloud Console](https://console.hetzner.cloud/)
2. Select the BasicStack project
3. Click "Add Server"
4. Configure:
- **Name:** `k3s-worker-N` (use next available number)
- **Location:** Match existing cluster (check with `kubectl get nodes -o wide`)
- **Image:** Ubuntu 24.04 LTS
- **Type:** CX32 or larger (minimum 4 vCPU, 8 GB RAM)
- **Network:** Attach to existing private network (`10.42.1.0/24`)
- **SSH Keys:** Select both `cto-paperclip` AND `silentmaxx`
- **Firewall:** Attach `fw-k3s`
5. Click "Create & Buy now"
6. **Record the IPs:**
- Private IP: `________________` (from Hetzner network tab)
- Public IP: `________________` (from server overview)
### Via hcloud CLI
```bash
# List available networks
hcloud network list
# List available SSH keys (verify both keys exist)
hcloud ssh-key list | grep -E 'cto-paperclip|silentmaxx'
# Create the server (adjust location as needed)
hcloud server create \
--name k3s-worker-4 \
--type cx32 \
--image ubuntu-24.04 \
--ssh-key cto-paperclip \
--ssh-key silentmaxx \
--network k3s-network \
--firewall fw-k3s \
--location nbg1
# Record the IPs from output
```
---
## Step 2: Verify Firewall Configuration
**Why:** Flannel VXLAN uses public IPs for pod networking tunnels. Without the firewall rule, pods cannot communicate.
```bash
# Check if public IP is in fw-k3s
hcloud firewall describe fw-k3s | grep <public-ip>
# If missing, add the rule:
hcloud firewall add-rule fw-k3s \
--direction in \
--protocol tcp \
--source-ips <public-ip>/32 \
--port any
```
**Checklist:**
- [ ] Firewall rule exists for node's public IP
---
## Step 3: Verify SSH Access
Test connectivity before proceeding:
```bash
# Test public IP
ssh root@<public-ip> 'hostname && date'
# Test private IP (if accessible)
ssh root@<private-ip> 'hostname && date'
```
**Checklist:**
- [ ] SSH access confirmed to public IP
- [ ] SSH access confirmed to private IP
---
## Step 4: Run Provisioning Script
**Location:** `infrastructure/scripts/provision-k3s-worker.sh`
**What it does:**
- Detects current k3s cluster version
- Prompts for firewall confirmation
- Retrieves join token from control plane
- Installs k3s with version pinning
- Verifies successful cluster join
**Execution:**
```bash
cd /path/to/stack.basicstack.de/infrastructure/scripts
# Syntax: ./provision-k3s-worker.sh <private-ip> <public-ip>
./provision-k3s-worker.sh <PRIVATE-IP> <PUBLIC-IP>
# Example:
# ./provision-k3s-worker.sh 10.42.1.6 167.233.121.122
```
**During execution:**
- Confirm firewall update when prompted (press Enter after verifying Step 2)
- Wait for k3s installation (may take 5-10 minutes)
- Review the final node list output
**Checklist:**
- [ ] Script executed successfully
- [ ] No error messages in output
- [ ] Node appears in final `kubectl get nodes` output
---
## Step 5: Verify Node Health
Run these verification commands:
```bash
# 1. Check node status (should be Ready)
kubectl get nodes -o wide
# 2. Verify internal IP matches Hetzner private IP
kubectl get node k3s-worker-N -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'
# Expected: 10.42.1.X (the private IP from Step 1)
# 3. Check k3s version consistency
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
# All nodes should have the same version
# 4. Verify flannel configuration
kubectl get node k3s-worker-N -o yaml | grep flannel
# Should see flannel.alpha.coreos.com annotations
# 5. Check pod CIDR assignment
kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}'
# Should return something like 10.244.X.0/24
```
**Expected Results:**
- Node status: `Ready`
- Internal IP: Matches private IP from Hetzner network
- k3s version: Matches existing nodes
- Flannel annotations: Present
- Pod CIDR: Assigned (10.244.X.0/24)
**Checklist:**
- [ ] Node status is Ready
- [ ] Internal IP is correct (Hetzner private IP)
- [ ] k3s version matches cluster
- [ ] Flannel annotations present
- [ ] Pod CIDR assigned
---
## Step 6: Label the Node (Optional)
Add standard worker label:
```bash
kubectl label nodes k3s-worker-N node-role.kubernetes.io/worker=worker
```
**Checklist:**
- [ ] Node labeled
---
## Step 7: Monitor Pod Scheduling
Watch for pods to be scheduled on the new node:
```bash
# Check overall pod distribution
kubectl get pods --all-namespaces -o wide | grep k3s-worker-N
# If adding capacity for specific workload (e.g., Stalwart):
kubectl get pod -n stalwart -l app=stalwart -o wide
# Watch for scheduling events
kubectl get events --sort-by='.lastTimestamp' | grep k3s-worker-N
```
**Checklist:**
- [ ] Pods are being scheduled on new node
- [ ] Target workload (if specified) successfully scheduled
---
## Step 8: Update Documentation
Update the cluster architecture table in `infrastructure/K3S_OPERATIONS.md`:
1. Get the assigned pod CIDR:
```bash
kubectl get node k3s-worker-N -o jsonpath='{.spec.podCIDR}'
```
2. Edit `infrastructure/K3S_OPERATIONS.md` and add to the "Node CIDRs" table:
```markdown
| k3s-worker-N | 10.244.X.0/24 | 10.42.1.Y | <public-ip> |
```
3. Commit the change:
```bash
cd infrastructure
git add K3S_OPERATIONS.md
git commit -m "docs: add k3s-worker-N to cluster architecture table
Added new worker node with Pod CIDR 10.244.X.0/24.
Node added to resolve resource constraints for workload scheduling.
Co-Authored-By: Paperclip <noreply@paperclip.ing>"
git push
```
**Checklist:**
- [ ] K3S_OPERATIONS.md updated with new node
- [ ] Changes committed and pushed
---
## Success Criteria
All checkboxes above should be completed:
- [ ] VM created with correct specifications
- [ ] Both SSH keys (cto-paperclip, silentmaxx) added
- [ ] Firewall rule configured
- [ ] SSH access verified
- [ ] Provisioning script executed successfully
- [ ] Node status: Ready
- [ ] Internal IP matches Hetzner private IP
- [ ] k3s version matches cluster
- [ ] Flannel annotations present
- [ ] Pod CIDR assigned
- [ ] Pods scheduling on new node
- [ ] Documentation updated
---
## Rollback Procedure
If issues occur and the node needs to be removed:
```bash
# 1. Drain the node (move all pods off)
kubectl drain k3s-worker-N --ignore-daemonsets --delete-emptydir-data --timeout=5m
# 2. Delete from cluster
kubectl delete node k3s-worker-N
# 3. Remove firewall rule (get rule ID first)
hcloud firewall describe fw-k3s
hcloud firewall delete-rule fw-k3s <rule-id>
# 4. Delete Hetzner VM
hcloud server delete k3s-worker-N
```
---
## Troubleshooting
### Node Not Appearing in kubectl get nodes
**Check:**
1. Firewall rule is configured (Step 2)
2. k3s-agent service is running on the node:
```bash
ssh root@<node-ip> 'systemctl status k3s-agent'
```
3. Connectivity to control plane:
```bash
ssh root@<node-ip> 'curl -k https://10.42.1.1:6443'
```
### Internal IP Shows Public IP Instead of Private IP
**Cause:** Node not properly connected to Hetzner private network.
**Fix:**
1. Verify network attachment in Hetzner Cloud Console
2. Ensure private network interface is up:
```bash
ssh root@<node-ip> 'ip addr show'
```
3. May need to recreate the VM with network properly attached
### k3s Version Mismatch
**Cause:** Provisioning script couldn't detect cluster version, or manual installation was attempted.
**Fix:**
1. Cordon the node:
```bash
kubectl cordon k3s-worker-N
```
2. Get correct version:
```bash
CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | sed 's/v//')
```
3. Reinstall with correct version:
```bash
K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token')
ssh root@<node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
--server https://10.42.1.1:6443 \
--token '${K3S_TOKEN}'"
```
4. Uncordon:
```bash
kubectl uncordon k3s-worker-N
```
---
## Reference Links
- [K3S_OPERATIONS.md](./K3S_OPERATIONS.md) - Full operational guide
- [CLUSTER_ACCESS.md](./CLUSTER_ACCESS.md) - Access and authentication
- [Hetzner Cloud Docs](https://docs.hetzner.com/cloud/)
- [k3s Documentation](https://docs.k3s.io/)
---
**Last Updated:** 2026-07-26
**Maintained By:** CTO Agent