From d5e88d884ed93c3b43bee1d7763fe3c38caf046a Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 30 Aug 2026 10:07:25 +0000 Subject: [PATCH] docs(infra): refresh k3s ops doc and add --node-ip to provision script (DEV-410) - provision-k3s-worker.sh now sets --node-ip= on the agent install so k3s does not default to the public IP (DEV-224 root cause) - K3S_OPERATIONS.md node table reflects the current cluster (cp-2/cp-3, worker-4/5/6), documents the Hetzner Cloud Volume datacenter constraint, and calls out the --node-ip requirement in the manual join steps Co-Authored-By: Paperclip --- infrastructure/K3S_OPERATIONS.md | 35 ++++++++++++------- .../scripts/provision-k3s-worker.sh | 4 ++- 2 files changed, 26 insertions(+), 13 deletions(-) diff --git a/infrastructure/K3S_OPERATIONS.md b/infrastructure/K3S_OPERATIONS.md index 741a14e..6ce49ae 100644 --- a/infrastructure/K3S_OPERATIONS.md +++ b/infrastructure/K3S_OPERATIONS.md @@ -3,9 +3,9 @@ ## Current Cluster Status - **Version:** v1.36.2+k3s1 -- **Last Updated:** 2026-07-06 -- **Control Plane:** k3s-cp-1 (178.105.17.239 / 10.42.1.1) -- **Workers:** k3s-worker-1, k3s-worker-2, k3s-worker-3, k3s-update-runner +- **Last Updated:** 2026-08-30 +- **Control Plane:** k3s-cp-1 (178.105.17.239 / 10.42.1.1), k3s-cp-2 (188.245.85.199 / 10.42.1.8), k3s-cp-3 (49.13.92.162 / 10.42.1.9) — etcd HA +- **Workers:** k3s-worker-1, k3s-worker-2, k3s-worker-3, k3s-worker-4 (nbg1), k3s-worker-5, k3s-worker-6, k3s-update-runner - **Update Method:** k3s system-upgrade-controller (automated) ## Quick Reference @@ -79,12 +79,15 @@ The script will: ```bash ssh root@ "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \ --server https://10.42.1.1:6443 \ - --token '${K3S_TOKEN}'" + --token '${K3S_TOKEN}' \ + --node-ip " ``` + **Critical:** The `--node-ip=` flag is required. Without it, k3s defaults to the public IP as the node's InternalIP, which breaks LoadBalancer service routing (see DEV-224). Verify with `kubectl get nodes -o wide` — the INTERNAL-IP column MUST show the private (10.42.1.x) IP. + 5. **Verify node joined:** ```bash - kubectl get nodes + kubectl get nodes -o wide # INTERNAL-IP must be private (10.42.1.x) kubectl get node -o yaml | grep flannel ``` @@ -410,13 +413,20 @@ failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16" ### Node CIDRs -| Node | Pod CIDR | Private IP | Public IP | -|------|----------|-----------|-----------| -| k3s-cp-1 | 10.244.0.0/24 | 10.42.1.1 | 178.105.17.239 | -| k3s-worker-1 | 10.244.1.0/24 | 10.42.1.2 | - | -| k3s-worker-2 | 10.244.2.0/24 | 10.42.1.3 | - | -| k3s-update-runner | 10.244.3.0/24 | 167.233.79.65 | 167.233.79.65 | -| k3s-worker-3 | 10.244.5.0/24 | 10.42.1.5 | 167.233.121.121 | +| Node | Private IP | Public IP | Datacenter | Role | +|------|-----------|-----------|------------|------| +| k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | control-plane, etcd | +| k3s-cp-2 | 10.42.1.8 | 188.245.85.199 | fsn1 | control-plane, etcd | +| k3s-cp-3 | 10.42.1.9 | 49.13.92.162 | fsn1 | control-plane, etcd | +| k3s-worker-1 | 10.42.1.2 | 178.105.216.48 | fsn1 | worker | +| k3s-worker-2 | 10.42.1.3 | 49.13.134.255 | fsn1 | worker | +| k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | worker | +| k3s-worker-4 | 10.42.1.6 | 128.140.3.80 | nbg1 | worker (stateless workloads only) | +| k3s-worker-5 | 10.42.1.7 | 167.233.192.86 | fsn1 | worker (DEV-410) | +| k3s-worker-6 | 10.42.1.10 | 49.12.223.133 | fsn1 | worker (cx33 4vCPU/8GB) | +| k3s-update-runner | 167.233.79.65 | 167.233.79.65 | fsn1 | update-runner | + +**Datacenter note:** Hetzner Cloud Volumes are datacenter-scoped. Pods with PersistentVolumes only schedule on nodes in the same datacenter as the PV. Provision stateful-workload workers in **fsn1** to match the existing PV pool. k3s-worker-4 remains in nbg1 and is only usable for stateless workloads. ### Critical Firewall Rule @@ -428,6 +438,7 @@ failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16" | Date | Version | Changes | By | |------|---------|---------|-----| +| 2026-08-30 | v1.36.2+k3s1 | DEV-410: Added k3s-worker-5 (fsn1) to unblock Stalwart scheduling; k3s-worker-4 (nbg1) retained for stateless workloads; provision script now sets --node-ip; refreshed node table with cp-2/cp-3 and worker-4/5/6 | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Fixed DNS service CIDR mismatch (kubelet cluster-dns configuration), corrected Service CIDR documentation to 10.96.0.0/16, added DNS configuration guide | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1 | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent | diff --git a/infrastructure/scripts/provision-k3s-worker.sh b/infrastructure/scripts/provision-k3s-worker.sh index b7479db..2cda77e 100755 --- a/infrastructure/scripts/provision-k3s-worker.sh +++ b/infrastructure/scripts/provision-k3s-worker.sh @@ -82,7 +82,8 @@ echo "This may take a few minutes..." ssh root@${NEW_NODE_IP} "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \ --server https://${CONTROL_PLANE_IP}:6443 \ - --token '${K3S_TOKEN}'" + --token '${K3S_TOKEN}' \ + --node-ip ${NEW_NODE_IP}" if [ $? -ne 0 ]; then echo -e "${RED}Error: k3s installation failed${NC}" @@ -111,3 +112,4 @@ echo "Next steps:" echo " 1. Label the node if needed: kubectl label nodes node-role.kubernetes.io/worker=worker" echo " 2. Monitor pod scheduling: kubectl get pods --all-namespaces -o wide | grep " echo " 3. Verify flannel annotation: kubectl get node -o yaml | grep flannel" +echo " 4. Verify INTERNAL-IP is the private IP (not public): kubectl get node -o wide"