basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

docs(infra): refresh k3s ops doc and add --node-ip to provision script (DEV-410)

Browse source 
- provision-k3s-worker.sh now sets --node-ip=<private-ip> on the agent
  install so k3s does not default to the public IP (DEV-224 root cause)
- K3S_OPERATIONS.md node table reflects the current cluster (cp-2/cp-3,
  worker-4/5/6), documents the Hetzner Cloud Volume datacenter constraint,
  and calls out the --node-ip requirement in the manual join steps

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-08-30 10:07:25 +00:00
parent a30274d1eb
commit d5e88d884e
2 changed files with 26 additions and 13 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

35
infrastructure/K3S_OPERATIONS.md
View file

@ -3,9 +3,9 @@
## Current Cluster Status ## Current Cluster Status
- **Version:** v1.36.2+k3s1 - **Version:** v1.36.2+k3s1
- **Last Updated:** 2026-07-06 - **Last Updated:** 2026-08-30
- **Control Plane:** k3s-cp-1 (178.105.17.239 / 10.42.1.1) - **Control Plane:** k3s-cp-1 (178.105.17.239 / 10.42.1.1), k3s-cp-2 (188.245.85.199 / 10.42.1.8), k3s-cp-3 (49.13.92.162 / 10.42.1.9) — etcd HA
- **Workers:** k3s-worker-1, k3s-worker-2, k3s-worker-3, k3s-update-runner - **Workers:** k3s-worker-1, k3s-worker-2, k3s-worker-3, k3s-worker-4 (nbg1), k3s-worker-5, k3s-worker-6, k3s-update-runner
- **Update Method:** k3s system-upgrade-controller (automated) - **Update Method:** k3s system-upgrade-controller (automated)
## Quick Reference ## Quick Reference
@ -79,12 +79,15 @@ The script will:
```bash ```bash
ssh root@<new-node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \ ssh root@<new-node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
--server https://10.42.1.1:6443 \ --server https://10.42.1.1:6443 \
--token '${K3S_TOKEN}'" --token '${K3S_TOKEN}' \
--node-ip <new-node-private-ip>"
``` ```
**Critical:** The `--node-ip=<private-ip>` flag is required. Without it, k3s defaults to the public IP as the node's InternalIP, which breaks LoadBalancer service routing (see DEV-224). Verify with `kubectl get nodes -o wide` — the INTERNAL-IP column MUST show the private (10.42.1.x) IP.
5. **Verify node joined:** 5. **Verify node joined:**
```bash ```bash
kubectl get nodes kubectl get nodes -o wide # INTERNAL-IP must be private (10.42.1.x)
kubectl get node <node-name> -o yaml | grep flannel kubectl get node <node-name> -o yaml | grep flannel
``` ```
@ -410,13 +413,20 @@ failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16"
### Node CIDRs ### Node CIDRs
| Node | Pod CIDR | Private IP | Public IP | | Node | Private IP | Public IP | Datacenter | Role |
|------|----------|-----------|-----------| |------|-----------|-----------|------------|------|
| k3s-cp-1 | 10.244.0.0/24 | 10.42.1.1 | 178.105.17.239 | | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | control-plane, etcd |
| k3s-worker-1 | 10.244.1.0/24 | 10.42.1.2 | - | | k3s-cp-2 | 10.42.1.8 | 188.245.85.199 | fsn1 | control-plane, etcd |
| k3s-worker-2 | 10.244.2.0/24 | 10.42.1.3 | - | | k3s-cp-3 | 10.42.1.9 | 49.13.92.162 | fsn1 | control-plane, etcd |
| k3s-update-runner | 10.244.3.0/24 | 167.233.79.65 | 167.233.79.65 | | k3s-worker-1 | 10.42.1.2 | 178.105.216.48 | fsn1 | worker |
| k3s-worker-3 | 10.244.5.0/24 | 10.42.1.5 | 167.233.121.121 | | k3s-worker-2 | 10.42.1.3 | 49.13.134.255 | fsn1 | worker |
| k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | worker |
| k3s-worker-4 | 10.42.1.6 | 128.140.3.80 | nbg1 | worker (stateless workloads only) |
| k3s-worker-5 | 10.42.1.7 | 167.233.192.86 | fsn1 | worker (DEV-410) |
| k3s-worker-6 | 10.42.1.10 | 49.12.223.133 | fsn1 | worker (cx33 4vCPU/8GB) |
| k3s-update-runner | 167.233.79.65 | 167.233.79.65 | fsn1 | update-runner |
**Datacenter note:** Hetzner Cloud Volumes are datacenter-scoped. Pods with PersistentVolumes only schedule on nodes in the same datacenter as the PV. Provision stateful-workload workers in **fsn1** to match the existing PV pool. k3s-worker-4 remains in nbg1 and is only usable for stateless workloads.
### Critical Firewall Rule ### Critical Firewall Rule
@ -428,6 +438,7 @@ failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16"
| Date | Version | Changes | By | | Date | Version | Changes | By |
|------|---------|---------|-----| |------|---------|---------|-----|
| 2026-08-30 | v1.36.2+k3s1 | DEV-410: Added k3s-worker-5 (fsn1) to unblock Stalwart scheduling; k3s-worker-4 (nbg1) retained for stateless workloads; provision script now sets --node-ip; refreshed node table with cp-2/cp-3 and worker-4/5/6 | CTO Agent |
| 2026-07-06 | v1.36.2+k3s1 | Fixed DNS service CIDR mismatch (kubelet cluster-dns configuration), corrected Service CIDR documentation to 10.96.0.0/16, added DNS configuration guide | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Fixed DNS service CIDR mismatch (kubelet cluster-dns configuration), corrected Service CIDR documentation to 10.96.0.0/16, added DNS configuration guide | CTO Agent |
| 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1 | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1 | CTO Agent |
| 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent |

4
infrastructure/scripts/provision-k3s-worker.sh
View file

@ -82,7 +82,8 @@ echo "This may take a few minutes..."
ssh root@${NEW_NODE_IP} "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \ ssh root@${NEW_NODE_IP} "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
--server https://${CONTROL_PLANE_IP}:6443 \ --server https://${CONTROL_PLANE_IP}:6443 \
--token '${K3S_TOKEN}'" --token '${K3S_TOKEN}' \
--node-ip ${NEW_NODE_IP}"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo -e "${RED}Error: k3s installation failed${NC}" echo -e "${RED}Error: k3s installation failed${NC}"
@ -111,3 +112,4 @@ echo "Next steps:"
echo " 1. Label the node if needed: kubectl label nodes <node-name> node-role.kubernetes.io/worker=worker" echo " 1. Label the node if needed: kubectl label nodes <node-name> node-role.kubernetes.io/worker=worker"
echo " 2. Monitor pod scheduling: kubectl get pods --all-namespaces -o wide | grep <node-name>" echo " 2. Monitor pod scheduling: kubectl get pods --all-namespaces -o wide | grep <node-name>"
echo " 3. Verify flannel annotation: kubectl get node <node-name> -o yaml | grep flannel" echo " 3. Verify flannel annotation: kubectl get node <node-name> -o yaml | grep flannel"
echo " 4. Verify INTERNAL-IP is the private IP (not public): kubectl get node <node-name> -o wide"