basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Add k3s cluster management automation and documentation

Browse source 
- Add k3s node provisioning script with version pinning
- Add comprehensive K3S_OPERATIONS.md documentation
- Add k3s system-upgrade-controller configuration

This addresses DEV-221: prevents version skew issues by:
1. Enforcing version pinning when adding new nodes
2. Providing automated provisioning script
3. Setting up automated upgrades via upgrade controller
4. Documenting all cluster operations procedures

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-07-06 17:08:21 +00:00
parent 07e2e476ce
commit 2c0e7f22b1
5 changed files with 709 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

386
infrastructure/K3S_OPERATIONS.md Normal file
View file

@ -0,0 +1,386 @@
# k3s Cluster Operations
## Current Cluster Status
- **Version:** v1.36.2+k3s1
- **Last Updated:** 2026-07-06
- **Control Plane:** k3s-cp-1 (178.105.17.239 / 10.42.1.1)
- **Workers:** k3s-worker-1, k3s-worker-2, k3s-worker-3, k3s-update-runner
- **Update Method:** k3s system-upgrade-controller (automated)
## Quick Reference
```bash
# Check cluster version
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
# Check for version skew
VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u | wc -l)
if [ "$VERSIONS" -gt 1 ]; then echo "⚠️ Version skew detected!"; fi
# View current cluster state
kubectl get nodes -o wide
```
---
## Adding a New Worker Node
### Prerequisites
1. Hetzner Cloud server running Ubuntu 24.04 LTS
2. SSH access configured
3. `hcloud` CLI installed (for firewall management)
### Procedure
Use the automated provisioning script:
```bash
cd infrastructure/scripts
./provision-k3s-worker.sh <new-node-private-ip> <new-node-public-ip>
```
Example:
```bash
./provision-k3s-worker.sh 10.42.1.6 167.233.121.122
```
The script will:
1. Check current cluster version
2. Prompt for Hetzner firewall update
3. Verify SSH access
4. Install k3s with matching version
5. Verify node joined successfully
### Manual Steps (if script unavailable)
1. **Get cluster version:**
```bash
CLUSTER_VERSION=$(ssh root@k3s-cp-1 'k3s --version' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+')
echo "Cluster version: v${CLUSTER_VERSION}"
```
2. **Update Hetzner firewall `fw-k3s`:**
```bash
# Add new node's public IP to firewall rules
hcloud firewall add-rule fw-k3s --direction in --protocol tcp \
--source-ips <new-node-public-ip>/32 --port any
```
**Critical:** Flannel VXLAN uses public IPs for tunnels. If the public IP is not in the firewall, pods on that node cannot reach DNS or other pods.
3. **Get K3S join token:**
```bash
K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token')
```
4. **Install k3s on new node with version pinning:**
```bash
ssh root@<new-node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
--server https://10.42.1.1:6443 \
--token '${K3S_TOKEN}'"
```
5. **Verify node joined:**
```bash
kubectl get nodes
kubectl get node <node-name> -o yaml | grep flannel
```
---
## Upgrading the Cluster
### Upgrade Strategy
**We use k3s system-upgrade-controller for coordinated, automated upgrades.**
The upgrade controller:
- Ensures version consistency across the cluster
- Performs rolling upgrades (one node at a time)
- Respects pod disruption budgets
- Automatically drains and cordons nodes during upgrade
- Prevents version skew
### Planning an Upgrade
1. **Review k3s release notes:**
- Visit: https://github.com/k3s-io/k3s/releases
- Check for breaking changes
- Note any special upgrade instructions
2. **Check current version:**
```bash
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
```
3. **Schedule maintenance window** (optional for minor upgrades)
### Triggering an Upgrade
The system-upgrade-controller watches for Plan updates. To upgrade:
1. **Edit the upgrade plan:**
```bash
kubectl edit plan k3s-server -n system-upgrade
kubectl edit plan k3s-agent -n system-upgrade
```
2. **Update the version field:**
```yaml
spec:
version: v1.37.0+k3s1 # Change to desired version
```
3. **Monitor the upgrade:**
```bash
# Watch nodes being upgraded
kubectl get nodes -w
# Check upgrade jobs
kubectl get jobs -n system-upgrade
# View upgrade controller logs
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller -f
```
4. **Verify completion:**
```bash
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
kubectl get pods --all-namespaces -o wide
```
### Manual Upgrade (Emergency Only)
If the upgrade controller is unavailable:
#### Control Plane
```bash
ssh root@178.105.17.239
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh -
systemctl restart k3s
# Wait 5 minutes for control plane to stabilize
```
#### Worker Nodes (one at a time)
```bash
# Drain node
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data
# Upgrade node
ssh root@<worker-node>
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh -s - agent
systemctl restart k3s-agent
# Verify and uncordon
kubectl get nodes
kubectl uncordon <worker-node>
# Wait for pods to stabilize before proceeding to next node
kubectl get pods --all-namespaces -o wide | grep <worker-node>
```
### Emergency Rollback
If issues occur after upgrade:
```bash
# On affected node
ssh root@<node>
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.36.2+k3s1" sh -s - agent
systemctl restart k3s-agent
```
Then investigate the issue before retrying the upgrade.
---
## k3s System-Upgrade-Controller
### Overview
The system-upgrade-controller manages automated k3s upgrades across the cluster.
- **Namespace:** `system-upgrade`
- **Documentation:** https://github.com/k3s-io/k3s-upgrade
- **Plans:** `k3s-server` (control plane), `k3s-agent` (workers)
### Checking Upgrade Status
```bash
# View current plans
kubectl get plans -n system-upgrade
# View upgrade jobs
kubectl get jobs -n system-upgrade
# View controller logs
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller
```
### Configuration
Upgrade plans are in: `infrastructure/k3s-upgrade/`
To modify upgrade behavior:
```bash
kubectl edit plan k3s-server -n system-upgrade
kubectl edit plan k3s-agent -n system-upgrade
```
Key configuration options:
- `version`: Target k3s version
- `concurrency`: Number of nodes to upgrade simultaneously (default: 1)
- `cordon`: Cordon nodes before upgrade (default: true)
- `drain`: Drain pods before upgrade (default: true)
---
## Monitoring and Alerts
### Version Skew Detection
Check for version inconsistencies:
```bash
#!/bin/bash
VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u)
VERSION_COUNT=$(echo "$VERSIONS" | wc -l)
if [ "$VERSION_COUNT" -gt 1 ]; then
echo "❌ ERROR: k3s version mismatch detected!"
echo "Versions found:"
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
exit 1
fi
echo "✅ All nodes running same k3s version: $VERSIONS"
```
### Prometheus Alerts
If Prometheus is deployed, version skew alerts are configured in:
`infrastructure/monitoring/prometheus-alerts/k3s-version-skew.yaml`
---
## Troubleshooting
### Node Not Joining Cluster
1. **Check firewall rules:**
```bash
hcloud firewall describe fw-k3s
# Verify node's public IP is in allowed sources
```
2. **Check k3s-agent service:**
```bash
ssh root@<node>
systemctl status k3s-agent
journalctl -u k3s-agent -n 100
```
3. **Verify connectivity to control plane:**
```bash
ssh root@<node>
curl -k https://10.42.1.1:6443
```
### Pods Not Scheduling on Node
1. **Check node status:**
```bash
kubectl describe node <node-name>
```
2. **Check for taints:**
```bash
kubectl get node <node-name> -o jsonpath='{.spec.taints}'
```
3. **Check flannel:**
```bash
kubectl get node <node-name> -o yaml | grep flannel
kubectl logs -n kube-system -l app=flannel
```
### Version Mismatch After Node Addition
**Symptom:** New node has different k3s version than cluster.
**Fix:**
```bash
# Cordon node
kubectl cordon <node-name>
# Reinstall with correct version
CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+')
ssh root@<node>
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v${CLUSTER_VERSION}" sh -s - agent \
--server https://10.42.1.1:6443 \
--token '<token>'
systemctl restart k3s-agent
# Uncordon
kubectl uncordon <node-name>
```
---
## Best Practices
1. **Always pin k3s version** when adding nodes
2. **Use the provisioning script** to ensure consistency
3. **Update firewall rules** before adding nodes (Flannel requires public IP access)
4. **Test upgrades** in a non-production environment first
5. **Monitor version consistency** regularly
6. **Use system-upgrade-controller** for all upgrades
7. **Never bypass version checks** - version skew causes networking issues
---
## Cluster Architecture
### Network Configuration
- **Pod CIDR:** 10.244.0.0/16
- **Service CIDR:** 10.96.0.0/12
- **CNI:** Flannel (VXLAN mode)
- **Flannel Backend:** Uses **public IPs** for VXLAN tunnels (not private IPs)
### Node CIDRs
| Node | Pod CIDR | Private IP | Public IP |
|------|----------|-----------|-----------|
| k3s-cp-1 | 10.244.0.0/24 | 10.42.1.1 | 178.105.17.239 |
| k3s-worker-1 | 10.244.1.0/24 | 10.42.1.2 | - |
| k3s-worker-2 | 10.244.2.0/24 | 10.42.1.3 | - |
| k3s-update-runner | 10.244.3.0/24 | 167.233.79.65 | 167.233.79.65 |
| k3s-worker-3 | 10.244.5.0/24 | 10.42.1.5 | 167.233.121.121 |
### Critical Firewall Rule
**All nodes with public IPs must be in the `fw-k3s` firewall** to allow Flannel VXLAN tunnels. Without this, pods cannot communicate across nodes.
---
## Change History
| Date | Version | Changes | By |
|------|---------|---------|-----|
| 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent |
| 2026-06-10 | v1.35.5+k3s1 | Original cluster deployment | - |
---
## References
- [k3s Documentation](https://docs.k3s.io/)
- [k3s Releases](https://github.com/k3s-io/k3s/releases)
- [k3s System Upgrade Controller](https://github.com/k3s-io/k3s-upgrade)
- [Hetzner Cloud Firewall](https://docs.hetzner.com/cloud/firewalls/)

126
infrastructure/k3s-upgrade/README.md Normal file
View file

@ -0,0 +1,126 @@
# k3s System Upgrade Controller
This directory contains configuration for automated k3s cluster upgrades using the [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller).
## Installation
The system-upgrade-controller should already be installed. If not, install it with:
```bash
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
```
Verify installation:
```bash
kubectl get pods -n system-upgrade
kubectl get plans -n system-upgrade
```
## Upgrade Plans
Two upgrade plans are configured:
- `k3s-server.yaml` - Upgrades control plane nodes
- `k3s-agent.yaml` - Upgrades worker nodes
## Triggering an Upgrade
To upgrade the cluster to a new k3s version:
1. **Edit the plans to set the desired version:**
```bash
kubectl edit plan k3s-server -n system-upgrade
kubectl edit plan k3s-agent -n system-upgrade
```
Change the `version` field:
```yaml
spec:
version: v1.37.0+k3s1 # Update to desired version
```
2. **Or apply updated plan files:**
```bash
# Update version in the YAML files first
kubectl apply -f infrastructure/k3s-upgrade/k3s-server.yaml
kubectl apply -f infrastructure/k3s-upgrade/k3s-agent.yaml
```
3. **Monitor the upgrade:**
```bash
# Watch nodes being upgraded
kubectl get nodes -w
# Check upgrade jobs
kubectl get jobs -n system-upgrade
# View controller logs
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller -f
```
## How It Works
1. The controller watches for Plan updates
2. When a new version is detected, it creates Jobs for each matching node
3. Nodes are cordoned and drained before upgrade
4. The upgrade job runs `k3s-upgrade` script to update k3s
5. Node is rebooted (if configured) and uncordoned
6. Process repeats for next node (respecting concurrency limits)
## Configuration Options
Key fields in the Plan spec:
- **version**: Target k3s version (e.g., `v1.36.2+k3s1`)
- **concurrency**: Number of nodes to upgrade simultaneously (default: 1)
- **nodeSelector**: Which nodes to upgrade (e.g., control-plane, worker)
- **cordon**: Cordon nodes before upgrade (default: true)
- **drain**: Drain pods before upgrade (default: true)
- **upgrade.cattle.io/tolerations**: Allow upgrade pods on tainted nodes
## Safety
- Upgrades are **rolling** - one node at a time by default
- Nodes are **drained** before upgrade to avoid pod disruption
- The controller respects **Pod Disruption Budgets**
- Failed upgrades can be **rolled back** by changing the version back
## Troubleshooting
### Upgrade stuck
Check job status:
```bash
kubectl get jobs -n system-upgrade
kubectl describe job <job-name> -n system-upgrade
kubectl logs -n system-upgrade job/<job-name>
```
### Plan not triggering
Ensure the version changed:
```bash
kubectl get plan k3s-server -n system-upgrade -o yaml | grep version
```
Check controller logs:
```bash
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller
```
### Manual intervention needed
Delete stuck jobs:
```bash
kubectl delete job <job-name> -n system-upgrade
```
Uncordon nodes manually if needed:
```bash
kubectl uncordon <node-name>
```
## References
- [system-upgrade-controller Documentation](https://github.com/rancher/system-upgrade-controller)
- [k3s Automated Upgrades](https://docs.k3s.io/upgrades/automated)
- [k3s Releases](https://github.com/k3s-io/k3s/releases)

40
infrastructure/k3s-upgrade/k3s-agent.yaml Normal file
View file

@ -0,0 +1,40 @@
# Upgrade plan for k3s worker nodes
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: k3s-agent
namespace: system-upgrade
spec:
# Target k3s version - update this to trigger upgrades
version: v1.36.2+k3s1
# Node selector - target worker nodes (not control plane)
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
# Service account with permissions to drain nodes
serviceAccountName: system-upgrade
# Cordon node before upgrade
cordon: true
# Drain pods before upgrade
drain:
force: true
skipWaitForDeleteTimeout: 60
timeout: 120s
deleteEmptyDir: true
ignoreDaemonSets: true
# Only upgrade workers after control plane is done
prepare:
image: rancher/k3s-upgrade
args:
- prepare
- k3s-server
# Container spec for upgrade job
upgrade:
image: rancher/k3s-upgrade

44
infrastructure/k3s-upgrade/k3s-server.yaml Normal file
View file

@ -0,0 +1,44 @@
# Upgrade plan for k3s control plane nodes
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: k3s-server
namespace: system-upgrade
spec:
# Target k3s version - update this to trigger upgrades
version: v1.36.2+k3s1
# Node selector - target control plane nodes
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists
# Service account with permissions to drain nodes
serviceAccountName: system-upgrade
# Cordon node before upgrade
cordon: true
# Drain pods before upgrade (with tolerations)
drain:
force: true
skipWaitForDeleteTimeout: 60
timeout: 120s
deleteEmptyDir: true
ignoreDaemonSets: true
# Container spec for upgrade job
upgrade:
image: rancher/k3s-upgrade
# Tolerate control plane taints
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule

113
infrastructure/scripts/provision-k3s-worker.sh Executable file
View file

@ -0,0 +1,113 @@
#!/bin/bash
#
# k3s Worker Node Provisioning Script
# Ensures version consistency across the cluster
#
# Usage: ./provision-k3s-worker.sh <new-node-ip> [<new-node-public-ip>]
#
set -e
# Configuration
CONTROL_PLANE_IP="10.42.1.1"
CONTROL_PLANE_PUBLIC="178.105.17.239"
NEW_NODE_IP="${1}"
NEW_NODE_PUBLIC_IP="${2:-$NEW_NODE_IP}"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
if [ -z "$NEW_NODE_IP" ]; then
echo -e "${RED}Error: Missing required argument${NC}"
echo "Usage: $0 <new-node-ip> [<new-node-public-ip>]"
echo ""
echo "Example: $0 10.42.1.6 167.233.121.122"
exit 1
fi
echo -e "${GREEN}=== k3s Worker Node Provisioning ===${NC}"
echo "Control Plane: $CONTROL_PLANE_PUBLIC ($CONTROL_PLANE_IP)"
echo "New Node: $NEW_NODE_IP (public: $NEW_NODE_PUBLIC_IP)"
echo ""
# Step 1: Get current cluster version
echo -e "${YELLOW}[1/6]${NC} Checking current cluster version..."
CLUSTER_VERSION=$(ssh root@${CONTROL_PLANE_PUBLIC} 'k3s --version' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+' || echo "")
if [ -z "$CLUSTER_VERSION" ]; then
echo -e "${RED}Error: Could not determine cluster version${NC}"
exit 1
fi
echo -e "${GREEN}✓${NC} Cluster running k3s v${CLUSTER_VERSION}"
echo ""
# Step 2: Check Hetzner firewall
echo -e "${YELLOW}[2/6]${NC} Firewall configuration check"
echo -e "${RED}⚠ MANUAL ACTION REQUIRED${NC}"
echo "Ensure ${NEW_NODE_PUBLIC_IP} is added to Hetzner firewall 'fw-k3s'"
echo ""
echo "Command to add (if not already present):"
echo " hcloud firewall add-rule fw-k3s --direction in --protocol tcp --source-ips ${NEW_NODE_PUBLIC_IP}/32 --port any"
echo ""
read -p "Press Enter when firewall is updated..."
echo ""
# Step 3: Verify SSH access
echo -e "${YELLOW}[3/6]${NC} Verifying SSH access to new node..."
if ! ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new root@${NEW_NODE_IP} 'echo ok' &>/dev/null; then
echo -e "${RED}Error: Cannot reach ${NEW_NODE_IP} via SSH${NC}"
exit 1
fi
echo -e "${GREEN}✓${NC} SSH access confirmed"
echo ""
# Step 4: Get K3S_TOKEN
echo -e "${YELLOW}[4/6]${NC} Retrieving k3s join token..."
K3S_TOKEN=$(ssh root@${CONTROL_PLANE_PUBLIC} "cat /var/lib/rancher/k3s/server/node-token")
if [ -z "$K3S_TOKEN" ]; then
echo -e "${RED}Error: Could not retrieve K3S_TOKEN${NC}"
exit 1
fi
echo -e "${GREEN}✓${NC} Token retrieved"
echo ""
# Step 5: Install k3s on new node with matching version
echo -e "${YELLOW}[5/6]${NC} Installing k3s v${CLUSTER_VERSION} on ${NEW_NODE_IP}..."
echo "This may take a few minutes..."
ssh root@${NEW_NODE_IP} "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
--server https://${CONTROL_PLANE_IP}:6443 \
--token '${K3S_TOKEN}'"
if [ $? -ne 0 ]; then
echo -e "${RED}Error: k3s installation failed${NC}"
exit 1
fi
echo -e "${GREEN}✓${NC} k3s installed"
echo ""
# Step 6: Verify node joined successfully
echo -e "${YELLOW}[6/6]${NC} Verifying node joined the cluster..."
sleep 15 # Give the node time to register
echo ""
echo "Cluster nodes:"
kubectl get nodes -o wide
echo ""
echo "Node version details:"
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,CONTAINER-RUNTIME:.status.nodeInfo.containerRuntimeVersion
echo ""
echo -e "${GREEN}✓✓✓ Node provisioned successfully!${NC}"
echo ""
echo "Next steps:"
echo " 1. Label the node if needed: kubectl label nodes <node-name> node-role.kubernetes.io/worker=worker"
echo " 2. Monitor pod scheduling: kubectl get pods --all-namespaces -o wide | grep <node-name>"
echo " 3. Verify flannel annotation: kubectl get node <node-name> -o yaml | grep flannel"