stack.basicstack.de/infrastructure/K3S_OPERATIONS.md

k3s Cluster Operations

Current Cluster Status

• Version: v1.36.2+k3s1
• Last Updated: 2026-07-06
• Control Plane: k3s-cp-1 (178.105.17.239 / 10.42.1.1)
• Workers: k3s-worker-1, k3s-worker-2, k3s-worker-3, k3s-update-runner
• Update Method: k3s system-upgrade-controller (automated)

Quick Reference

# Check cluster version
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion

# Check for version skew
VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u | wc -l)
if [ "$VERSIONS" -gt 1 ]; then echo "⚠️  Version skew detected!"; fi

# View current cluster state
kubectl get nodes -o wide

---

Adding a New Worker Node

Prerequisites

1. Hetzner Cloud server running Ubuntu 24.04 LTS
2. SSH access configured
3. hcloud CLI installed (for firewall management)

Procedure

Use the automated provisioning script:

cd infrastructure/scripts
./provision-k3s-worker.sh <new-node-private-ip> <new-node-public-ip>

Example:

./provision-k3s-worker.sh 10.42.1.6 167.233.121.122

The script will:

1. Check current cluster version
2. Prompt for Hetzner firewall update
3. Verify SSH access
4. Install k3s with matching version
5. Verify node joined successfully

Manual Steps (if script unavailable)

1. Get cluster version:

   CLUSTER_VERSION=$(ssh root@k3s-cp-1 'k3s --version' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+')
   echo "Cluster version: v${CLUSTER_VERSION}"
   

2. Update Hetzner firewall fw-k3s:

   # Add new node's public IP to firewall rules
   hcloud firewall add-rule fw-k3s --direction in --protocol tcp \
     --source-ips <new-node-public-ip>/32 --port any
   

   Critical: Flannel VXLAN uses public IPs for tunnels. If the public IP is not in the firewall, pods on that node cannot reach DNS or other pods.

3. Get K3S join token:

   K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token')
   

4. Install k3s on new node with version pinning:

   ssh root@<new-node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
     --server https://10.42.1.1:6443 \
     --token '${K3S_TOKEN}'"
   

5. Verify node joined:

   kubectl get nodes
   kubectl get node <node-name> -o yaml | grep flannel
   

---

Upgrading the Cluster

Upgrade Strategy

We use k3s system-upgrade-controller for coordinated, automated upgrades.

The upgrade controller:

• Ensures version consistency across the cluster
• Performs rolling upgrades (one node at a time)
• Respects pod disruption budgets
• Automatically drains and cordons nodes during upgrade
• Prevents version skew

Planning an Upgrade

1. Review k3s release notes:

   • Visit: https://github.com/k3s-io/k3s/releases
   • Check for breaking changes
   • Note any special upgrade instructions

2. Check current version:

   kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
   

3. Schedule maintenance window (optional for minor upgrades)

Triggering an Upgrade

The system-upgrade-controller watches for Plan updates. To upgrade:

1. Edit the upgrade plan:

   kubectl edit plan k3s-server -n system-upgrade
   kubectl edit plan k3s-agent -n system-upgrade
   

2. Update the version field:

   spec:
     version: v1.37.0+k3s1  # Change to desired version
   

3. Monitor the upgrade:

   # Watch nodes being upgraded
   kubectl get nodes -w
   
   # Check upgrade jobs
   kubectl get jobs -n system-upgrade
   
   # View upgrade controller logs
   kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller -f
   

4. Verify completion:

   kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
   kubectl get pods --all-namespaces -o wide
   

Manual Upgrade (Emergency Only)

If the upgrade controller is unavailable:

Control Plane

ssh root@178.105.17.239
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh -
systemctl restart k3s
# Wait 5 minutes for control plane to stabilize

Worker Nodes (one at a time)

# Drain node
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

# Upgrade node
ssh root@<worker-node>
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh -s - agent
systemctl restart k3s-agent

# Verify and uncordon
kubectl get nodes
kubectl uncordon <worker-node>

# Wait for pods to stabilize before proceeding to next node
kubectl get pods --all-namespaces -o wide | grep <worker-node>

Emergency Rollback

If issues occur after upgrade:

# On affected node
ssh root@<node>
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.36.2+k3s1" sh -s - agent
systemctl restart k3s-agent

Then investigate the issue before retrying the upgrade.

---

k3s System-Upgrade-Controller

Overview

The system-upgrade-controller manages automated k3s upgrades across the cluster.

• Namespace: system-upgrade
• Documentation: https://github.com/k3s-io/k3s-upgrade
• Plans: k3s-server (control plane), k3s-agent (workers)

Checking Upgrade Status

# View current plans
kubectl get plans -n system-upgrade

# View upgrade jobs
kubectl get jobs -n system-upgrade

# View controller logs
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller

Configuration

Upgrade plans are in: infrastructure/k3s-upgrade/

To modify upgrade behavior:

kubectl edit plan k3s-server -n system-upgrade
kubectl edit plan k3s-agent -n system-upgrade

Key configuration options:

• version: Target k3s version
• concurrency: Number of nodes to upgrade simultaneously (default: 1)
• cordon: Cordon nodes before upgrade (default: true)
• drain: Drain pods before upgrade (default: true)

---

Monitoring and Alerts

Version Skew Detection

Check for version inconsistencies:

#!/bin/bash
VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u)
VERSION_COUNT=$(echo "$VERSIONS" | wc -l)

if [ "$VERSION_COUNT" -gt 1 ]; then
  echo "❌ ERROR: k3s version mismatch detected!"
  echo "Versions found:"
  kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
  exit 1
fi

echo "✅ All nodes running same k3s version: $VERSIONS"

Prometheus Alerts

If Prometheus is deployed, version skew alerts are configured in: infrastructure/monitoring/prometheus-alerts/k3s-version-skew.yaml

---

Troubleshooting

Node Not Joining Cluster

1. Check firewall rules:

   hcloud firewall describe fw-k3s
   # Verify node's public IP is in allowed sources
   

2. Check k3s-agent service:

   ssh root@<node>
   systemctl status k3s-agent
   journalctl -u k3s-agent -n 100
   

3. Verify connectivity to control plane:

   ssh root@<node>
   curl -k https://10.42.1.1:6443
   

Pods Not Scheduling on Node

1. Check node status:

   kubectl describe node <node-name>
   

2. Check for taints:

   kubectl get node <node-name> -o jsonpath='{.spec.taints}'
   

3. Check flannel:

   kubectl get node <node-name> -o yaml | grep flannel
   kubectl logs -n kube-system -l app=flannel
   

Version Mismatch After Node Addition

Symptom: New node has different k3s version than cluster.

Fix:

# Cordon node
kubectl cordon <node-name>

# Reinstall with correct version
CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+')

ssh root@<node>
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v${CLUSTER_VERSION}" sh -s - agent \
  --server https://10.42.1.1:6443 \
  --token '<token>'
systemctl restart k3s-agent

# Uncordon
kubectl uncordon <node-name>

API Server Instability / Connection Refused Errors

Symptom: Intermittent API server connection failures with errors like:

The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
Error from server (ServiceUnavailable): the server is currently unable to handle the request

Root Cause: Cluster CIDR mismatch between bootstrap configuration and actual node PodCIDR allocations. This causes flannel to crash repeatedly with:

failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16" specified in the flannel net config doesn't contain "Y.Y.Y.Y/24" PodCIDR

Fix:

1. Verify the actual pod CIDR being used by nodes:

   kubectl get nodes -o custom-columns=NAME:.metadata.name,POD-CIDR:.spec.podCIDR
   

2. Edit the k3s service file to explicitly set the cluster-cidr:

   ssh root@178.105.17.239
   
   # Backup the service file
   cp /etc/systemd/system/k3s.service /etc/systemd/system/k3s.service.backup
   
   # Edit the service file
   nano /etc/systemd/system/k3s.service
   
   # Add --cluster-cidr flag to the ExecStart line:
   ExecStart=/usr/local/bin/k3s \
       server \
       --cluster-cidr=10.244.0.0/16 \
   

3. Reload and restart k3s:

   systemctl daemon-reload
   systemctl restart k3s
   

4. Verify stability:

   # Test API server with multiple consecutive calls
   for i in {1..20}; do kubectl get nodes --no-headers | wc -l; sleep 2; done
   # Should return node count consistently without errors
   

Prevention: Always explicitly set --cluster-cidr in the k3s server configuration to match the actual network setup.

---

Best Practices

1. Always pin k3s version when adding nodes
2. Use the provisioning script to ensure consistency
3. Update firewall rules before adding nodes (Flannel requires public IP access)
4. Test upgrades in a non-production environment first
5. Monitor version consistency regularly
6. Use system-upgrade-controller for all upgrades
7. Never bypass version checks - version skew causes networking issues

---

Cluster Architecture

Network Configuration

• Pod CIDR: 10.244.0.0/16
• Service CIDR: 10.43.0.0/16
• CNI: Flannel (VXLAN mode)
• Flannel Backend: Uses public IPs for VXLAN tunnels (not private IPs)

Important: The cluster-cidr must be explicitly set in the k3s service configuration to prevent bootstrap/runtime mismatch. See Troubleshooting section for details.

Node CIDRs

Node	Pod CIDR	Private IP	Public IP
k3s-cp-1	10.244.0.0/24	10.42.1.1	178.105.17.239
k3s-worker-1	10.244.1.0/24	10.42.1.2	-
k3s-worker-2	10.244.2.0/24	10.42.1.3	-
k3s-update-runner	10.244.3.0/24	167.233.79.65	167.233.79.65
k3s-worker-3	10.244.5.0/24	10.42.1.5	167.233.121.121

Critical Firewall Rule

All nodes with public IPs must be in the fw-k3s firewall to allow Flannel VXLAN tunnels. Without this, pods cannot communicate across nodes.

---

Change History

Date	Version	Changes	By
2026-07-06	v1.36.2+k3s1	Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1, corrected Service CIDR documentation to 10.43.0.0/16	CTO Agent
2026-07-06	v1.36.2+k3s1	Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller	CTO Agent
2026-06-10	v1.35.5+k3s1	Original cluster deployment	-

---

References

• k3s Documentation
• k3s Releases
• k3s System Upgrade Controller
• Hetzner Cloud Firewall