stack.basicstack.de/infrastructure/K3S_OPERATIONS.md

# k3s Cluster Operations

## Current Cluster Status

- **Version:** v1.36.2+k3s1
- **Last Updated:** 2026-07-06
- **Control Plane:** k3s-cp-1 (178.105.17.239 / 10.42.1.1)
- **Workers:** k3s-worker-1, k3s-worker-2, k3s-worker-3, k3s-update-runner
- **Update Method:** k3s system-upgrade-controller (automated)

## Quick Reference

```bash
# Check cluster version
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion

# Check for version skew
VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u | wc -l)
if [ "$VERSIONS" -gt 1 ]; then echo "⚠️  Version skew detected!"; fi

# View current cluster state
kubectl get nodes -o wide
```

---

## Adding a New Worker Node

### Prerequisites

1. Hetzner Cloud server running Ubuntu 24.04 LTS
2. SSH access configured
3. `hcloud` CLI installed (for firewall management)

### Procedure

Use the automated provisioning script:

```bash
cd infrastructure/scripts
./provision-k3s-worker.sh <new-node-private-ip> <new-node-public-ip>
```

Example:
```bash
./provision-k3s-worker.sh 10.42.1.6 167.233.121.122
```

The script will:
1. Check current cluster version
2. Prompt for Hetzner firewall update
3. Verify SSH access
4. Install k3s with matching version
5. Verify node joined successfully

### Manual Steps (if script unavailable)

1. **Get cluster version:**
   ```bash
   CLUSTER_VERSION=$(ssh root@k3s-cp-1 'k3s --version' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+')
   echo "Cluster version: v${CLUSTER_VERSION}"
   ```

2. **Update Hetzner firewall `fw-k3s`:**
   ```bash
   # Add new node's public IP to firewall rules
   hcloud firewall add-rule fw-k3s --direction in --protocol tcp \
     --source-ips <new-node-public-ip>/32 --port any
   ```
   
   **Critical:** Flannel VXLAN uses public IPs for tunnels. If the public IP is not in the firewall, pods on that node cannot reach DNS or other pods.

3. **Get K3S join token:**
   ```bash
   K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token')
   ```

4. **Install k3s on new node with version pinning:**
   ```bash
   ssh root@<new-node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
     --server https://10.42.1.1:6443 \
     --token '${K3S_TOKEN}'"
   ```

5. **Verify node joined:**
   ```bash
   kubectl get nodes
   kubectl get node <node-name> -o yaml | grep flannel
   ```

---

## Upgrading the Cluster

### Upgrade Strategy

**We use k3s system-upgrade-controller for coordinated, automated upgrades.**

The upgrade controller:
- Ensures version consistency across the cluster
- Performs rolling upgrades (one node at a time)
- Respects pod disruption budgets
- Automatically drains and cordons nodes during upgrade
- Prevents version skew

### Planning an Upgrade

1. **Review k3s release notes:**
   - Visit: https://github.com/k3s-io/k3s/releases
   - Check for breaking changes
   - Note any special upgrade instructions

2. **Check current version:**
   ```bash
   kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
   ```

3. **Schedule maintenance window** (optional for minor upgrades)

### Triggering an Upgrade

The system-upgrade-controller watches for Plan updates. To upgrade:

1. **Edit the upgrade plan:**
   ```bash
   kubectl edit plan k3s-server -n system-upgrade
   kubectl edit plan k3s-agent -n system-upgrade
   ```

2. **Update the version field:**
   ```yaml
   spec:
     version: v1.37.0+k3s1  # Change to desired version
   ```

3. **Monitor the upgrade:**
   ```bash
   # Watch nodes being upgraded
   kubectl get nodes -w
   
   # Check upgrade jobs
   kubectl get jobs -n system-upgrade
   
   # View upgrade controller logs
   kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller -f
   ```

4. **Verify completion:**
   ```bash
   kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
   kubectl get pods --all-namespaces -o wide
   ```

### Manual Upgrade (Emergency Only)

If the upgrade controller is unavailable:

#### Control Plane

```bash
ssh root@178.105.17.239
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh -
systemctl restart k3s
# Wait 5 minutes for control plane to stabilize
```

#### Worker Nodes (one at a time)

```bash
# Drain node
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

# Upgrade node
ssh root@<worker-node>
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh -s - agent
systemctl restart k3s-agent

# Verify and uncordon
kubectl get nodes
kubectl uncordon <worker-node>

# Wait for pods to stabilize before proceeding to next node
kubectl get pods --all-namespaces -o wide | grep <worker-node>
```

### Emergency Rollback

If issues occur after upgrade:

```bash
# On affected node
ssh root@<node>
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.36.2+k3s1" sh -s - agent
systemctl restart k3s-agent
```

Then investigate the issue before retrying the upgrade.

---

## k3s System-Upgrade-Controller

### Overview

The system-upgrade-controller manages automated k3s upgrades across the cluster.

- **Namespace:** `system-upgrade`
- **Documentation:** https://github.com/k3s-io/k3s-upgrade
- **Plans:** `k3s-server` (control plane), `k3s-agent` (workers)

### Checking Upgrade Status

```bash
# View current plans
kubectl get plans -n system-upgrade

# View upgrade jobs
kubectl get jobs -n system-upgrade

# View controller logs
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller
```

### Configuration

Upgrade plans are in: `infrastructure/k3s-upgrade/`

To modify upgrade behavior:
```bash
kubectl edit plan k3s-server -n system-upgrade
kubectl edit plan k3s-agent -n system-upgrade
```

Key configuration options:
- `version`: Target k3s version
- `concurrency`: Number of nodes to upgrade simultaneously (default: 1)
- `cordon`: Cordon nodes before upgrade (default: true)
- `drain`: Drain pods before upgrade (default: true)

---

## Monitoring and Alerts

### Version Skew Detection

Check for version inconsistencies:

```bash
#!/bin/bash
VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u)
VERSION_COUNT=$(echo "$VERSIONS" | wc -l)

if [ "$VERSION_COUNT" -gt 1 ]; then
  echo "❌ ERROR: k3s version mismatch detected!"
  echo "Versions found:"
  kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
  exit 1
fi

echo "✅ All nodes running same k3s version: $VERSIONS"
```

### Prometheus Alerts

If Prometheus is deployed, version skew alerts are configured in:
`infrastructure/monitoring/prometheus-alerts/k3s-version-skew.yaml`

---

## Troubleshooting

### Node Not Joining Cluster

1. **Check firewall rules:**
   ```bash
   hcloud firewall describe fw-k3s
   # Verify node's public IP is in allowed sources
   ```

2. **Check k3s-agent service:**
   ```bash
   ssh root@<node>
   systemctl status k3s-agent
   journalctl -u k3s-agent -n 100
   ```

3. **Verify connectivity to control plane:**
   ```bash
   ssh root@<node>
   curl -k https://10.42.1.1:6443
   ```

### Pods Not Scheduling on Node

1. **Check node status:**
   ```bash
   kubectl describe node <node-name>
   ```

2. **Check for taints:**
   ```bash
   kubectl get node <node-name> -o jsonpath='{.spec.taints}'
   ```

3. **Check flannel:**
   ```bash
   kubectl get node <node-name> -o yaml | grep flannel
   kubectl logs -n kube-system -l app=flannel
   ```

### Version Mismatch After Node Addition

**Symptom:** New node has different k3s version than cluster.

**Fix:**
```bash
# Cordon node
kubectl cordon <node-name>

# Reinstall with correct version
CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+')

ssh root@<node>
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v${CLUSTER_VERSION}" sh -s - agent \
  --server https://10.42.1.1:6443 \
  --token '<token>'
systemctl restart k3s-agent

# Uncordon
kubectl uncordon <node-name>
```

### API Server Instability / Connection Refused Errors

**Symptom:** Intermittent API server connection failures with errors like:
```
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
Error from server (ServiceUnavailable): the server is currently unable to handle the request
```

**Root Cause:** Cluster CIDR mismatch between bootstrap configuration and actual node PodCIDR allocations. This causes flannel to crash repeatedly with:
```
failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16" specified in the flannel net config doesn't contain "Y.Y.Y.Y/24" PodCIDR
```

**Fix:**
1. Verify the actual pod CIDR being used by nodes:
   ```bash
   kubectl get nodes -o custom-columns=NAME:.metadata.name,POD-CIDR:.spec.podCIDR
   ```

2. Edit the k3s service file to explicitly set the cluster-cidr:
   ```bash
   ssh root@178.105.17.239
   
   # Backup the service file
   cp /etc/systemd/system/k3s.service /etc/systemd/system/k3s.service.backup
   
   # Edit the service file
   nano /etc/systemd/system/k3s.service
   
   # Add --cluster-cidr flag to the ExecStart line:
   ExecStart=/usr/local/bin/k3s \
       server \
       --cluster-cidr=10.244.0.0/16 \
   ```

3. Reload and restart k3s:
   ```bash
   systemctl daemon-reload
   systemctl restart k3s
   ```

4. Verify stability:
   ```bash
   # Test API server with multiple consecutive calls
   for i in {1..20}; do kubectl get nodes --no-headers | wc -l; sleep 2; done
   # Should return node count consistently without errors
   ```

**Prevention:** Always explicitly set `--cluster-cidr` in the k3s server configuration to match the actual network setup.

---

## Best Practices

1. **Always pin k3s version** when adding nodes
2. **Use the provisioning script** to ensure consistency
3. **Update firewall rules** before adding nodes (Flannel requires public IP access)
4. **Test upgrades** in a non-production environment first
5. **Monitor version consistency** regularly
6. **Use system-upgrade-controller** for all upgrades
7. **Never bypass version checks** - version skew causes networking issues

---

## Cluster Architecture

### Network Configuration

- **Pod CIDR:** 10.244.0.0/16
- **Service CIDR:** 10.96.0.0/16
- **Cluster DNS:** 10.96.0.10 (kube-dns service)
- **CNI:** Flannel (VXLAN mode)
- **Flannel Backend:** Uses **public IPs** for VXLAN tunnels (not private IPs)

**Important:** 
- The cluster-cidr must be explicitly set in the k3s service configuration to prevent bootstrap/runtime mismatch. See Troubleshooting section for details.
- The kubelet cluster-dns must be configured to match the kube-dns service ClusterIP. See `infrastructure/networking/DNS_CONFIGURATION.md` for details.

### Node CIDRs

| Node | Pod CIDR | Private IP | Public IP |
|------|----------|-----------|-----------|
| k3s-cp-1 | 10.244.0.0/24 | 10.42.1.1 | 178.105.17.239 |
| k3s-worker-1 | 10.244.1.0/24 | 10.42.1.2 | - |
| k3s-worker-2 | 10.244.2.0/24 | 10.42.1.3 | - |
| k3s-update-runner | 10.244.3.0/24 | 167.233.79.65 | 167.233.79.65 |
| k3s-worker-3 | 10.244.5.0/24 | 10.42.1.5 | 167.233.121.121 |

### Critical Firewall Rule

**All nodes with public IPs must be in the `fw-k3s` firewall** to allow Flannel VXLAN tunnels. Without this, pods cannot communicate across nodes.

---

## Change History

| Date | Version | Changes | By |
|------|---------|---------|-----|
| 2026-07-06 | v1.36.2+k3s1 | Fixed DNS service CIDR mismatch (kubelet cluster-dns configuration), corrected Service CIDR documentation to 10.96.0.0/16, added DNS configuration guide | CTO Agent |
| 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1 | CTO Agent |
| 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent |
| 2026-06-10 | v1.35.5+k3s1 | Original cluster deployment | - |

---

## References

- [k3s Documentation](https://docs.k3s.io/)
- [k3s Releases](https://github.com/k3s-io/k3s/releases)
- [k3s System Upgrade Controller](https://github.com/k3s-io/k3s-upgrade)
- [Hetzner Cloud Firewall](https://docs.hetzner.com/cloud/firewalls/)