diff --git a/infrastructure/K3S_OPERATIONS.md b/infrastructure/K3S_OPERATIONS.md new file mode 100644 index 0000000..8ea4fca --- /dev/null +++ b/infrastructure/K3S_OPERATIONS.md @@ -0,0 +1,386 @@ +# k3s Cluster Operations + +## Current Cluster Status + +- **Version:** v1.36.2+k3s1 +- **Last Updated:** 2026-07-06 +- **Control Plane:** k3s-cp-1 (178.105.17.239 / 10.42.1.1) +- **Workers:** k3s-worker-1, k3s-worker-2, k3s-worker-3, k3s-update-runner +- **Update Method:** k3s system-upgrade-controller (automated) + +## Quick Reference + +```bash +# Check cluster version +kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion + +# Check for version skew +VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u | wc -l) +if [ "$VERSIONS" -gt 1 ]; then echo "⚠️ Version skew detected!"; fi + +# View current cluster state +kubectl get nodes -o wide +``` + +--- + +## Adding a New Worker Node + +### Prerequisites + +1. Hetzner Cloud server running Ubuntu 24.04 LTS +2. SSH access configured +3. `hcloud` CLI installed (for firewall management) + +### Procedure + +Use the automated provisioning script: + +```bash +cd infrastructure/scripts +./provision-k3s-worker.sh +``` + +Example: +```bash +./provision-k3s-worker.sh 10.42.1.6 167.233.121.122 +``` + +The script will: +1. Check current cluster version +2. Prompt for Hetzner firewall update +3. Verify SSH access +4. Install k3s with matching version +5. Verify node joined successfully + +### Manual Steps (if script unavailable) + +1. **Get cluster version:** + ```bash + CLUSTER_VERSION=$(ssh root@k3s-cp-1 'k3s --version' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+') + echo "Cluster version: v${CLUSTER_VERSION}" + ``` + +2. **Update Hetzner firewall `fw-k3s`:** + ```bash + # Add new node's public IP to firewall rules + hcloud firewall add-rule fw-k3s --direction in --protocol tcp \ + --source-ips /32 --port any + ``` + + **Critical:** Flannel VXLAN uses public IPs for tunnels. If the public IP is not in the firewall, pods on that node cannot reach DNS or other pods. + +3. **Get K3S join token:** + ```bash + K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token') + ``` + +4. **Install k3s on new node with version pinning:** + ```bash + ssh root@ "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \ + --server https://10.42.1.1:6443 \ + --token '${K3S_TOKEN}'" + ``` + +5. **Verify node joined:** + ```bash + kubectl get nodes + kubectl get node -o yaml | grep flannel + ``` + +--- + +## Upgrading the Cluster + +### Upgrade Strategy + +**We use k3s system-upgrade-controller for coordinated, automated upgrades.** + +The upgrade controller: +- Ensures version consistency across the cluster +- Performs rolling upgrades (one node at a time) +- Respects pod disruption budgets +- Automatically drains and cordons nodes during upgrade +- Prevents version skew + +### Planning an Upgrade + +1. **Review k3s release notes:** + - Visit: https://github.com/k3s-io/k3s/releases + - Check for breaking changes + - Note any special upgrade instructions + +2. **Check current version:** + ```bash + kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion + ``` + +3. **Schedule maintenance window** (optional for minor upgrades) + +### Triggering an Upgrade + +The system-upgrade-controller watches for Plan updates. To upgrade: + +1. **Edit the upgrade plan:** + ```bash + kubectl edit plan k3s-server -n system-upgrade + kubectl edit plan k3s-agent -n system-upgrade + ``` + +2. **Update the version field:** + ```yaml + spec: + version: v1.37.0+k3s1 # Change to desired version + ``` + +3. **Monitor the upgrade:** + ```bash + # Watch nodes being upgraded + kubectl get nodes -w + + # Check upgrade jobs + kubectl get jobs -n system-upgrade + + # View upgrade controller logs + kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller -f + ``` + +4. **Verify completion:** + ```bash + kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion + kubectl get pods --all-namespaces -o wide + ``` + +### Manual Upgrade (Emergency Only) + +If the upgrade controller is unavailable: + +#### Control Plane + +```bash +ssh root@178.105.17.239 +curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh - +systemctl restart k3s +# Wait 5 minutes for control plane to stabilize +``` + +#### Worker Nodes (one at a time) + +```bash +# Drain node +kubectl drain --ignore-daemonsets --delete-emptydir-data + +# Upgrade node +ssh root@ +curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh -s - agent +systemctl restart k3s-agent + +# Verify and uncordon +kubectl get nodes +kubectl uncordon + +# Wait for pods to stabilize before proceeding to next node +kubectl get pods --all-namespaces -o wide | grep +``` + +### Emergency Rollback + +If issues occur after upgrade: + +```bash +# On affected node +ssh root@ +curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.36.2+k3s1" sh -s - agent +systemctl restart k3s-agent +``` + +Then investigate the issue before retrying the upgrade. + +--- + +## k3s System-Upgrade-Controller + +### Overview + +The system-upgrade-controller manages automated k3s upgrades across the cluster. + +- **Namespace:** `system-upgrade` +- **Documentation:** https://github.com/k3s-io/k3s-upgrade +- **Plans:** `k3s-server` (control plane), `k3s-agent` (workers) + +### Checking Upgrade Status + +```bash +# View current plans +kubectl get plans -n system-upgrade + +# View upgrade jobs +kubectl get jobs -n system-upgrade + +# View controller logs +kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller +``` + +### Configuration + +Upgrade plans are in: `infrastructure/k3s-upgrade/` + +To modify upgrade behavior: +```bash +kubectl edit plan k3s-server -n system-upgrade +kubectl edit plan k3s-agent -n system-upgrade +``` + +Key configuration options: +- `version`: Target k3s version +- `concurrency`: Number of nodes to upgrade simultaneously (default: 1) +- `cordon`: Cordon nodes before upgrade (default: true) +- `drain`: Drain pods before upgrade (default: true) + +--- + +## Monitoring and Alerts + +### Version Skew Detection + +Check for version inconsistencies: + +```bash +#!/bin/bash +VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u) +VERSION_COUNT=$(echo "$VERSIONS" | wc -l) + +if [ "$VERSION_COUNT" -gt 1 ]; then + echo "❌ ERROR: k3s version mismatch detected!" + echo "Versions found:" + kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion + exit 1 +fi + +echo "✅ All nodes running same k3s version: $VERSIONS" +``` + +### Prometheus Alerts + +If Prometheus is deployed, version skew alerts are configured in: +`infrastructure/monitoring/prometheus-alerts/k3s-version-skew.yaml` + +--- + +## Troubleshooting + +### Node Not Joining Cluster + +1. **Check firewall rules:** + ```bash + hcloud firewall describe fw-k3s + # Verify node's public IP is in allowed sources + ``` + +2. **Check k3s-agent service:** + ```bash + ssh root@ + systemctl status k3s-agent + journalctl -u k3s-agent -n 100 + ``` + +3. **Verify connectivity to control plane:** + ```bash + ssh root@ + curl -k https://10.42.1.1:6443 + ``` + +### Pods Not Scheduling on Node + +1. **Check node status:** + ```bash + kubectl describe node + ``` + +2. **Check for taints:** + ```bash + kubectl get node -o jsonpath='{.spec.taints}' + ``` + +3. **Check flannel:** + ```bash + kubectl get node -o yaml | grep flannel + kubectl logs -n kube-system -l app=flannel + ``` + +### Version Mismatch After Node Addition + +**Symptom:** New node has different k3s version than cluster. + +**Fix:** +```bash +# Cordon node +kubectl cordon + +# Reinstall with correct version +CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+') + +ssh root@ +curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v${CLUSTER_VERSION}" sh -s - agent \ + --server https://10.42.1.1:6443 \ + --token '' +systemctl restart k3s-agent + +# Uncordon +kubectl uncordon +``` + +--- + +## Best Practices + +1. **Always pin k3s version** when adding nodes +2. **Use the provisioning script** to ensure consistency +3. **Update firewall rules** before adding nodes (Flannel requires public IP access) +4. **Test upgrades** in a non-production environment first +5. **Monitor version consistency** regularly +6. **Use system-upgrade-controller** for all upgrades +7. **Never bypass version checks** - version skew causes networking issues + +--- + +## Cluster Architecture + +### Network Configuration + +- **Pod CIDR:** 10.244.0.0/16 +- **Service CIDR:** 10.96.0.0/12 +- **CNI:** Flannel (VXLAN mode) +- **Flannel Backend:** Uses **public IPs** for VXLAN tunnels (not private IPs) + +### Node CIDRs + +| Node | Pod CIDR | Private IP | Public IP | +|------|----------|-----------|-----------| +| k3s-cp-1 | 10.244.0.0/24 | 10.42.1.1 | 178.105.17.239 | +| k3s-worker-1 | 10.244.1.0/24 | 10.42.1.2 | - | +| k3s-worker-2 | 10.244.2.0/24 | 10.42.1.3 | - | +| k3s-update-runner | 10.244.3.0/24 | 167.233.79.65 | 167.233.79.65 | +| k3s-worker-3 | 10.244.5.0/24 | 10.42.1.5 | 167.233.121.121 | + +### Critical Firewall Rule + +**All nodes with public IPs must be in the `fw-k3s` firewall** to allow Flannel VXLAN tunnels. Without this, pods cannot communicate across nodes. + +--- + +## Change History + +| Date | Version | Changes | By | +|------|---------|---------|-----| +| 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent | +| 2026-06-10 | v1.35.5+k3s1 | Original cluster deployment | - | + +--- + +## References + +- [k3s Documentation](https://docs.k3s.io/) +- [k3s Releases](https://github.com/k3s-io/k3s/releases) +- [k3s System Upgrade Controller](https://github.com/k3s-io/k3s-upgrade) +- [Hetzner Cloud Firewall](https://docs.hetzner.com/cloud/firewalls/) diff --git a/infrastructure/k3s-upgrade/README.md b/infrastructure/k3s-upgrade/README.md new file mode 100644 index 0000000..cf488cd --- /dev/null +++ b/infrastructure/k3s-upgrade/README.md @@ -0,0 +1,126 @@ +# k3s System Upgrade Controller + +This directory contains configuration for automated k3s cluster upgrades using the [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller). + +## Installation + +The system-upgrade-controller should already be installed. If not, install it with: + +```bash +kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml +``` + +Verify installation: +```bash +kubectl get pods -n system-upgrade +kubectl get plans -n system-upgrade +``` + +## Upgrade Plans + +Two upgrade plans are configured: +- `k3s-server.yaml` - Upgrades control plane nodes +- `k3s-agent.yaml` - Upgrades worker nodes + +## Triggering an Upgrade + +To upgrade the cluster to a new k3s version: + +1. **Edit the plans to set the desired version:** + ```bash + kubectl edit plan k3s-server -n system-upgrade + kubectl edit plan k3s-agent -n system-upgrade + ``` + + Change the `version` field: + ```yaml + spec: + version: v1.37.0+k3s1 # Update to desired version + ``` + +2. **Or apply updated plan files:** + ```bash + # Update version in the YAML files first + kubectl apply -f infrastructure/k3s-upgrade/k3s-server.yaml + kubectl apply -f infrastructure/k3s-upgrade/k3s-agent.yaml + ``` + +3. **Monitor the upgrade:** + ```bash + # Watch nodes being upgraded + kubectl get nodes -w + + # Check upgrade jobs + kubectl get jobs -n system-upgrade + + # View controller logs + kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller -f + ``` + +## How It Works + +1. The controller watches for Plan updates +2. When a new version is detected, it creates Jobs for each matching node +3. Nodes are cordoned and drained before upgrade +4. The upgrade job runs `k3s-upgrade` script to update k3s +5. Node is rebooted (if configured) and uncordoned +6. Process repeats for next node (respecting concurrency limits) + +## Configuration Options + +Key fields in the Plan spec: + +- **version**: Target k3s version (e.g., `v1.36.2+k3s1`) +- **concurrency**: Number of nodes to upgrade simultaneously (default: 1) +- **nodeSelector**: Which nodes to upgrade (e.g., control-plane, worker) +- **cordon**: Cordon nodes before upgrade (default: true) +- **drain**: Drain pods before upgrade (default: true) +- **upgrade.cattle.io/tolerations**: Allow upgrade pods on tainted nodes + +## Safety + +- Upgrades are **rolling** - one node at a time by default +- Nodes are **drained** before upgrade to avoid pod disruption +- The controller respects **Pod Disruption Budgets** +- Failed upgrades can be **rolled back** by changing the version back + +## Troubleshooting + +### Upgrade stuck + +Check job status: +```bash +kubectl get jobs -n system-upgrade +kubectl describe job -n system-upgrade +kubectl logs -n system-upgrade job/ +``` + +### Plan not triggering + +Ensure the version changed: +```bash +kubectl get plan k3s-server -n system-upgrade -o yaml | grep version +``` + +Check controller logs: +```bash +kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller +``` + +### Manual intervention needed + +Delete stuck jobs: +```bash +kubectl delete job -n system-upgrade +``` + +Uncordon nodes manually if needed: +```bash +kubectl uncordon +``` + +## References + +- [system-upgrade-controller Documentation](https://github.com/rancher/system-upgrade-controller) +- [k3s Automated Upgrades](https://docs.k3s.io/upgrades/automated) +- [k3s Releases](https://github.com/k3s-io/k3s/releases) diff --git a/infrastructure/k3s-upgrade/k3s-agent.yaml b/infrastructure/k3s-upgrade/k3s-agent.yaml new file mode 100644 index 0000000..7e8c0e2 --- /dev/null +++ b/infrastructure/k3s-upgrade/k3s-agent.yaml @@ -0,0 +1,40 @@ +# Upgrade plan for k3s worker nodes +apiVersion: upgrade.cattle.io/v1 +kind: Plan +metadata: + name: k3s-agent + namespace: system-upgrade +spec: + # Target k3s version - update this to trigger upgrades + version: v1.36.2+k3s1 + + # Node selector - target worker nodes (not control plane) + nodeSelector: + matchExpressions: + - key: node-role.kubernetes.io/control-plane + operator: DoesNotExist + + # Service account with permissions to drain nodes + serviceAccountName: system-upgrade + + # Cordon node before upgrade + cordon: true + + # Drain pods before upgrade + drain: + force: true + skipWaitForDeleteTimeout: 60 + timeout: 120s + deleteEmptyDir: true + ignoreDaemonSets: true + + # Only upgrade workers after control plane is done + prepare: + image: rancher/k3s-upgrade + args: + - prepare + - k3s-server + + # Container spec for upgrade job + upgrade: + image: rancher/k3s-upgrade diff --git a/infrastructure/k3s-upgrade/k3s-server.yaml b/infrastructure/k3s-upgrade/k3s-server.yaml new file mode 100644 index 0000000..0e846f6 --- /dev/null +++ b/infrastructure/k3s-upgrade/k3s-server.yaml @@ -0,0 +1,44 @@ +# Upgrade plan for k3s control plane nodes +apiVersion: upgrade.cattle.io/v1 +kind: Plan +metadata: + name: k3s-server + namespace: system-upgrade +spec: + # Target k3s version - update this to trigger upgrades + version: v1.36.2+k3s1 + + # Node selector - target control plane nodes + nodeSelector: + matchExpressions: + - key: node-role.kubernetes.io/control-plane + operator: Exists + + # Service account with permissions to drain nodes + serviceAccountName: system-upgrade + + # Cordon node before upgrade + cordon: true + + # Drain pods before upgrade (with tolerations) + drain: + force: true + skipWaitForDeleteTimeout: 60 + timeout: 120s + deleteEmptyDir: true + ignoreDaemonSets: true + + # Container spec for upgrade job + upgrade: + image: rancher/k3s-upgrade + + # Tolerate control plane taints + tolerations: + - key: CriticalAddonsOnly + operator: Exists + - key: node-role.kubernetes.io/control-plane + operator: Exists + effect: NoSchedule + - key: node-role.kubernetes.io/master + operator: Exists + effect: NoSchedule diff --git a/infrastructure/scripts/provision-k3s-worker.sh b/infrastructure/scripts/provision-k3s-worker.sh new file mode 100755 index 0000000..b7479db --- /dev/null +++ b/infrastructure/scripts/provision-k3s-worker.sh @@ -0,0 +1,113 @@ +#!/bin/bash +# +# k3s Worker Node Provisioning Script +# Ensures version consistency across the cluster +# +# Usage: ./provision-k3s-worker.sh [] +# + +set -e + +# Configuration +CONTROL_PLANE_IP="10.42.1.1" +CONTROL_PLANE_PUBLIC="178.105.17.239" +NEW_NODE_IP="${1}" +NEW_NODE_PUBLIC_IP="${2:-$NEW_NODE_IP}" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +if [ -z "$NEW_NODE_IP" ]; then + echo -e "${RED}Error: Missing required argument${NC}" + echo "Usage: $0 []" + echo "" + echo "Example: $0 10.42.1.6 167.233.121.122" + exit 1 +fi + +echo -e "${GREEN}=== k3s Worker Node Provisioning ===${NC}" +echo "Control Plane: $CONTROL_PLANE_PUBLIC ($CONTROL_PLANE_IP)" +echo "New Node: $NEW_NODE_IP (public: $NEW_NODE_PUBLIC_IP)" +echo "" + +# Step 1: Get current cluster version +echo -e "${YELLOW}[1/6]${NC} Checking current cluster version..." +CLUSTER_VERSION=$(ssh root@${CONTROL_PLANE_PUBLIC} 'k3s --version' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+' || echo "") + +if [ -z "$CLUSTER_VERSION" ]; then + echo -e "${RED}Error: Could not determine cluster version${NC}" + exit 1 +fi + +echo -e "${GREEN}✓${NC} Cluster running k3s v${CLUSTER_VERSION}" +echo "" + +# Step 2: Check Hetzner firewall +echo -e "${YELLOW}[2/6]${NC} Firewall configuration check" +echo -e "${RED}⚠ MANUAL ACTION REQUIRED${NC}" +echo "Ensure ${NEW_NODE_PUBLIC_IP} is added to Hetzner firewall 'fw-k3s'" +echo "" +echo "Command to add (if not already present):" +echo " hcloud firewall add-rule fw-k3s --direction in --protocol tcp --source-ips ${NEW_NODE_PUBLIC_IP}/32 --port any" +echo "" +read -p "Press Enter when firewall is updated..." +echo "" + +# Step 3: Verify SSH access +echo -e "${YELLOW}[3/6]${NC} Verifying SSH access to new node..." +if ! ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new root@${NEW_NODE_IP} 'echo ok' &>/dev/null; then + echo -e "${RED}Error: Cannot reach ${NEW_NODE_IP} via SSH${NC}" + exit 1 +fi +echo -e "${GREEN}✓${NC} SSH access confirmed" +echo "" + +# Step 4: Get K3S_TOKEN +echo -e "${YELLOW}[4/6]${NC} Retrieving k3s join token..." +K3S_TOKEN=$(ssh root@${CONTROL_PLANE_PUBLIC} "cat /var/lib/rancher/k3s/server/node-token") + +if [ -z "$K3S_TOKEN" ]; then + echo -e "${RED}Error: Could not retrieve K3S_TOKEN${NC}" + exit 1 +fi +echo -e "${GREEN}✓${NC} Token retrieved" +echo "" + +# Step 5: Install k3s on new node with matching version +echo -e "${YELLOW}[5/6]${NC} Installing k3s v${CLUSTER_VERSION} on ${NEW_NODE_IP}..." +echo "This may take a few minutes..." + +ssh root@${NEW_NODE_IP} "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \ + --server https://${CONTROL_PLANE_IP}:6443 \ + --token '${K3S_TOKEN}'" + +if [ $? -ne 0 ]; then + echo -e "${RED}Error: k3s installation failed${NC}" + exit 1 +fi + +echo -e "${GREEN}✓${NC} k3s installed" +echo "" + +# Step 6: Verify node joined successfully +echo -e "${YELLOW}[6/6]${NC} Verifying node joined the cluster..." +sleep 15 # Give the node time to register + +echo "" +echo "Cluster nodes:" +kubectl get nodes -o wide + +echo "" +echo "Node version details:" +kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,CONTAINER-RUNTIME:.status.nodeInfo.containerRuntimeVersion + +echo "" +echo -e "${GREEN}✓✓✓ Node provisioned successfully!${NC}" +echo "" +echo "Next steps:" +echo " 1. Label the node if needed: kubectl label nodes node-role.kubernetes.io/worker=worker" +echo " 2. Monitor pod scheduling: kubectl get pods --all-namespaces -o wide | grep " +echo " 3. Verify flannel annotation: kubectl get node -o yaml | grep flannel"