Add k3s cluster management automation and documentation
- Add k3s node provisioning script with version pinning - Add comprehensive K3S_OPERATIONS.md documentation - Add k3s system-upgrade-controller configuration This addresses DEV-221: prevents version skew issues by: 1. Enforcing version pinning when adding new nodes 2. Providing automated provisioning script 3. Setting up automated upgrades via upgrade controller 4. Documenting all cluster operations procedures Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
07e2e476ce
commit
2c0e7f22b1
5 changed files with 709 additions and 0 deletions
386
infrastructure/K3S_OPERATIONS.md
Normal file
386
infrastructure/K3S_OPERATIONS.md
Normal file
|
|
@ -0,0 +1,386 @@
|
||||||
|
# k3s Cluster Operations
|
||||||
|
|
||||||
|
## Current Cluster Status
|
||||||
|
|
||||||
|
- **Version:** v1.36.2+k3s1
|
||||||
|
- **Last Updated:** 2026-07-06
|
||||||
|
- **Control Plane:** k3s-cp-1 (178.105.17.239 / 10.42.1.1)
|
||||||
|
- **Workers:** k3s-worker-1, k3s-worker-2, k3s-worker-3, k3s-update-runner
|
||||||
|
- **Update Method:** k3s system-upgrade-controller (automated)
|
||||||
|
|
||||||
|
## Quick Reference
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check cluster version
|
||||||
|
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
|
||||||
|
|
||||||
|
# Check for version skew
|
||||||
|
VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u | wc -l)
|
||||||
|
if [ "$VERSIONS" -gt 1 ]; then echo "⚠️ Version skew detected!"; fi
|
||||||
|
|
||||||
|
# View current cluster state
|
||||||
|
kubectl get nodes -o wide
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Adding a New Worker Node
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
1. Hetzner Cloud server running Ubuntu 24.04 LTS
|
||||||
|
2. SSH access configured
|
||||||
|
3. `hcloud` CLI installed (for firewall management)
|
||||||
|
|
||||||
|
### Procedure
|
||||||
|
|
||||||
|
Use the automated provisioning script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd infrastructure/scripts
|
||||||
|
./provision-k3s-worker.sh <new-node-private-ip> <new-node-public-ip>
|
||||||
|
```
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```bash
|
||||||
|
./provision-k3s-worker.sh 10.42.1.6 167.233.121.122
|
||||||
|
```
|
||||||
|
|
||||||
|
The script will:
|
||||||
|
1. Check current cluster version
|
||||||
|
2. Prompt for Hetzner firewall update
|
||||||
|
3. Verify SSH access
|
||||||
|
4. Install k3s with matching version
|
||||||
|
5. Verify node joined successfully
|
||||||
|
|
||||||
|
### Manual Steps (if script unavailable)
|
||||||
|
|
||||||
|
1. **Get cluster version:**
|
||||||
|
```bash
|
||||||
|
CLUSTER_VERSION=$(ssh root@k3s-cp-1 'k3s --version' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+')
|
||||||
|
echo "Cluster version: v${CLUSTER_VERSION}"
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Update Hetzner firewall `fw-k3s`:**
|
||||||
|
```bash
|
||||||
|
# Add new node's public IP to firewall rules
|
||||||
|
hcloud firewall add-rule fw-k3s --direction in --protocol tcp \
|
||||||
|
--source-ips <new-node-public-ip>/32 --port any
|
||||||
|
```
|
||||||
|
|
||||||
|
**Critical:** Flannel VXLAN uses public IPs for tunnels. If the public IP is not in the firewall, pods on that node cannot reach DNS or other pods.
|
||||||
|
|
||||||
|
3. **Get K3S join token:**
|
||||||
|
```bash
|
||||||
|
K3S_TOKEN=$(ssh root@178.105.17.239 'cat /var/lib/rancher/k3s/server/node-token')
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Install k3s on new node with version pinning:**
|
||||||
|
```bash
|
||||||
|
ssh root@<new-node-ip> "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
|
||||||
|
--server https://10.42.1.1:6443 \
|
||||||
|
--token '${K3S_TOKEN}'"
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Verify node joined:**
|
||||||
|
```bash
|
||||||
|
kubectl get nodes
|
||||||
|
kubectl get node <node-name> -o yaml | grep flannel
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Upgrading the Cluster
|
||||||
|
|
||||||
|
### Upgrade Strategy
|
||||||
|
|
||||||
|
**We use k3s system-upgrade-controller for coordinated, automated upgrades.**
|
||||||
|
|
||||||
|
The upgrade controller:
|
||||||
|
- Ensures version consistency across the cluster
|
||||||
|
- Performs rolling upgrades (one node at a time)
|
||||||
|
- Respects pod disruption budgets
|
||||||
|
- Automatically drains and cordons nodes during upgrade
|
||||||
|
- Prevents version skew
|
||||||
|
|
||||||
|
### Planning an Upgrade
|
||||||
|
|
||||||
|
1. **Review k3s release notes:**
|
||||||
|
- Visit: https://github.com/k3s-io/k3s/releases
|
||||||
|
- Check for breaking changes
|
||||||
|
- Note any special upgrade instructions
|
||||||
|
|
||||||
|
2. **Check current version:**
|
||||||
|
```bash
|
||||||
|
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Schedule maintenance window** (optional for minor upgrades)
|
||||||
|
|
||||||
|
### Triggering an Upgrade
|
||||||
|
|
||||||
|
The system-upgrade-controller watches for Plan updates. To upgrade:
|
||||||
|
|
||||||
|
1. **Edit the upgrade plan:**
|
||||||
|
```bash
|
||||||
|
kubectl edit plan k3s-server -n system-upgrade
|
||||||
|
kubectl edit plan k3s-agent -n system-upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Update the version field:**
|
||||||
|
```yaml
|
||||||
|
spec:
|
||||||
|
version: v1.37.0+k3s1 # Change to desired version
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Monitor the upgrade:**
|
||||||
|
```bash
|
||||||
|
# Watch nodes being upgraded
|
||||||
|
kubectl get nodes -w
|
||||||
|
|
||||||
|
# Check upgrade jobs
|
||||||
|
kubectl get jobs -n system-upgrade
|
||||||
|
|
||||||
|
# View upgrade controller logs
|
||||||
|
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller -f
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Verify completion:**
|
||||||
|
```bash
|
||||||
|
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
|
||||||
|
kubectl get pods --all-namespaces -o wide
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manual Upgrade (Emergency Only)
|
||||||
|
|
||||||
|
If the upgrade controller is unavailable:
|
||||||
|
|
||||||
|
#### Control Plane
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh root@178.105.17.239
|
||||||
|
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh -
|
||||||
|
systemctl restart k3s
|
||||||
|
# Wait 5 minutes for control plane to stabilize
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Worker Nodes (one at a time)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Drain node
|
||||||
|
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data
|
||||||
|
|
||||||
|
# Upgrade node
|
||||||
|
ssh root@<worker-node>
|
||||||
|
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="vX.Y.Z+k3s1" sh -s - agent
|
||||||
|
systemctl restart k3s-agent
|
||||||
|
|
||||||
|
# Verify and uncordon
|
||||||
|
kubectl get nodes
|
||||||
|
kubectl uncordon <worker-node>
|
||||||
|
|
||||||
|
# Wait for pods to stabilize before proceeding to next node
|
||||||
|
kubectl get pods --all-namespaces -o wide | grep <worker-node>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Emergency Rollback
|
||||||
|
|
||||||
|
If issues occur after upgrade:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On affected node
|
||||||
|
ssh root@<node>
|
||||||
|
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.36.2+k3s1" sh -s - agent
|
||||||
|
systemctl restart k3s-agent
|
||||||
|
```
|
||||||
|
|
||||||
|
Then investigate the issue before retrying the upgrade.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## k3s System-Upgrade-Controller
|
||||||
|
|
||||||
|
### Overview
|
||||||
|
|
||||||
|
The system-upgrade-controller manages automated k3s upgrades across the cluster.
|
||||||
|
|
||||||
|
- **Namespace:** `system-upgrade`
|
||||||
|
- **Documentation:** https://github.com/k3s-io/k3s-upgrade
|
||||||
|
- **Plans:** `k3s-server` (control plane), `k3s-agent` (workers)
|
||||||
|
|
||||||
|
### Checking Upgrade Status
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# View current plans
|
||||||
|
kubectl get plans -n system-upgrade
|
||||||
|
|
||||||
|
# View upgrade jobs
|
||||||
|
kubectl get jobs -n system-upgrade
|
||||||
|
|
||||||
|
# View controller logs
|
||||||
|
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
Upgrade plans are in: `infrastructure/k3s-upgrade/`
|
||||||
|
|
||||||
|
To modify upgrade behavior:
|
||||||
|
```bash
|
||||||
|
kubectl edit plan k3s-server -n system-upgrade
|
||||||
|
kubectl edit plan k3s-agent -n system-upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
Key configuration options:
|
||||||
|
- `version`: Target k3s version
|
||||||
|
- `concurrency`: Number of nodes to upgrade simultaneously (default: 1)
|
||||||
|
- `cordon`: Cordon nodes before upgrade (default: true)
|
||||||
|
- `drain`: Drain pods before upgrade (default: true)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Monitoring and Alerts
|
||||||
|
|
||||||
|
### Version Skew Detection
|
||||||
|
|
||||||
|
Check for version inconsistencies:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u)
|
||||||
|
VERSION_COUNT=$(echo "$VERSIONS" | wc -l)
|
||||||
|
|
||||||
|
if [ "$VERSION_COUNT" -gt 1 ]; then
|
||||||
|
echo "❌ ERROR: k3s version mismatch detected!"
|
||||||
|
echo "Versions found:"
|
||||||
|
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "✅ All nodes running same k3s version: $VERSIONS"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Prometheus Alerts
|
||||||
|
|
||||||
|
If Prometheus is deployed, version skew alerts are configured in:
|
||||||
|
`infrastructure/monitoring/prometheus-alerts/k3s-version-skew.yaml`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Node Not Joining Cluster
|
||||||
|
|
||||||
|
1. **Check firewall rules:**
|
||||||
|
```bash
|
||||||
|
hcloud firewall describe fw-k3s
|
||||||
|
# Verify node's public IP is in allowed sources
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Check k3s-agent service:**
|
||||||
|
```bash
|
||||||
|
ssh root@<node>
|
||||||
|
systemctl status k3s-agent
|
||||||
|
journalctl -u k3s-agent -n 100
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Verify connectivity to control plane:**
|
||||||
|
```bash
|
||||||
|
ssh root@<node>
|
||||||
|
curl -k https://10.42.1.1:6443
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pods Not Scheduling on Node
|
||||||
|
|
||||||
|
1. **Check node status:**
|
||||||
|
```bash
|
||||||
|
kubectl describe node <node-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Check for taints:**
|
||||||
|
```bash
|
||||||
|
kubectl get node <node-name> -o jsonpath='{.spec.taints}'
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Check flannel:**
|
||||||
|
```bash
|
||||||
|
kubectl get node <node-name> -o yaml | grep flannel
|
||||||
|
kubectl logs -n kube-system -l app=flannel
|
||||||
|
```
|
||||||
|
|
||||||
|
### Version Mismatch After Node Addition
|
||||||
|
|
||||||
|
**Symptom:** New node has different k3s version than cluster.
|
||||||
|
|
||||||
|
**Fix:**
|
||||||
|
```bash
|
||||||
|
# Cordon node
|
||||||
|
kubectl cordon <node-name>
|
||||||
|
|
||||||
|
# Reinstall with correct version
|
||||||
|
CLUSTER_VERSION=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+')
|
||||||
|
|
||||||
|
ssh root@<node>
|
||||||
|
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v${CLUSTER_VERSION}" sh -s - agent \
|
||||||
|
--server https://10.42.1.1:6443 \
|
||||||
|
--token '<token>'
|
||||||
|
systemctl restart k3s-agent
|
||||||
|
|
||||||
|
# Uncordon
|
||||||
|
kubectl uncordon <node-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Always pin k3s version** when adding nodes
|
||||||
|
2. **Use the provisioning script** to ensure consistency
|
||||||
|
3. **Update firewall rules** before adding nodes (Flannel requires public IP access)
|
||||||
|
4. **Test upgrades** in a non-production environment first
|
||||||
|
5. **Monitor version consistency** regularly
|
||||||
|
6. **Use system-upgrade-controller** for all upgrades
|
||||||
|
7. **Never bypass version checks** - version skew causes networking issues
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cluster Architecture
|
||||||
|
|
||||||
|
### Network Configuration
|
||||||
|
|
||||||
|
- **Pod CIDR:** 10.244.0.0/16
|
||||||
|
- **Service CIDR:** 10.96.0.0/12
|
||||||
|
- **CNI:** Flannel (VXLAN mode)
|
||||||
|
- **Flannel Backend:** Uses **public IPs** for VXLAN tunnels (not private IPs)
|
||||||
|
|
||||||
|
### Node CIDRs
|
||||||
|
|
||||||
|
| Node | Pod CIDR | Private IP | Public IP |
|
||||||
|
|------|----------|-----------|-----------|
|
||||||
|
| k3s-cp-1 | 10.244.0.0/24 | 10.42.1.1 | 178.105.17.239 |
|
||||||
|
| k3s-worker-1 | 10.244.1.0/24 | 10.42.1.2 | - |
|
||||||
|
| k3s-worker-2 | 10.244.2.0/24 | 10.42.1.3 | - |
|
||||||
|
| k3s-update-runner | 10.244.3.0/24 | 167.233.79.65 | 167.233.79.65 |
|
||||||
|
| k3s-worker-3 | 10.244.5.0/24 | 10.42.1.5 | 167.233.121.121 |
|
||||||
|
|
||||||
|
### Critical Firewall Rule
|
||||||
|
|
||||||
|
**All nodes with public IPs must be in the `fw-k3s` firewall** to allow Flannel VXLAN tunnels. Without this, pods cannot communicate across nodes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change History
|
||||||
|
|
||||||
|
| Date | Version | Changes | By |
|
||||||
|
|------|---------|---------|-----|
|
||||||
|
| 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent |
|
||||||
|
| 2026-06-10 | v1.35.5+k3s1 | Original cluster deployment | - |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [k3s Documentation](https://docs.k3s.io/)
|
||||||
|
- [k3s Releases](https://github.com/k3s-io/k3s/releases)
|
||||||
|
- [k3s System Upgrade Controller](https://github.com/k3s-io/k3s-upgrade)
|
||||||
|
- [Hetzner Cloud Firewall](https://docs.hetzner.com/cloud/firewalls/)
|
||||||
126
infrastructure/k3s-upgrade/README.md
Normal file
126
infrastructure/k3s-upgrade/README.md
Normal file
|
|
@ -0,0 +1,126 @@
|
||||||
|
# k3s System Upgrade Controller
|
||||||
|
|
||||||
|
This directory contains configuration for automated k3s cluster upgrades using the [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller).
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
The system-upgrade-controller should already be installed. If not, install it with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify installation:
|
||||||
|
```bash
|
||||||
|
kubectl get pods -n system-upgrade
|
||||||
|
kubectl get plans -n system-upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
## Upgrade Plans
|
||||||
|
|
||||||
|
Two upgrade plans are configured:
|
||||||
|
- `k3s-server.yaml` - Upgrades control plane nodes
|
||||||
|
- `k3s-agent.yaml` - Upgrades worker nodes
|
||||||
|
|
||||||
|
## Triggering an Upgrade
|
||||||
|
|
||||||
|
To upgrade the cluster to a new k3s version:
|
||||||
|
|
||||||
|
1. **Edit the plans to set the desired version:**
|
||||||
|
```bash
|
||||||
|
kubectl edit plan k3s-server -n system-upgrade
|
||||||
|
kubectl edit plan k3s-agent -n system-upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
Change the `version` field:
|
||||||
|
```yaml
|
||||||
|
spec:
|
||||||
|
version: v1.37.0+k3s1 # Update to desired version
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Or apply updated plan files:**
|
||||||
|
```bash
|
||||||
|
# Update version in the YAML files first
|
||||||
|
kubectl apply -f infrastructure/k3s-upgrade/k3s-server.yaml
|
||||||
|
kubectl apply -f infrastructure/k3s-upgrade/k3s-agent.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Monitor the upgrade:**
|
||||||
|
```bash
|
||||||
|
# Watch nodes being upgraded
|
||||||
|
kubectl get nodes -w
|
||||||
|
|
||||||
|
# Check upgrade jobs
|
||||||
|
kubectl get jobs -n system-upgrade
|
||||||
|
|
||||||
|
# View controller logs
|
||||||
|
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller -f
|
||||||
|
```
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
1. The controller watches for Plan updates
|
||||||
|
2. When a new version is detected, it creates Jobs for each matching node
|
||||||
|
3. Nodes are cordoned and drained before upgrade
|
||||||
|
4. The upgrade job runs `k3s-upgrade` script to update k3s
|
||||||
|
5. Node is rebooted (if configured) and uncordoned
|
||||||
|
6. Process repeats for next node (respecting concurrency limits)
|
||||||
|
|
||||||
|
## Configuration Options
|
||||||
|
|
||||||
|
Key fields in the Plan spec:
|
||||||
|
|
||||||
|
- **version**: Target k3s version (e.g., `v1.36.2+k3s1`)
|
||||||
|
- **concurrency**: Number of nodes to upgrade simultaneously (default: 1)
|
||||||
|
- **nodeSelector**: Which nodes to upgrade (e.g., control-plane, worker)
|
||||||
|
- **cordon**: Cordon nodes before upgrade (default: true)
|
||||||
|
- **drain**: Drain pods before upgrade (default: true)
|
||||||
|
- **upgrade.cattle.io/tolerations**: Allow upgrade pods on tainted nodes
|
||||||
|
|
||||||
|
## Safety
|
||||||
|
|
||||||
|
- Upgrades are **rolling** - one node at a time by default
|
||||||
|
- Nodes are **drained** before upgrade to avoid pod disruption
|
||||||
|
- The controller respects **Pod Disruption Budgets**
|
||||||
|
- Failed upgrades can be **rolled back** by changing the version back
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Upgrade stuck
|
||||||
|
|
||||||
|
Check job status:
|
||||||
|
```bash
|
||||||
|
kubectl get jobs -n system-upgrade
|
||||||
|
kubectl describe job <job-name> -n system-upgrade
|
||||||
|
kubectl logs -n system-upgrade job/<job-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Plan not triggering
|
||||||
|
|
||||||
|
Ensure the version changed:
|
||||||
|
```bash
|
||||||
|
kubectl get plan k3s-server -n system-upgrade -o yaml | grep version
|
||||||
|
```
|
||||||
|
|
||||||
|
Check controller logs:
|
||||||
|
```bash
|
||||||
|
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manual intervention needed
|
||||||
|
|
||||||
|
Delete stuck jobs:
|
||||||
|
```bash
|
||||||
|
kubectl delete job <job-name> -n system-upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
Uncordon nodes manually if needed:
|
||||||
|
```bash
|
||||||
|
kubectl uncordon <node-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [system-upgrade-controller Documentation](https://github.com/rancher/system-upgrade-controller)
|
||||||
|
- [k3s Automated Upgrades](https://docs.k3s.io/upgrades/automated)
|
||||||
|
- [k3s Releases](https://github.com/k3s-io/k3s/releases)
|
||||||
40
infrastructure/k3s-upgrade/k3s-agent.yaml
Normal file
40
infrastructure/k3s-upgrade/k3s-agent.yaml
Normal file
|
|
@ -0,0 +1,40 @@
|
||||||
|
# Upgrade plan for k3s worker nodes
|
||||||
|
apiVersion: upgrade.cattle.io/v1
|
||||||
|
kind: Plan
|
||||||
|
metadata:
|
||||||
|
name: k3s-agent
|
||||||
|
namespace: system-upgrade
|
||||||
|
spec:
|
||||||
|
# Target k3s version - update this to trigger upgrades
|
||||||
|
version: v1.36.2+k3s1
|
||||||
|
|
||||||
|
# Node selector - target worker nodes (not control plane)
|
||||||
|
nodeSelector:
|
||||||
|
matchExpressions:
|
||||||
|
- key: node-role.kubernetes.io/control-plane
|
||||||
|
operator: DoesNotExist
|
||||||
|
|
||||||
|
# Service account with permissions to drain nodes
|
||||||
|
serviceAccountName: system-upgrade
|
||||||
|
|
||||||
|
# Cordon node before upgrade
|
||||||
|
cordon: true
|
||||||
|
|
||||||
|
# Drain pods before upgrade
|
||||||
|
drain:
|
||||||
|
force: true
|
||||||
|
skipWaitForDeleteTimeout: 60
|
||||||
|
timeout: 120s
|
||||||
|
deleteEmptyDir: true
|
||||||
|
ignoreDaemonSets: true
|
||||||
|
|
||||||
|
# Only upgrade workers after control plane is done
|
||||||
|
prepare:
|
||||||
|
image: rancher/k3s-upgrade
|
||||||
|
args:
|
||||||
|
- prepare
|
||||||
|
- k3s-server
|
||||||
|
|
||||||
|
# Container spec for upgrade job
|
||||||
|
upgrade:
|
||||||
|
image: rancher/k3s-upgrade
|
||||||
44
infrastructure/k3s-upgrade/k3s-server.yaml
Normal file
44
infrastructure/k3s-upgrade/k3s-server.yaml
Normal file
|
|
@ -0,0 +1,44 @@
|
||||||
|
# Upgrade plan for k3s control plane nodes
|
||||||
|
apiVersion: upgrade.cattle.io/v1
|
||||||
|
kind: Plan
|
||||||
|
metadata:
|
||||||
|
name: k3s-server
|
||||||
|
namespace: system-upgrade
|
||||||
|
spec:
|
||||||
|
# Target k3s version - update this to trigger upgrades
|
||||||
|
version: v1.36.2+k3s1
|
||||||
|
|
||||||
|
# Node selector - target control plane nodes
|
||||||
|
nodeSelector:
|
||||||
|
matchExpressions:
|
||||||
|
- key: node-role.kubernetes.io/control-plane
|
||||||
|
operator: Exists
|
||||||
|
|
||||||
|
# Service account with permissions to drain nodes
|
||||||
|
serviceAccountName: system-upgrade
|
||||||
|
|
||||||
|
# Cordon node before upgrade
|
||||||
|
cordon: true
|
||||||
|
|
||||||
|
# Drain pods before upgrade (with tolerations)
|
||||||
|
drain:
|
||||||
|
force: true
|
||||||
|
skipWaitForDeleteTimeout: 60
|
||||||
|
timeout: 120s
|
||||||
|
deleteEmptyDir: true
|
||||||
|
ignoreDaemonSets: true
|
||||||
|
|
||||||
|
# Container spec for upgrade job
|
||||||
|
upgrade:
|
||||||
|
image: rancher/k3s-upgrade
|
||||||
|
|
||||||
|
# Tolerate control plane taints
|
||||||
|
tolerations:
|
||||||
|
- key: CriticalAddonsOnly
|
||||||
|
operator: Exists
|
||||||
|
- key: node-role.kubernetes.io/control-plane
|
||||||
|
operator: Exists
|
||||||
|
effect: NoSchedule
|
||||||
|
- key: node-role.kubernetes.io/master
|
||||||
|
operator: Exists
|
||||||
|
effect: NoSchedule
|
||||||
113
infrastructure/scripts/provision-k3s-worker.sh
Executable file
113
infrastructure/scripts/provision-k3s-worker.sh
Executable file
|
|
@ -0,0 +1,113 @@
|
||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# k3s Worker Node Provisioning Script
|
||||||
|
# Ensures version consistency across the cluster
|
||||||
|
#
|
||||||
|
# Usage: ./provision-k3s-worker.sh <new-node-ip> [<new-node-public-ip>]
|
||||||
|
#
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
CONTROL_PLANE_IP="10.42.1.1"
|
||||||
|
CONTROL_PLANE_PUBLIC="178.105.17.239"
|
||||||
|
NEW_NODE_IP="${1}"
|
||||||
|
NEW_NODE_PUBLIC_IP="${2:-$NEW_NODE_IP}"
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
if [ -z "$NEW_NODE_IP" ]; then
|
||||||
|
echo -e "${RED}Error: Missing required argument${NC}"
|
||||||
|
echo "Usage: $0 <new-node-ip> [<new-node-public-ip>]"
|
||||||
|
echo ""
|
||||||
|
echo "Example: $0 10.42.1.6 167.233.121.122"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "${GREEN}=== k3s Worker Node Provisioning ===${NC}"
|
||||||
|
echo "Control Plane: $CONTROL_PLANE_PUBLIC ($CONTROL_PLANE_IP)"
|
||||||
|
echo "New Node: $NEW_NODE_IP (public: $NEW_NODE_PUBLIC_IP)"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 1: Get current cluster version
|
||||||
|
echo -e "${YELLOW}[1/6]${NC} Checking current cluster version..."
|
||||||
|
CLUSTER_VERSION=$(ssh root@${CONTROL_PLANE_PUBLIC} 'k3s --version' | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+\+k3s[0-9]+' || echo "")
|
||||||
|
|
||||||
|
if [ -z "$CLUSTER_VERSION" ]; then
|
||||||
|
echo -e "${RED}Error: Could not determine cluster version${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "${GREEN}✓${NC} Cluster running k3s v${CLUSTER_VERSION}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 2: Check Hetzner firewall
|
||||||
|
echo -e "${YELLOW}[2/6]${NC} Firewall configuration check"
|
||||||
|
echo -e "${RED}⚠ MANUAL ACTION REQUIRED${NC}"
|
||||||
|
echo "Ensure ${NEW_NODE_PUBLIC_IP} is added to Hetzner firewall 'fw-k3s'"
|
||||||
|
echo ""
|
||||||
|
echo "Command to add (if not already present):"
|
||||||
|
echo " hcloud firewall add-rule fw-k3s --direction in --protocol tcp --source-ips ${NEW_NODE_PUBLIC_IP}/32 --port any"
|
||||||
|
echo ""
|
||||||
|
read -p "Press Enter when firewall is updated..."
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 3: Verify SSH access
|
||||||
|
echo -e "${YELLOW}[3/6]${NC} Verifying SSH access to new node..."
|
||||||
|
if ! ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new root@${NEW_NODE_IP} 'echo ok' &>/dev/null; then
|
||||||
|
echo -e "${RED}Error: Cannot reach ${NEW_NODE_IP} via SSH${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo -e "${GREEN}✓${NC} SSH access confirmed"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 4: Get K3S_TOKEN
|
||||||
|
echo -e "${YELLOW}[4/6]${NC} Retrieving k3s join token..."
|
||||||
|
K3S_TOKEN=$(ssh root@${CONTROL_PLANE_PUBLIC} "cat /var/lib/rancher/k3s/server/node-token")
|
||||||
|
|
||||||
|
if [ -z "$K3S_TOKEN" ]; then
|
||||||
|
echo -e "${RED}Error: Could not retrieve K3S_TOKEN${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo -e "${GREEN}✓${NC} Token retrieved"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 5: Install k3s on new node with matching version
|
||||||
|
echo -e "${YELLOW}[5/6]${NC} Installing k3s v${CLUSTER_VERSION} on ${NEW_NODE_IP}..."
|
||||||
|
echo "This may take a few minutes..."
|
||||||
|
|
||||||
|
ssh root@${NEW_NODE_IP} "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v${CLUSTER_VERSION}' sh -s - agent \
|
||||||
|
--server https://${CONTROL_PLANE_IP}:6443 \
|
||||||
|
--token '${K3S_TOKEN}'"
|
||||||
|
|
||||||
|
if [ $? -ne 0 ]; then
|
||||||
|
echo -e "${RED}Error: k3s installation failed${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "${GREEN}✓${NC} k3s installed"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 6: Verify node joined successfully
|
||||||
|
echo -e "${YELLOW}[6/6]${NC} Verifying node joined the cluster..."
|
||||||
|
sleep 15 # Give the node time to register
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Cluster nodes:"
|
||||||
|
kubectl get nodes -o wide
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Node version details:"
|
||||||
|
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,CONTAINER-RUNTIME:.status.nodeInfo.containerRuntimeVersion
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${GREEN}✓✓✓ Node provisioned successfully!${NC}"
|
||||||
|
echo ""
|
||||||
|
echo "Next steps:"
|
||||||
|
echo " 1. Label the node if needed: kubectl label nodes <node-name> node-role.kubernetes.io/worker=worker"
|
||||||
|
echo " 2. Monitor pod scheduling: kubectl get pods --all-namespaces -o wide | grep <node-name>"
|
||||||
|
echo " 3. Verify flannel annotation: kubectl get node <node-name> -o yaml | grep flannel"
|
||||||
Loading…
Add table
Reference in a new issue