stack.basicstack.de/infrastructure/k3s-upgrade/README.md

# k3s System Upgrade Controller

This directory contains configuration for automated k3s cluster upgrades using the [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller).

## Installation

The system-upgrade-controller should already be installed. If not, install it with:

```bash
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
```

Verify installation:
```bash
kubectl get pods -n system-upgrade
kubectl get plans -n system-upgrade
```

## Upgrade Plans

Two upgrade plans are configured:
- `k3s-server.yaml` - Upgrades control plane nodes
- `k3s-agent.yaml` - Upgrades worker nodes

## Triggering an Upgrade

To upgrade the cluster to a new k3s version:

1. **Edit the plans to set the desired version:**
   ```bash
   kubectl edit plan k3s-server -n system-upgrade
   kubectl edit plan k3s-agent -n system-upgrade
   ```
   
   Change the `version` field:
   ```yaml
   spec:
     version: v1.37.0+k3s1  # Update to desired version
   ```

2. **Or apply updated plan files:**
   ```bash
   # Update version in the YAML files first
   kubectl apply -f infrastructure/k3s-upgrade/k3s-server.yaml
   kubectl apply -f infrastructure/k3s-upgrade/k3s-agent.yaml
   ```

3. **Monitor the upgrade:**
   ```bash
   # Watch nodes being upgraded
   kubectl get nodes -w
   
   # Check upgrade jobs
   kubectl get jobs -n system-upgrade
   
   # View controller logs
   kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller -f
   ```

## How It Works

1. The controller watches for Plan updates
2. When a new version is detected, it creates Jobs for each matching node
3. Nodes are cordoned and drained before upgrade
4. The upgrade job runs `k3s-upgrade` script to update k3s
5. Node is rebooted (if configured) and uncordoned
6. Process repeats for next node (respecting concurrency limits)

## Configuration Options

Key fields in the Plan spec:

- **version**: Target k3s version (e.g., `v1.36.2+k3s1`)
- **concurrency**: Number of nodes to upgrade simultaneously (default: 1)
- **nodeSelector**: Which nodes to upgrade (e.g., control-plane, worker)
- **cordon**: Cordon nodes before upgrade (default: true)
- **drain**: Drain pods before upgrade (default: true)
- **upgrade.cattle.io/tolerations**: Allow upgrade pods on tainted nodes

## Safety

- Upgrades are **rolling** - one node at a time by default
- Nodes are **drained** before upgrade to avoid pod disruption
- The controller respects **Pod Disruption Budgets**
- Failed upgrades can be **rolled back** by changing the version back

## Troubleshooting

### Upgrade stuck

Check job status:
```bash
kubectl get jobs -n system-upgrade
kubectl describe job <job-name> -n system-upgrade
kubectl logs -n system-upgrade job/<job-name>
```

### Plan not triggering

Ensure the version changed:
```bash
kubectl get plan k3s-server -n system-upgrade -o yaml | grep version
```

Check controller logs:
```bash
kubectl logs -n system-upgrade -l upgrade.cattle.io/controller=system-upgrade-controller
```

### Manual intervention needed

Delete stuck jobs:
```bash
kubectl delete job <job-name> -n system-upgrade
```

Uncordon nodes manually if needed:
```bash
kubectl uncordon <node-name>
```

## References

- [system-upgrade-controller Documentation](https://github.com/rancher/system-upgrade-controller)
- [k3s Automated Upgrades](https://docs.k3s.io/upgrades/automated)
- [k3s Releases](https://github.com/k3s-io/k3s/releases)