basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Add weekly rolling OS-update procedure for k3s nodes (DEV-462)

Browse source 
- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update
  procedure (drain -> apt -> reboot -> verify -> uncordon -> health ->
  next). Explicit MUST NOT list around k3s config, PVs, and manifests.
- infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip
  routine (Sun 03:00 Europe/Berlin) that fires this procedure.
- infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh,
  os-update.sh, README. Enforces the same guardrails in code:
  workers-first-then-CP, one node at a time, no --force drains, halts on
  reboot/kubelet/health failure, never touches k3s config.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO 2026-08-09 15:55:39 +00:00
parent 73430bf378
commit 3b4146de65
7 changed files with 902 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

309
infrastructure/OS_UPDATE_PROCEDURE.md Normal file
View file

@ -0,0 +1,309 @@
# Weekly Rolling Ubuntu OS Update Procedure
**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node.
**Audience:** Agents (currently CTO, optionally a dedicated ClusterOps agent). Also usable manually by an operator.
**Scope — what this procedure does:**
- Runs `apt-get update`, `apt-get -y upgrade`, `apt-get -y dist-upgrade`, `apt-get -y autoremove` on each cluster node.
- Reboots the node if a reboot is required (kernel/libc updates).
- Drains and cordons each node before touching it, uncordons after verification.
- Verifies the node and cluster are healthy before moving on.
**Scope — what this procedure MUST NOT do:**
- **Never change the Kubernetes / k3s setup.** Do not touch `/etc/systemd/system/k3s*.service*`, `/etc/rancher/k3s/*`, k3s binary version, k3s config, kube-system manifests, network policies, or any manifest under `infrastructure/`, `apps/`, or applied via ArgoCD.
- **Do not upgrade k3s** here. k3s upgrades are handled separately by system-upgrade-controller (see `K3S_OPERATIONS.md` and `k3s-upgrade/`).
- **Do not delete PersistentVolumes, PVCs, or workload manifests.**
- Do not "fix" application-level problems on a node — that's out of scope. Only fix problems in the OS/apt/reboot layer of the current node being updated. Anything else → stop, report, escalate.
---
## Cluster topology (context)
| Role | Node | Private IP | Public IP | Datacenter | Notes |
|------|------|-----------|-----------|------------|-------|
| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | update LAST |
| worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | |
| worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | |
| worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | |
| worker | k3s-worker-4 | 10.42.1.6 | 128.140.3.80 | nbg1 | |
| worker | k3s-worker-5 | 10.42.1.7 | 167.233.192.86 | fsn1 | |
| runner | k3s-update-runner | 167.233.79.65 | 167.233.79.65 | fsn1 | k3s-upgrade helper, still an updatable node |
Always re-derive the live list before running — nodes may have been added/removed:
```bash
ssh root@178.105.17.239 'kubectl get nodes -o wide'
```
**Order rule:** update ALL workers first, control plane LAST. Within workers, update in this order to protect stateful workloads:
1. runner + workers that host no PVs (safest — lowest disruption)
2. remaining workers
3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
4. **k3s-cp-1 last** — single control plane; the API server goes away during its reboot.
**Concurrency:** exactly one node at a time. Never in parallel.
---
## Access
Prereqs are the same as `CLUSTER_ACCESS.md`:
- SSH key for `root` on every node (jump via control plane for private-IP workers).
- `kubectl` available (either from the operator machine, or by SSH-ing to the control plane and using it there).
- `hcloud` CLI configured (only needed for firewall-related recovery — not for normal runs).
Environment variables the scripts expect:
- `CONTROL_PLANE_HOST` — default `178.105.17.239`
- `CONTROL_PLANE_PRIVATE` — default `10.42.1.1`
- Drain timeout: `DRAIN_TIMEOUT_SECONDS` — default `600`
- Reboot wait: `REBOOT_MAX_WAIT_SECONDS` — default `600`
- Post-uncordon settle: `POST_UNCORDON_WAIT_SECONDS` — default `180`
---
## Preflight (run once per cycle, before touching any node)
1. **Cluster is currently healthy.** If any of the checks below fail, STOP and open an issue — do not start OS updates on an already-degraded cluster.
```bash
ssh root@$CONTROL_PLANE_HOST bash -s <<'EOF'
set -e
kubectl get nodes
echo "--- Not-ready nodes:"
kubectl get nodes -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Ready" and .status!="True")) | .metadata.name'
echo "--- Pods not Running/Completed:"
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded | grep -v 'STATUS' || echo " (none)"
echo "--- Non-Ready pods (Running but not Ready):"
kubectl get pods -A -o json | jq -r '.items[] | select(.status.phase=="Running") | select([.status.conditions[]?|select(.type=="Ready")|.status]|contains(["False"])) | "\(.metadata.namespace)/\(.metadata.name)"'
EOF
```
Only proceed if: all nodes `Ready`, no non-Running/Succeeded pods, no Running-but-not-Ready pods (small transient counts are OK — retry once and continue if it clears).
2. **Snapshot k3s datastore** (control plane only — this is a checkpoint you can restore etcd from if the control-plane reboot goes badly):
```bash
ssh root@$CONTROL_PLANE_HOST 'k3s etcd-snapshot save --name pre-os-update-$(date +%Y%m%d)'
ssh root@$CONTROL_PLANE_HOST 'k3s etcd-snapshot list | tail -5'
```
3. **List the nodes to update** and derive the ordered plan. Persist to `/tmp/os-update-plan.txt` on the control plane for auditability. See `scripts/os-update/os-update.sh` for the reference implementation of the ordering rule.
---
## Per-node procedure
Repeat for each node in the ordered plan. The reference implementation is `infrastructure/scripts/os-update/update-node.sh <node-name>`; the manual steps below are what that script does.
Throughout: every command's stdout+stderr goes to a per-run log at `/tmp/os-update-<node>-<timestamp>.log` on the control plane. Attach the log to the issue at the end.
### 1. Pre-check the node
```bash
NODE=<node-name>
kubectl get node "$NODE"
kubectl describe node "$NODE" | grep -A2 Conditions
```
Confirm: `Ready=True`, not already cordoned, no `DiskPressure`/`MemoryPressure`/`PIDPressure`.
### 2. Cordon and drain
```bash
kubectl cordon "$NODE"
kubectl drain "$NODE" \
--ignore-daemonsets \
--delete-emptydir-data \
--disable-eviction=false \
--timeout="${DRAIN_TIMEOUT_SECONDS:-600}s"
```
If drain fails on a PodDisruptionBudget:
- Do NOT force-delete pods (breaks HA guarantees).
- Log the blocking PDB, uncordon the node, mark the node as `SKIPPED_PDB` in the plan, and continue with the next node. Escalate the PDB conflict on the issue at the end.
If drain fails on a lone pod without a controller:
- Do NOT `--force`. Same as above — uncordon, mark `SKIPPED_ORPHAN_POD`, continue.
### 3. Update the OS
On the node itself:
```bash
ssh -o StrictHostKeyChecking=accept-new root@<node-ssh-target> bash -s <<'REMOTE'
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
# Refresh package lists
apt-get update
# Configure apt to keep existing config files silently (no interactive prompts)
APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
apt-get $APT_OPTS upgrade
apt-get $APT_OPTS dist-upgrade
apt-get $APT_OPTS autoremove --purge
apt-get clean
# Report reboot need
if [ -f /var/run/reboot-required ]; then
echo "REBOOT_REQUIRED=yes"
cat /var/run/reboot-required.pkgs 2>/dev/null || true
else
echo "REBOOT_REQUIRED=no"
fi
REMOTE
```
For private-IP workers, target them from the control plane (`ssh -J root@$CONTROL_PLANE_HOST root@10.42.1.X`) or run the whole block after SSH-ing to the control plane first.
Common OS-only fixes the agent MAY perform on the node if apt fails:
- `dpkg --configure -a` after an interrupted install
- `apt-get -f install` to resolve broken deps
- Free disk with `journalctl --vacuum-time=3d` or `apt-get clean` if `/` is full
- Restart a system service that is stuck (`systemctl restart <unit>`) — but NOT `k3s`, `k3s-agent`, `containerd`, `flanneld`, or any container runtime
**Never**: change k3s config, delete PVs, uninstall packages the OS didn't schedule, edit `/etc/rancher/`, or reinstall k3s. If a fix would touch any of those, stop and escalate.
### 4. Reboot if required
If `REBOOT_REQUIRED=yes`:
```bash
ssh root@<node-ssh-target> 'systemctl reboot' || true
# Wait until SSH responds again (max REBOOT_MAX_WAIT_SECONDS)
deadline=$(( $(date +%s) + ${REBOOT_MAX_WAIT_SECONDS:-600} ))
while [ $(date +%s) -lt $deadline ]; do
sleep 10
if ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new root@<node-ssh-target> 'uptime' 2>/dev/null; then
echo "Node back up"; break
fi
done
```
If the node doesn't come back within the timeout: escalate immediately. Do NOT rebuild the node or touch k3s — a rebuild requires the ADD_WORKER_NODE procedure and is a separate approved action.
### 5. Wait for k3s-agent / k3s to be ready again
```bash
# Wait for kubelet Ready condition (control plane view)
deadline=$(( $(date +%s) + 300 ))
while [ $(date +%s) -lt $deadline ]; do
READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
[ "$READY" = "True" ] && break
sleep 5
done
kubectl get node "$NODE"
```
If Ready never returns True: escalate. Do NOT change k3s config.
### 6. Uncordon
```bash
kubectl uncordon "$NODE"
```
### 7. Post-node health verification
Wait for pods to reschedule and settle, then verify:
```bash
sleep "${POST_UNCORDON_WAIT_SECONDS:-180}"
# All nodes Ready?
kubectl get nodes
kubectl get nodes -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Ready" and .status!="True")) | .metadata.name' | grep . && { echo "NOT-READY NODES"; exit 1; } || true
# Any pod not Running/Succeeded?
BAD=$(kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers 2>/dev/null | wc -l)
[ "$BAD" -gt 0 ] && { echo "BAD PODS: $BAD"; kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded; exit 1; } || true
# Any Deployment/StatefulSet under desired replicas?
kubectl get deploy -A -o json | jq -r '.items[] | select((.status.readyReplicas // 0) < (.spec.replicas // 1)) | "deploy \(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"'
kubectl get sts -A -o json | jq -r '.items[] | select((.status.readyReplicas // 0) < (.spec.replicas // 1)) | "sts \(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"'
```
Only proceed to the next node when ALL of the above are green. If not:
- Give it another 3 minutes and re-check (workloads with large images may still be pulling).
- If still not green: STOP the cycle. Uncordon everything, leave the cluster in a stable state, and open a follow-up issue with the failing workloads. Do NOT proceed to update more nodes.
### 8. Control plane special handling
`k3s-cp-1` is the single control plane node. During its reboot:
- The kube-apiserver is unavailable — kubectl commands from the operator machine will error out.
- The `kubectl` waits in step 5/7 must run from a machine that is NOT the control plane, or must be scheduled after the control plane's SSH is back and `curl -k https://localhost:6443/healthz` returns `ok`.
- Skip the drain for DaemonSet pods on the control plane (`--ignore-daemonsets` covers that), but hosted apps that scheduled onto CP (rare — verify with `kubectl get pods -A --field-selector spec.nodeName=k3s-cp-1`) will be evicted.
---
## Finalization
After all nodes are done:
1. Final cluster health snapshot (same preflight commands as at the start).
2. Print apt history summary per node so the audit trail has "what changed":
```bash
for host in <all-nodes>; do
echo "=== $host ==="
ssh root@$host 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
done
```
3. Optional: prune old etcd snapshots to keep disk in check:
```bash
ssh root@$CONTROL_PLANE_HOST 'k3s etcd-snapshot list'
# delete anything older than 14 days if desired (manual review)
```
4. Attach the per-run logs to the Paperclip issue that triggered this run.
5. Update the issue with:
- Nodes updated (list)
- Nodes skipped (list + reason)
- Any package that required manual intervention
- Reboots performed
- Final `kubectl get nodes` output
If everything is clean → mark the issue `done`.
If a node was skipped or errored → mark `blocked` with the unblock action, or open a child issue for the specific failure.
---
## Recovery / what to do when it goes wrong
**Node stuck cordoned after failure:** `kubectl uncordon <node>` — always leave the cluster back in its normal scheduling state.
**Node fails to boot after reboot:**
- Check Hetzner console for boot errors (kernel panic, initramfs).
- Try `hcloud server reboot <name>` from `hcloud` CLI.
- If the node cannot recover: escalate. Do NOT delete or rebuild the Hetzner server without board approval; that is the ADD_WORKER_NODE flow.
**k3s-agent won't start after reboot:**
- Check `journalctl -u k3s-agent -n 100`.
- Do NOT edit the k3s-agent unit file. Do NOT re-run the k3s installer.
- Escalate. This is out of scope for the OS-update procedure.
**Pods CrashLoopBackOff after node came back:**
- Not an OS-update problem to fix — the node is healthy, apt succeeded. Leave the node uncordoned, stop the cycle, and open an application-level issue.
**PDB blocked drain:**
- Never `--force` the drain. Leave the node uncordoned, skip it, note in the report which PDB blocked and which workload owns it.
**Datastore snapshot restore (last resort — CP only):**
- Only if the control plane is broken beyond repair. See `K3S_OPERATIONS.md` for the `--cluster-reset --cluster-reset-restore-path=<snapshot>` procedure. Requires board approval — do NOT execute unattended.
---
## Automation entry points
- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything.
- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry.
- `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure.
Read the script sources for the exact behavior before running them. They mirror this procedure step for step.
---
## Weekly schedule
A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this procedure once per week. The routine creates a task whose description points here. The assigned agent reads this document and executes the automation.
---
## Change history
| Date | Change | By |
|------|--------|-----|
| 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) |

47
infrastructure/OS_UPDATE_ROUTINE.md Normal file
View file

@ -0,0 +1,47 @@
# Weekly OS Update — Paperclip Routine
**Owner:** CTO agent (until a dedicated ClusterOps agent is spun up)
**Schedule:** every **Sunday 03:00 Europe/Berlin** (low-traffic window, before Monday operations)
**Companion procedure:** [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md)
## What the routine does
Each fire creates a Paperclip task whose description points to `OS_UPDATE_PROCEDURE.md`. The assigned agent:
1. Checks out the task.
2. Reads the procedure doc.
3. Runs `infrastructure/scripts/os-update/os-update.sh` end to end.
4. Attaches per-run logs (`/tmp/os-update-<stamp>/`) to the task.
5. Closes the task `done` on success, or `blocked` with a named unblock action on skip/error.
## Routine configuration (Paperclip)
- `title`: "Weekly rolling OS update — k3s cluster nodes"
- `assigneeAgentId`: CTO (agent id `4b5f09a2-22d8-4e3d-9ac4-46d008ad1385`)
- `projectId`: `d24057b8-02ca-41cd-b7a0-6151298c6e2c` (BasicStack Phase-2)
- `goalId`: `b4dfe4a8-5c37-4f62-aad9-fa340dcd34a4` (self-hosted Kubernetes at Hetzner)
- `priority`: `medium`
- `concurrencyPolicy`: `coalesce_if_active` (a previous run still open? merge into it, don't stack)
- `catchUpPolicy`: `skip_missed` (no back-fires if Paperclip was down)
- Trigger: `schedule` — cron `0 3 * * 0`, timezone `Europe/Berlin`
The task description created by each fire is a link to the procedure doc in Forgejo, so the doc is the single source of truth even if the routine metadata drifts.
## Reassigning to a dedicated agent
If we later hire a ClusterOps agent, only two things change:
1. `PATCH /api/routines/{routineId}` — update `assigneeAgentId`.
2. Nothing in `OS_UPDATE_PROCEDURE.md` or the scripts changes; the procedure is agent-neutral.
## Pausing / disabling
To pause without losing config:
```bash
curl -X PATCH -H "Authorization: Bearer $PAPERCLIP_API_KEY" -H "Content-Type: application/json" \
-d '{"status":"paused"}' \
"$PAPERCLIP_API_URL/routines/{routineId}"
```
Resume with `{"status":"active"}`.

8
infrastructure/README.md
View file

@ -2,6 +2,14 @@
This directory contains cluster-wide infrastructure configurations that support all applications. This directory contains cluster-wide infrastructure configurations that support all applications.
## Operational procedures (agent-facing)
- **[OS_UPDATE_PROCEDURE.md](OS_UPDATE_PROCEDURE.md)** — weekly rolling Ubuntu OS update for all cluster nodes (drain → apt → reboot → verify → uncordon → cluster health → next). Never touches k3s config.
- **[OS_UPDATE_ROUTINE.md](OS_UPDATE_ROUTINE.md)** — the Paperclip routine that fires the above weekly.
- **[K3S_OPERATIONS.md](K3S_OPERATIONS.md)** — k3s version upgrades (separate concern from OS updates).
- **[ADD_WORKER_NODE.md](ADD_WORKER_NODE.md)** — adding a worker node.
- **[CLUSTER_ACCESS.md](CLUSTER_ACCESS.md)** — SSH / kubectl access.
## Structure ## Structure
### `networking/` ### `networking/`

47
infrastructure/scripts/os-update/README.md Normal file
View file

@ -0,0 +1,47 @@
# OS Update Automation
Scripts that implement the weekly rolling Ubuntu OS-update procedure.
**Authoritative doc:** [`../../OS_UPDATE_PROCEDURE.md`](../../OS_UPDATE_PROCEDURE.md) — read it first. The scripts here mirror that procedure step-for-step.
## Files
| Script | Purpose |
|--------|---------|
| `cluster-health.sh` | Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node. |
| `update-node.sh <node>` | Drain, apt-update, reboot-if-required, wait for Ready, uncordon, post-node health check. Retriable per-node. |
| `os-update.sh` | Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest. |
## Order of operations (encoded in `os-update.sh`)
1. Workers with no stateful affinity concerns first.
2. `fsn1` workers (potential Stalwart hosts) last among workers — see [`stalwart-datacenter-affinity`](../../K3S_OPERATIONS.md) note.
3. `k3s-cp-1` last (single control plane).
4. One node at a time. Never in parallel.
## Guardrails the scripts enforce
- Preflight cluster health failure → refuse to start.
- Drain with a PDB conflict → uncordon and mark the node `SKIPPED_DRAIN_FAILED`, never `--force`.
- Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config.
- kubelet doesn't return `Ready` → hard stop, escalate. Do NOT touch k3s config.
- Post-node cluster health check red → hard stop, do not proceed to the next node.
- Absolute rule: **no k3s config, no manifests, no PVs, no service files touched** — apt/dpkg only.
## Typical invocations
```bash
# Dry-run: print the ordered plan, touch nothing.
./os-update.sh --dry-run
# Full cycle (weekly, triggered by the Paperclip routine).
./os-update.sh
# Retry a single node (after fixing a manual issue).
./update-node.sh k3s-worker-3
# Restart a partial cycle from a specific node onward.
./os-update.sh --start-from k3s-worker-4
```
Logs land in `/tmp/os-update-<UTC-timestamp>/` on the machine that ran the cycle.

130
infrastructure/scripts/os-update/cluster-health.sh Executable file
View file

@ -0,0 +1,130 @@
#!/bin/bash
# cluster-health.sh — verify k3s cluster is fully healthy.
# Exits 0 on green, 1 on any failure. Used as preflight and post-node check by os-update.sh.
#
# Runs kubectl commands against whatever the current KUBECONFIG resolves to;
# invoke via `ssh root@$CONTROL_PLANE_HOST bash -s < cluster-health.sh` to
# check the cluster from an operator machine without local kubeconfig.
#
# Environment overrides:
# RETRY_ON_TRANSIENT=1 — one retry after 30s for non-Ready-but-Running pods
# VERBOSE=1 — dump full failing rows on non-zero exit
set -euo pipefail
fail=0
echo "=== nodes ==="
kubectl get nodes -o wide
not_ready=$(kubectl get nodes -o json | jq -r '
.items[]
| select(.status.conditions[] | select(.type=="Ready" and .status!="True"))
| .metadata.name
' || true)
if [ -n "$not_ready" ]; then
echo "FAIL: nodes not Ready: $not_ready"
fail=1
fi
echo
echo "=== node pressure conditions ==="
pressure=$(kubectl get nodes -o json | jq -r '
.items[]
| . as $n
| .status.conditions[]
| select(.type=="DiskPressure" or .type=="MemoryPressure" or .type=="PIDPressure")
| select(.status=="True")
| "\($n.metadata.name) \(.type)=True"
' || true)
if [ -n "$pressure" ]; then
echo "FAIL: node pressure: $pressure"
fail=1
else
echo " (none)"
fi
check_bad_pods() {
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers 2>/dev/null || true
}
check_notready_pods() {
kubectl get pods -A -o json | jq -r '
.items[]
| select(.status.phase=="Running")
| select([.status.conditions[]?|select(.type=="Ready")|.status] | contains(["False"]))
| "\(.metadata.namespace)/\(.metadata.name)"
' 2>/dev/null || true
}
echo
echo "=== pods not Running/Succeeded ==="
bad_pods=$(check_bad_pods)
if [ -n "$bad_pods" ]; then
if [ "${RETRY_ON_TRANSIENT:-0}" = "1" ]; then
echo " transient? re-checking in 30s..."
sleep 30
bad_pods=$(check_bad_pods)
fi
fi
if [ -n "$bad_pods" ]; then
echo "FAIL: pods not Running/Succeeded:"
echo "$bad_pods"
fail=1
else
echo " (none)"
fi
echo
echo "=== pods Running but not Ready ==="
notready=$(check_notready_pods)
if [ -n "$notready" ] && [ "${RETRY_ON_TRANSIENT:-0}" = "1" ]; then
echo " transient? re-checking in 30s..."
sleep 30
notready=$(check_notready_pods)
fi
if [ -n "$notready" ]; then
echo "FAIL: pods Running-but-not-Ready:"
echo "$notready"
fail=1
else
echo " (none)"
fi
echo
echo "=== deployments below desired replicas ==="
deploy_bad=$(kubectl get deploy -A -o json | jq -r '
.items[]
| select((.status.readyReplicas // 0) < (.spec.replicas // 1))
| "\(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"
' || true)
if [ -n "$deploy_bad" ]; then
echo "FAIL: deployments not fully ready:"
echo "$deploy_bad"
fail=1
else
echo " (none)"
fi
echo
echo "=== statefulsets below desired replicas ==="
sts_bad=$(kubectl get sts -A -o json | jq -r '
.items[]
| select((.status.readyReplicas // 0) < (.spec.replicas // 1))
| "\(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"
' || true)
if [ -n "$sts_bad" ]; then
echo "FAIL: statefulsets not fully ready:"
echo "$sts_bad"
fail=1
else
echo " (none)"
fi
if [ "$fail" -ne 0 ]; then
echo
echo "CLUSTER-HEALTH: FAIL"
exit 1
fi
echo
echo "CLUSTER-HEALTH: OK"

178
infrastructure/scripts/os-update/os-update.sh Executable file
View file

@ -0,0 +1,178 @@
#!/bin/bash
# os-update.sh — full weekly rolling OS-update cycle.
#
# Behavior:
# 1. preflight cluster health (fail-closed)
# 2. take an etcd snapshot
# 3. compute node order (workers first, control plane last;
# Stalwart-hosting fsn1 workers moved to end of workers group)
# 4. call update-node.sh for each node, halting on any failure
# 5. finalization: health snapshot + apt history digest
#
# Usage:
# os-update.sh [--dry-run] [--start-from <node>] [--only <node>]
#
# Environment:
# CONTROL_PLANE_HOST (default 178.105.17.239)
# All env vars honored by update-node.sh are honored here as well.
#
# Read OS_UPDATE_PROCEDURE.md alongside this script; the script mirrors it
# step-for-step and the doc is the authoritative reference.
set -euo pipefail
CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
DRY_RUN=0
START_FROM=""
ONLY=""
while [ $# -gt 0 ]; do
case "$1" in
--dry-run) DRY_RUN=1; shift ;;
--start-from) START_FROM="$2"; shift 2 ;;
--only) ONLY="$2"; shift 2 ;;
-h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown arg: $1" >&2; exit 2 ;;
esac
done
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
LOG_DIR="/tmp/os-update-${STAMP}"
mkdir -p "$LOG_DIR"
log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOG_DIR/main.log"; }
die() { log "FATAL: $*"; exit 1; }
log "=== os-update.sh cycle $STAMP ==="
log "log dir: $LOG_DIR"
# --- 0. sanity checks ---------------------------------------------------------
command -v kubectl >/dev/null || die "kubectl not on PATH"
command -v jq >/dev/null || die "jq not on PATH (needed for health checks)"
# --- 1. preflight -------------------------------------------------------------
log "[preflight] cluster health"
if ! RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh" | tee "$LOG_DIR/preflight.log"; then
die "cluster is not healthy at preflight — refuse to start OS updates"
fi
# --- 2. etcd snapshot ---------------------------------------------------------
if [ "$DRY_RUN" -eq 0 ]; then
log "[preflight] taking k3s etcd snapshot"
ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new root@"$CONTROL_PLANE_HOST" \
"k3s etcd-snapshot save --name pre-os-update-$(date +%Y%m%d)" | tee "$LOG_DIR/etcd-snapshot.log" || \
log "WARN: etcd snapshot failed — continuing but this is a risk on CP reboot"
else
log "[preflight] DRY-RUN — skipping etcd snapshot"
fi
# --- 3. node ordering ---------------------------------------------------------
# Ordering rule:
# - workers before control plane
# - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last
# - k3s-cp-1 always last
STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true)
# If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers.
CP_NAME="k3s-cp-1"
ALL_NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n')
workers=()
stalwart_workers=()
for n in $ALL_NODES; do
[ "$n" = "$CP_NAME" ] && continue
if [ -n "$STALWART_NODE" ] && [ "$n" = "$STALWART_NODE" ]; then
stalwart_workers+=("$n")
continue
fi
# Any fsn1 worker is a potential Stalwart host — schedule after non-Stalwart nodes.
loc=$(kubectl get node "$n" -o jsonpath='{.metadata.labels.csi\.hetzner\.cloud/location}' 2>/dev/null || echo "")
if [ "$loc" = "fsn1" ]; then
stalwart_workers+=("$n")
else
workers+=("$n")
fi
done
ORDER=("${workers[@]}" "${stalwart_workers[@]}" "$CP_NAME")
if [ -n "$ONLY" ]; then
ORDER=("$ONLY")
elif [ -n "$START_FROM" ]; then
new=()
skip=1
for n in "${ORDER[@]}"; do
[ "$n" = "$START_FROM" ] && skip=0
[ $skip -eq 0 ] && new+=("$n")
done
ORDER=("${new[@]}")
fi
log "[plan] ordered nodes (${#ORDER[@]}): ${ORDER[*]}"
printf '%s\n' "${ORDER[@]}" > "$LOG_DIR/plan.txt"
if [ "$DRY_RUN" -eq 1 ]; then
log "DRY-RUN — plan written, no node touched. Exiting."
exit 0
fi
# --- 4. per-node loop ---------------------------------------------------------
updated=()
skipped=()
for n in "${ORDER[@]}"; do
log "===================================================================="
log "==> updating $n"
log "===================================================================="
NODE_LOG="$LOG_DIR/${n}.log"
set +e
"$SCRIPT_DIR/update-node.sh" "$n" 2>&1 | tee "$NODE_LOG"
rc=${PIPESTATUS[0]}
set -e
case $rc in
0) updated+=("$n") ;;
3) skipped+=("$n:drain-blocked") ;;
*) die "update-node.sh failed for $n (rc=$rc). Cycle halted. See $NODE_LOG" ;;
esac
done
# --- 5. finalization ----------------------------------------------------------
log "===================================================================="
log "==> finalization"
log "===================================================================="
log "[final] cluster health"
"$SCRIPT_DIR/cluster-health.sh" | tee "$LOG_DIR/final-health.log" || \
die "final cluster health failed after cycle. Do NOT declare success."
log "[final] apt history digest"
{
for n in "${updated[@]}"; do
echo "=== $n ==="
# resolve ssh target via a mini-eval of node_ssh_target-equivalent
case "$n" in
k3s-cp-1) t="root@178.105.17.239" ;;
k3s-worker-1) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
k3s-worker-2) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
k3s-worker-3) t="root@167.233.121.121" ;;
k3s-worker-4) t="root@128.140.3.80" ;;
k3s-worker-5) t="root@167.233.192.86" ;;
k3s-update-runner) t="root@167.233.79.65" ;;
*) echo " (unknown ssh target)"; continue ;;
esac
ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new $t \
'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -40' 2>/dev/null \
|| echo " (could not read apt history)"
done
} | tee "$LOG_DIR/apt-history.log"
log "=== summary ==="
log "updated (${#updated[@]}): ${updated[*]:-none}"
log "skipped (${#skipped[@]}): ${skipped[*]:-none}"
log "logs: $LOG_DIR"
if [ ${#skipped[@]} -gt 0 ]; then
log "cycle finished with skipped nodes — return code 4 so the caller can escalate"
exit 4
fi
log "cycle complete."

183
infrastructure/scripts/os-update/update-node.sh Executable file
View file

@ -0,0 +1,183 @@
#!/bin/bash
# update-node.sh — drain, apt-update, reboot-if-needed, wait, uncordon a single node.
#
# Usage:
# update-node.sh <node-name>
#
# Runs from an operator machine (or from the control plane); needs kubectl and
# ssh access to root@<node-ssh-target>. Node → SSH target resolution is in
# `node_ssh_target` below; edit that mapping when you add nodes.
#
# NEVER touches k3s config, k3s services, containerd, or any manifest.
# Only fixes it will attempt: apt/dpkg recovery on the same node (see step 3).
#
# Environment overrides:
# DRAIN_TIMEOUT_SECONDS (default 600)
# REBOOT_MAX_WAIT_SECONDS (default 600)
# POST_UNCORDON_WAIT_SECONDS (default 180)
# CONTROL_PLANE_HOST (default 178.105.17.239)
# SSH_OPTS (default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new")
# ASSUME_YES=1 to skip interactive confirmations
set -euo pipefail
NODE="${1:-}"
if [ -z "$NODE" ]; then
echo "usage: $0 <node-name>" >&2
exit 2
fi
DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}"
REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}"
POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}"
CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}"
log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*"; }
die() { log "FATAL: $*"; exit 1; }
# node -> ssh target. Private-IP workers are reached via ProxyJump through the CP.
node_ssh_target() {
case "$1" in
k3s-cp-1) echo "root@178.105.17.239" ;;
k3s-worker-1) echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
k3s-worker-2) echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
k3s-worker-3) echo "root@167.233.121.121" ;;
k3s-worker-4) echo "root@128.140.3.80" ;;
k3s-worker-5) echo "root@167.233.192.86" ;;
k3s-update-runner) echo "root@167.233.79.65" ;;
*) die "unknown node $1 — update node_ssh_target() in $0" ;;
esac
}
SSH_TARGET=$(node_ssh_target "$NODE")
log "=== update-node.sh $NODE ==="
log "ssh target: $SSH_TARGET"
# --- 1. pre-check --------------------------------------------------------------
log "[1/7] pre-check"
kubectl get node "$NODE" >/dev/null || die "node $NODE not found in cluster"
READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
[ "$READY" = "True" ] || die "node $NODE is not Ready before we start (Ready=$READY)"
for cond in DiskPressure MemoryPressure PIDPressure; do
v=$(kubectl get node "$NODE" -o jsonpath="{.status.conditions[?(@.type==\"$cond\")].status}")
[ "$v" = "True" ] && die "node $NODE has $cond=True — refuse to update"
done
# --- 2. cordon + drain --------------------------------------------------------
log "[2/7] cordon + drain (timeout ${DRAIN_TIMEOUT_SECONDS}s)"
kubectl cordon "$NODE"
set +e
kubectl drain "$NODE" \
--ignore-daemonsets \
--delete-emptydir-data \
--timeout="${DRAIN_TIMEOUT_SECONDS}s"
DRAIN_RC=$?
set -e
if [ $DRAIN_RC -ne 0 ]; then
log "drain FAILED (rc=$DRAIN_RC). Never force. Uncordoning $NODE and marking SKIPPED."
kubectl uncordon "$NODE" || true
echo "SKIPPED_DRAIN_FAILED $NODE"
exit 3
fi
# --- 3. apt update on the node ------------------------------------------------
log "[3/7] apt update/upgrade on $NODE"
REMOTE_APT=$(cat <<'REMOTE'
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
# Recover from any half-finished dpkg state before touching apt.
if ! dpkg --audit | grep -qE .; then
:
else
echo "dpkg audit reported issues, running dpkg --configure -a"
dpkg --configure -a || true
fi
apt-get update
# Try upgrade; on broken deps, one attempt at apt-get -f install then retry.
if ! apt-get $APT_OPTS upgrade; then
echo "upgrade failed, attempting apt-get -f install"
apt-get $APT_OPTS -f install
apt-get $APT_OPTS upgrade
fi
apt-get $APT_OPTS dist-upgrade
apt-get $APT_OPTS autoremove --purge
apt-get clean
if [ -f /var/run/reboot-required ]; then
echo "REBOOT_REQUIRED=yes"
echo "REBOOT_REASON<<EOF"
cat /var/run/reboot-required.pkgs 2>/dev/null || echo "(no package list)"
echo "EOF"
else
echo "REBOOT_REQUIRED=no"
fi
REMOTE
)
APT_OUT=$(ssh $SSH_OPTS $SSH_TARGET "bash -s" <<< "$REMOTE_APT")
echo "$APT_OUT" | sed 's/^/ /'
if echo "$APT_OUT" | grep -q '^REBOOT_REQUIRED=yes'; then
REBOOT=1
else
REBOOT=0
fi
# --- 4. reboot if required ----------------------------------------------------
if [ "$REBOOT" -eq 1 ]; then
log "[4/7] reboot required — rebooting $NODE"
ssh $SSH_OPTS $SSH_TARGET 'systemctl reboot' || true
# Give SSH a moment to actually drop before we start polling.
sleep 15
deadline=$(( $(date +%s) + REBOOT_MAX_WAIT_SECONDS ))
while [ $(date +%s) -lt $deadline ]; do
if ssh $SSH_OPTS -o ConnectTimeout=5 $SSH_TARGET 'uptime' >/dev/null 2>&1; then
log " $NODE ssh is back"
break
fi
sleep 10
done
if ! ssh $SSH_OPTS -o ConnectTimeout=5 $SSH_TARGET 'uptime' >/dev/null 2>&1; then
die "node $NODE did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate"
fi
else
log "[4/7] no reboot needed"
fi
# --- 5. wait for kubelet Ready ------------------------------------------------
log "[5/7] wait for kubelet Ready on $NODE"
deadline=$(( $(date +%s) + 300 ))
while [ $(date +%s) -lt $deadline ]; do
READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
[ "$READY" = "True" ] && break
sleep 5
done
[ "$READY" = "True" ] || die "kubelet on $NODE never returned Ready — escalate (do NOT change k3s config)"
log " Ready=True"
# --- 6. uncordon --------------------------------------------------------------
log "[6/7] uncordon $NODE"
kubectl uncordon "$NODE"
# --- 7. post-node settle ------------------------------------------------------
log "[7/7] post-node settle (${POST_UNCORDON_WAIT_SECONDS}s) + health check"
sleep "$POST_UNCORDON_WAIT_SECONDS"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
if RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh"; then
log "=== $NODE update: OK ==="
else
die "cluster health failed after updating $NODE — STOP the cycle, do NOT continue"
fi