Add weekly rolling OS-update procedure for k3s nodes (DEV-462)

- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update procedure (drain -> apt -> reboot -> verify -> uncordon -> health -> next). Explicit MUST NOT list around k3s config, PVs, and manifests. - infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip routine (Sun 03:00 Europe/Berlin) that fires this procedure. - infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh, os-update.sh, README. Enforces the same guardrails in code: workers-first-then-CP, one node at a time, no --force drains, halts on reboot/kubelet/health failure, never touches k3s config. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-09 15:55:39 +00:00 · 2026-08-09 15:55:39 +00:00 · 3b4146de65
commit 3b4146de65
parent 73430bf378
7 changed files with 902 additions and 0 deletions
--- a/infrastructure/OS_UPDATE_PROCEDURE.md
+++ b/infrastructure/OS_UPDATE_PROCEDURE.md
@ -0,0 +1,309 @@
+# Weekly Rolling Ubuntu OS Update Procedure
+
+**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node.
+
+**Audience:** Agents (currently CTO, optionally a dedicated ClusterOps agent). Also usable manually by an operator.
+
+**Scope — what this procedure does:**
+- Runs `apt-get update`, `apt-get -y upgrade`, `apt-get -y dist-upgrade`, `apt-get -y autoremove` on each cluster node.
+- Reboots the node if a reboot is required (kernel/libc updates).
+- Drains and cordons each node before touching it, uncordons after verification.
+- Verifies the node and cluster are healthy before moving on.
+
+**Scope — what this procedure MUST NOT do:**
+- **Never change the Kubernetes / k3s setup.** Do not touch `/etc/systemd/system/k3s*.service*`, `/etc/rancher/k3s/*`, k3s binary version, k3s config, kube-system manifests, network policies, or any manifest under `infrastructure/`, `apps/`, or applied via ArgoCD.
+- **Do not upgrade k3s** here. k3s upgrades are handled separately by system-upgrade-controller (see `K3S_OPERATIONS.md` and `k3s-upgrade/`).
+- **Do not delete PersistentVolumes, PVCs, or workload manifests.**
+- Do not "fix" application-level problems on a node — that's out of scope. Only fix problems in the OS/apt/reboot layer of the current node being updated. Anything else → stop, report, escalate.
+
+---
+
+## Cluster topology (context)
+
+| Role | Node | Private IP | Public IP | Datacenter | Notes |
+|------|------|-----------|-----------|------------|-------|
+| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | update LAST |
+| worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | |
+| worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | |
+| worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | |
+| worker | k3s-worker-4 | 10.42.1.6 | 128.140.3.80 | nbg1 | |
+| worker | k3s-worker-5 | 10.42.1.7 | 167.233.192.86 | fsn1 | |
+| runner | k3s-update-runner | 167.233.79.65 | 167.233.79.65 | fsn1 | k3s-upgrade helper, still an updatable node |
+
+Always re-derive the live list before running — nodes may have been added/removed:
+
+```bash
+ssh root@178.105.17.239 'kubectl get nodes -o wide'
+```
+
+**Order rule:** update ALL workers first, control plane LAST. Within workers, update in this order to protect stateful workloads:
+1. runner + workers that host no PVs (safest — lowest disruption)
+2. remaining workers
+3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
+4. **k3s-cp-1 last** — single control plane; the API server goes away during its reboot.
+
+**Concurrency:** exactly one node at a time. Never in parallel.
+
+---
+
+## Access
+
+Prereqs are the same as `CLUSTER_ACCESS.md`:
+- SSH key for `root` on every node (jump via control plane for private-IP workers).
+- `kubectl` available (either from the operator machine, or by SSH-ing to the control plane and using it there).
+- `hcloud` CLI configured (only needed for firewall-related recovery — not for normal runs).
+
+Environment variables the scripts expect:
+- `CONTROL_PLANE_HOST` — default `178.105.17.239`
+- `CONTROL_PLANE_PRIVATE` — default `10.42.1.1`
+- Drain timeout: `DRAIN_TIMEOUT_SECONDS` — default `600`
+- Reboot wait: `REBOOT_MAX_WAIT_SECONDS` — default `600`
+- Post-uncordon settle: `POST_UNCORDON_WAIT_SECONDS` — default `180`
+
+---
+
+## Preflight (run once per cycle, before touching any node)
+
+1. **Cluster is currently healthy.** If any of the checks below fail, STOP and open an issue — do not start OS updates on an already-degraded cluster.
+   ```bash
+   ssh root@$CONTROL_PLANE_HOST bash -s <<'EOF'
+   set -e
+   kubectl get nodes
+   echo "--- Not-ready nodes:"
+   kubectl get nodes -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Ready" and .status!="True")) | .metadata.name'
+   echo "--- Pods not Running/Completed:"
+   kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded | grep -v 'STATUS' || echo "  (none)"
+   echo "--- Non-Ready pods (Running but not Ready):"
+   kubectl get pods -A -o json | jq -r '.items[] | select(.status.phase=="Running") | select([.status.conditions[]?|select(.type=="Ready")|.status]|contains(["False"])) | "\(.metadata.namespace)/\(.metadata.name)"'
+   EOF
+   ```
+   Only proceed if: all nodes `Ready`, no non-Running/Succeeded pods, no Running-but-not-Ready pods (small transient counts are OK — retry once and continue if it clears).
+
+2. **Snapshot k3s datastore** (control plane only — this is a checkpoint you can restore etcd from if the control-plane reboot goes badly):
+   ```bash
+   ssh root@$CONTROL_PLANE_HOST 'k3s etcd-snapshot save --name pre-os-update-$(date +%Y%m%d)'
+   ssh root@$CONTROL_PLANE_HOST 'k3s etcd-snapshot list | tail -5'
+   ```
+
+3. **List the nodes to update** and derive the ordered plan. Persist to `/tmp/os-update-plan.txt` on the control plane for auditability. See `scripts/os-update/os-update.sh` for the reference implementation of the ordering rule.
+
+---
+
+## Per-node procedure
+
+Repeat for each node in the ordered plan. The reference implementation is `infrastructure/scripts/os-update/update-node.sh <node-name>`; the manual steps below are what that script does.
+
+Throughout: every command's stdout+stderr goes to a per-run log at `/tmp/os-update-<node>-<timestamp>.log` on the control plane. Attach the log to the issue at the end.
+
+### 1. Pre-check the node
+
+```bash
+NODE=<node-name>
+kubectl get node "$NODE"
+kubectl describe node "$NODE" | grep -A2 Conditions
+```
+
+Confirm: `Ready=True`, not already cordoned, no `DiskPressure`/`MemoryPressure`/`PIDPressure`.
+
+### 2. Cordon and drain
+
+```bash
+kubectl cordon "$NODE"
+kubectl drain "$NODE" \
+  --ignore-daemonsets \
+  --delete-emptydir-data \
+  --disable-eviction=false \
+  --timeout="${DRAIN_TIMEOUT_SECONDS:-600}s"
+```
+
+If drain fails on a PodDisruptionBudget:
+- Do NOT force-delete pods (breaks HA guarantees).
+- Log the blocking PDB, uncordon the node, mark the node as `SKIPPED_PDB` in the plan, and continue with the next node. Escalate the PDB conflict on the issue at the end.
+
+If drain fails on a lone pod without a controller:
+- Do NOT `--force`. Same as above — uncordon, mark `SKIPPED_ORPHAN_POD`, continue.
+
+### 3. Update the OS
+
+On the node itself:
+
+```bash
+ssh -o StrictHostKeyChecking=accept-new root@<node-ssh-target> bash -s <<'REMOTE'
+set -euo pipefail
+export DEBIAN_FRONTEND=noninteractive
+# Refresh package lists
+apt-get update
+# Configure apt to keep existing config files silently (no interactive prompts)
+APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
+apt-get $APT_OPTS upgrade
+apt-get $APT_OPTS dist-upgrade
+apt-get $APT_OPTS autoremove --purge
+apt-get clean
+# Report reboot need
+if [ -f /var/run/reboot-required ]; then
+  echo "REBOOT_REQUIRED=yes"
+  cat /var/run/reboot-required.pkgs 2>/dev/null || true
+else
+  echo "REBOOT_REQUIRED=no"
+fi
+REMOTE
+```
+
+For private-IP workers, target them from the control plane (`ssh -J root@$CONTROL_PLANE_HOST root@10.42.1.X`) or run the whole block after SSH-ing to the control plane first.
+
+Common OS-only fixes the agent MAY perform on the node if apt fails:
+- `dpkg --configure -a` after an interrupted install
+- `apt-get -f install` to resolve broken deps
+- Free disk with `journalctl --vacuum-time=3d` or `apt-get clean` if `/` is full
+- Restart a system service that is stuck (`systemctl restart <unit>`) — but NOT `k3s`, `k3s-agent`, `containerd`, `flanneld`, or any container runtime
+
+**Never**: change k3s config, delete PVs, uninstall packages the OS didn't schedule, edit `/etc/rancher/`, or reinstall k3s. If a fix would touch any of those, stop and escalate.
+
+### 4. Reboot if required
+
+If `REBOOT_REQUIRED=yes`:
+
+```bash
+ssh root@<node-ssh-target> 'systemctl reboot' || true
+# Wait until SSH responds again (max REBOOT_MAX_WAIT_SECONDS)
+deadline=$(( $(date +%s) + ${REBOOT_MAX_WAIT_SECONDS:-600} ))
+while [ $(date +%s) -lt $deadline ]; do
+  sleep 10
+  if ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new root@<node-ssh-target> 'uptime' 2>/dev/null; then
+    echo "Node back up"; break
+  fi
+done
+```
+
+If the node doesn't come back within the timeout: escalate immediately. Do NOT rebuild the node or touch k3s — a rebuild requires the ADD_WORKER_NODE procedure and is a separate approved action.
+
+### 5. Wait for k3s-agent / k3s to be ready again
+
+```bash
+# Wait for kubelet Ready condition (control plane view)
+deadline=$(( $(date +%s) + 300 ))
+while [ $(date +%s) -lt $deadline ]; do
+  READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
+  [ "$READY" = "True" ] && break
+  sleep 5
+done
+kubectl get node "$NODE"
+```
+
+If Ready never returns True: escalate. Do NOT change k3s config.
+
+### 6. Uncordon
+
+```bash
+kubectl uncordon "$NODE"
+```
+
+### 7. Post-node health verification
+
+Wait for pods to reschedule and settle, then verify:
+
+```bash
+sleep "${POST_UNCORDON_WAIT_SECONDS:-180}"
+
+# All nodes Ready?
+kubectl get nodes
+kubectl get nodes -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Ready" and .status!="True")) | .metadata.name' | grep . && { echo "NOT-READY NODES"; exit 1; } || true
+
+# Any pod not Running/Succeeded?
+BAD=$(kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers 2>/dev/null | wc -l)
+[ "$BAD" -gt 0 ] && { echo "BAD PODS: $BAD"; kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded; exit 1; } || true
+
+# Any Deployment/StatefulSet under desired replicas?
+kubectl get deploy -A -o json | jq -r '.items[] | select((.status.readyReplicas // 0) < (.spec.replicas // 1)) | "deploy \(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"'
+kubectl get sts -A -o json | jq -r '.items[] | select((.status.readyReplicas // 0) < (.spec.replicas // 1)) | "sts \(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"'
+```
+
+Only proceed to the next node when ALL of the above are green. If not:
+- Give it another 3 minutes and re-check (workloads with large images may still be pulling).
+- If still not green: STOP the cycle. Uncordon everything, leave the cluster in a stable state, and open a follow-up issue with the failing workloads. Do NOT proceed to update more nodes.
+
+### 8. Control plane special handling
+
+`k3s-cp-1` is the single control plane node. During its reboot:
+- The kube-apiserver is unavailable — kubectl commands from the operator machine will error out.
+- The `kubectl` waits in step 5/7 must run from a machine that is NOT the control plane, or must be scheduled after the control plane's SSH is back and `curl -k https://localhost:6443/healthz` returns `ok`.
+- Skip the drain for DaemonSet pods on the control plane (`--ignore-daemonsets` covers that), but hosted apps that scheduled onto CP (rare — verify with `kubectl get pods -A --field-selector spec.nodeName=k3s-cp-1`) will be evicted.
+
+---
+
+## Finalization
+
+After all nodes are done:
+
+1. Final cluster health snapshot (same preflight commands as at the start).
+2. Print apt history summary per node so the audit trail has "what changed":
+   ```bash
+   for host in <all-nodes>; do
+     echo "=== $host ==="
+     ssh root@$host 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
+   done
+   ```
+3. Optional: prune old etcd snapshots to keep disk in check:
+   ```bash
+   ssh root@$CONTROL_PLANE_HOST 'k3s etcd-snapshot list'
+   # delete anything older than 14 days if desired (manual review)
+   ```
+4. Attach the per-run logs to the Paperclip issue that triggered this run.
+5. Update the issue with:
+   - Nodes updated (list)
+   - Nodes skipped (list + reason)
+   - Any package that required manual intervention
+   - Reboots performed
+   - Final `kubectl get nodes` output
+
+If everything is clean → mark the issue `done`.
+If a node was skipped or errored → mark `blocked` with the unblock action, or open a child issue for the specific failure.
+
+---
+
+## Recovery / what to do when it goes wrong
+
+**Node stuck cordoned after failure:** `kubectl uncordon <node>` — always leave the cluster back in its normal scheduling state.
+
+**Node fails to boot after reboot:**
+- Check Hetzner console for boot errors (kernel panic, initramfs).
+- Try `hcloud server reboot <name>` from `hcloud` CLI.
+- If the node cannot recover: escalate. Do NOT delete or rebuild the Hetzner server without board approval; that is the ADD_WORKER_NODE flow.
+
+**k3s-agent won't start after reboot:**
+- Check `journalctl -u k3s-agent -n 100`.
+- Do NOT edit the k3s-agent unit file. Do NOT re-run the k3s installer.
+- Escalate. This is out of scope for the OS-update procedure.
+
+**Pods CrashLoopBackOff after node came back:**
+- Not an OS-update problem to fix — the node is healthy, apt succeeded. Leave the node uncordoned, stop the cycle, and open an application-level issue.
+
+**PDB blocked drain:**
+- Never `--force` the drain. Leave the node uncordoned, skip it, note in the report which PDB blocked and which workload owns it.
+
+**Datastore snapshot restore (last resort — CP only):**
+- Only if the control plane is broken beyond repair. See `K3S_OPERATIONS.md` for the `--cluster-reset --cluster-reset-restore-path=<snapshot>` procedure. Requires board approval — do NOT execute unattended.
+
+---
+
+## Automation entry points
+
+- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything.
+- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry.
+- `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure.
+
+Read the script sources for the exact behavior before running them. They mirror this procedure step for step.
+
+---
+
+## Weekly schedule
+
+A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this procedure once per week. The routine creates a task whose description points here. The assigned agent reads this document and executes the automation.
+
+---
+
+## Change history
+
+| Date | Change | By |
+|------|--------|-----|
+| 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) |
--- a/infrastructure/OS_UPDATE_ROUTINE.md
+++ b/infrastructure/OS_UPDATE_ROUTINE.md
@ -0,0 +1,47 @@
+# Weekly OS Update — Paperclip Routine
+
+**Owner:** CTO agent (until a dedicated ClusterOps agent is spun up)
+**Schedule:** every **Sunday 03:00 Europe/Berlin** (low-traffic window, before Monday operations)
+**Companion procedure:** [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md)
+
+## What the routine does
+
+Each fire creates a Paperclip task whose description points to `OS_UPDATE_PROCEDURE.md`. The assigned agent:
+
+1. Checks out the task.
+2. Reads the procedure doc.
+3. Runs `infrastructure/scripts/os-update/os-update.sh` end to end.
+4. Attaches per-run logs (`/tmp/os-update-<stamp>/`) to the task.
+5. Closes the task `done` on success, or `blocked` with a named unblock action on skip/error.
+
+## Routine configuration (Paperclip)
+
+- `title`: "Weekly rolling OS update — k3s cluster nodes"
+- `assigneeAgentId`: CTO (agent id `4b5f09a2-22d8-4e3d-9ac4-46d008ad1385`)
+- `projectId`: `d24057b8-02ca-41cd-b7a0-6151298c6e2c` (BasicStack Phase-2)
+- `goalId`: `b4dfe4a8-5c37-4f62-aad9-fa340dcd34a4` (self-hosted Kubernetes at Hetzner)
+- `priority`: `medium`
+- `concurrencyPolicy`: `coalesce_if_active` (a previous run still open? merge into it, don't stack)
+- `catchUpPolicy`: `skip_missed` (no back-fires if Paperclip was down)
+- Trigger: `schedule` — cron `0 3 * * 0`, timezone `Europe/Berlin`
+
+The task description created by each fire is a link to the procedure doc in Forgejo, so the doc is the single source of truth even if the routine metadata drifts.
+
+## Reassigning to a dedicated agent
+
+If we later hire a ClusterOps agent, only two things change:
+
+1. `PATCH /api/routines/{routineId}` — update `assigneeAgentId`.
+2. Nothing in `OS_UPDATE_PROCEDURE.md` or the scripts changes; the procedure is agent-neutral.
+
+## Pausing / disabling
+
+To pause without losing config:
+
+```bash
+curl -X PATCH -H "Authorization: Bearer $PAPERCLIP_API_KEY" -H "Content-Type: application/json" \
+  -d '{"status":"paused"}' \
+  "$PAPERCLIP_API_URL/routines/{routineId}"
+```
+
+Resume with `{"status":"active"}`.
--- a/infrastructure/README.md
+++ b/infrastructure/README.md
@ -2,6 +2,14 @@

 This directory contains cluster-wide infrastructure configurations that support all applications.

+## Operational procedures (agent-facing)
+
+- **[OS_UPDATE_PROCEDURE.md](OS_UPDATE_PROCEDURE.md)** — weekly rolling Ubuntu OS update for all cluster nodes (drain → apt → reboot → verify → uncordon → cluster health → next). Never touches k3s config.
+- **[OS_UPDATE_ROUTINE.md](OS_UPDATE_ROUTINE.md)** — the Paperclip routine that fires the above weekly.
+- **[K3S_OPERATIONS.md](K3S_OPERATIONS.md)** — k3s version upgrades (separate concern from OS updates).
+- **[ADD_WORKER_NODE.md](ADD_WORKER_NODE.md)** — adding a worker node.
+- **[CLUSTER_ACCESS.md](CLUSTER_ACCESS.md)** — SSH / kubectl access.
+
 ## Structure

 ### `networking/`
--- a/infrastructure/scripts/os-update/README.md
+++ b/infrastructure/scripts/os-update/README.md
@ -0,0 +1,47 @@
+# OS Update Automation
+
+Scripts that implement the weekly rolling Ubuntu OS-update procedure.
+
+**Authoritative doc:** [`../../OS_UPDATE_PROCEDURE.md`](../../OS_UPDATE_PROCEDURE.md) — read it first. The scripts here mirror that procedure step-for-step.
+
+## Files
+
+| Script | Purpose |
+|--------|---------|
+| `cluster-health.sh` | Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node. |
+| `update-node.sh <node>` | Drain, apt-update, reboot-if-required, wait for Ready, uncordon, post-node health check. Retriable per-node. |
+| `os-update.sh` | Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest. |
+
+## Order of operations (encoded in `os-update.sh`)
+
+1. Workers with no stateful affinity concerns first.
+2. `fsn1` workers (potential Stalwart hosts) last among workers — see [`stalwart-datacenter-affinity`](../../K3S_OPERATIONS.md) note.
+3. `k3s-cp-1` last (single control plane).
+4. One node at a time. Never in parallel.
+
+## Guardrails the scripts enforce
+
+- Preflight cluster health failure → refuse to start.
+- Drain with a PDB conflict → uncordon and mark the node `SKIPPED_DRAIN_FAILED`, never `--force`.
+- Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config.
+- kubelet doesn't return `Ready` → hard stop, escalate. Do NOT touch k3s config.
+- Post-node cluster health check red → hard stop, do not proceed to the next node.
+- Absolute rule: **no k3s config, no manifests, no PVs, no service files touched** — apt/dpkg only.
+
+## Typical invocations
+
+```bash
+# Dry-run: print the ordered plan, touch nothing.
+./os-update.sh --dry-run
+
+# Full cycle (weekly, triggered by the Paperclip routine).
+./os-update.sh
+
+# Retry a single node (after fixing a manual issue).
+./update-node.sh k3s-worker-3
+
+# Restart a partial cycle from a specific node onward.
+./os-update.sh --start-from k3s-worker-4
+```
+
+Logs land in `/tmp/os-update-<UTC-timestamp>/` on the machine that ran the cycle.
--- a/infrastructure/scripts/os-update/cluster-health.sh
+++ b/infrastructure/scripts/os-update/cluster-health.sh
@ -0,0 +1,130 @@
+#!/bin/bash
+# cluster-health.sh — verify k3s cluster is fully healthy.
+# Exits 0 on green, 1 on any failure. Used as preflight and post-node check by os-update.sh.
+#
+# Runs kubectl commands against whatever the current KUBECONFIG resolves to;
+# invoke via `ssh root@$CONTROL_PLANE_HOST bash -s < cluster-health.sh` to
+# check the cluster from an operator machine without local kubeconfig.
+#
+# Environment overrides:
+#   RETRY_ON_TRANSIENT=1  — one retry after 30s for non-Ready-but-Running pods
+#   VERBOSE=1             — dump full failing rows on non-zero exit
+
+set -euo pipefail
+
+fail=0
+
+echo "=== nodes ==="
+kubectl get nodes -o wide
+
+not_ready=$(kubectl get nodes -o json | jq -r '
+  .items[]
+  | select(.status.conditions[] | select(.type=="Ready" and .status!="True"))
+  | .metadata.name
+' || true)
+if [ -n "$not_ready" ]; then
+  echo "FAIL: nodes not Ready: $not_ready"
+  fail=1
+fi
+
+echo
+echo "=== node pressure conditions ==="
+pressure=$(kubectl get nodes -o json | jq -r '
+  .items[]
+  | . as $n
+  | .status.conditions[]
+  | select(.type=="DiskPressure" or .type=="MemoryPressure" or .type=="PIDPressure")
+  | select(.status=="True")
+  | "\($n.metadata.name) \(.type)=True"
+' || true)
+if [ -n "$pressure" ]; then
+  echo "FAIL: node pressure: $pressure"
+  fail=1
+else
+  echo "  (none)"
+fi
+
+check_bad_pods() {
+  kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers 2>/dev/null || true
+}
+check_notready_pods() {
+  kubectl get pods -A -o json | jq -r '
+    .items[]
+    | select(.status.phase=="Running")
+    | select([.status.conditions[]?|select(.type=="Ready")|.status] | contains(["False"]))
+    | "\(.metadata.namespace)/\(.metadata.name)"
+  ' 2>/dev/null || true
+}
+
+echo
+echo "=== pods not Running/Succeeded ==="
+bad_pods=$(check_bad_pods)
+if [ -n "$bad_pods" ]; then
+  if [ "${RETRY_ON_TRANSIENT:-0}" = "1" ]; then
+    echo "  transient? re-checking in 30s..."
+    sleep 30
+    bad_pods=$(check_bad_pods)
+  fi
+fi
+if [ -n "$bad_pods" ]; then
+  echo "FAIL: pods not Running/Succeeded:"
+  echo "$bad_pods"
+  fail=1
+else
+  echo "  (none)"
+fi
+
+echo
+echo "=== pods Running but not Ready ==="
+notready=$(check_notready_pods)
+if [ -n "$notready" ] && [ "${RETRY_ON_TRANSIENT:-0}" = "1" ]; then
+  echo "  transient? re-checking in 30s..."
+  sleep 30
+  notready=$(check_notready_pods)
+fi
+if [ -n "$notready" ]; then
+  echo "FAIL: pods Running-but-not-Ready:"
+  echo "$notready"
+  fail=1
+else
+  echo "  (none)"
+fi
+
+echo
+echo "=== deployments below desired replicas ==="
+deploy_bad=$(kubectl get deploy -A -o json | jq -r '
+  .items[]
+  | select((.status.readyReplicas // 0) < (.spec.replicas // 1))
+  | "\(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"
+' || true)
+if [ -n "$deploy_bad" ]; then
+  echo "FAIL: deployments not fully ready:"
+  echo "$deploy_bad"
+  fail=1
+else
+  echo "  (none)"
+fi
+
+echo
+echo "=== statefulsets below desired replicas ==="
+sts_bad=$(kubectl get sts -A -o json | jq -r '
+  .items[]
+  | select((.status.readyReplicas // 0) < (.spec.replicas // 1))
+  | "\(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"
+' || true)
+if [ -n "$sts_bad" ]; then
+  echo "FAIL: statefulsets not fully ready:"
+  echo "$sts_bad"
+  fail=1
+else
+  echo "  (none)"
+fi
+
+if [ "$fail" -ne 0 ]; then
+  echo
+  echo "CLUSTER-HEALTH: FAIL"
+  exit 1
+fi
+
+echo
+echo "CLUSTER-HEALTH: OK"
--- a/infrastructure/scripts/os-update/os-update.sh
+++ b/infrastructure/scripts/os-update/os-update.sh
@ -0,0 +1,178 @@
+#!/bin/bash
+# os-update.sh — full weekly rolling OS-update cycle.
+#
+# Behavior:
+#   1. preflight cluster health (fail-closed)
+#   2. take an etcd snapshot
+#   3. compute node order (workers first, control plane last;
+#      Stalwart-hosting fsn1 workers moved to end of workers group)
+#   4. call update-node.sh for each node, halting on any failure
+#   5. finalization: health snapshot + apt history digest
+#
+# Usage:
+#   os-update.sh [--dry-run] [--start-from <node>] [--only <node>]
+#
+# Environment:
+#   CONTROL_PLANE_HOST (default 178.105.17.239)
+#   All env vars honored by update-node.sh are honored here as well.
+#
+# Read OS_UPDATE_PROCEDURE.md alongside this script; the script mirrors it
+# step-for-step and the doc is the authoritative reference.
+set -euo pipefail
+
+CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
+DRY_RUN=0
+START_FROM=""
+ONLY=""
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --dry-run)     DRY_RUN=1; shift ;;
+    --start-from)  START_FROM="$2"; shift 2 ;;
+    --only)        ONLY="$2"; shift 2 ;;
+    -h|--help)     grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *)             echo "unknown arg: $1" >&2; exit 2 ;;
+  esac
+done
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+STAMP=$(date -u +%Y%m%dT%H%M%SZ)
+LOG_DIR="/tmp/os-update-${STAMP}"
+mkdir -p "$LOG_DIR"
+
+log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOG_DIR/main.log"; }
+die() { log "FATAL: $*"; exit 1; }
+
+log "=== os-update.sh cycle $STAMP ==="
+log "log dir: $LOG_DIR"
+
+# --- 0. sanity checks ---------------------------------------------------------
+command -v kubectl >/dev/null || die "kubectl not on PATH"
+command -v jq >/dev/null      || die "jq not on PATH (needed for health checks)"
+
+# --- 1. preflight -------------------------------------------------------------
+log "[preflight] cluster health"
+if ! RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh" | tee "$LOG_DIR/preflight.log"; then
+  die "cluster is not healthy at preflight — refuse to start OS updates"
+fi
+
+# --- 2. etcd snapshot ---------------------------------------------------------
+if [ "$DRY_RUN" -eq 0 ]; then
+  log "[preflight] taking k3s etcd snapshot"
+  ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new root@"$CONTROL_PLANE_HOST" \
+    "k3s etcd-snapshot save --name pre-os-update-$(date +%Y%m%d)" | tee "$LOG_DIR/etcd-snapshot.log" || \
+      log "WARN: etcd snapshot failed — continuing but this is a risk on CP reboot"
+else
+  log "[preflight] DRY-RUN — skipping etcd snapshot"
+fi
+
+# --- 3. node ordering ---------------------------------------------------------
+# Ordering rule:
+#   - workers before control plane
+#   - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last
+#   - k3s-cp-1 always last
+STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true)
+# If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers.
+CP_NAME="k3s-cp-1"
+
+ALL_NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n')
+
+workers=()
+stalwart_workers=()
+for n in $ALL_NODES; do
+  [ "$n" = "$CP_NAME" ] && continue
+  if [ -n "$STALWART_NODE" ] && [ "$n" = "$STALWART_NODE" ]; then
+    stalwart_workers+=("$n")
+    continue
+  fi
+  # Any fsn1 worker is a potential Stalwart host — schedule after non-Stalwart nodes.
+  loc=$(kubectl get node "$n" -o jsonpath='{.metadata.labels.csi\.hetzner\.cloud/location}' 2>/dev/null || echo "")
+  if [ "$loc" = "fsn1" ]; then
+    stalwart_workers+=("$n")
+  else
+    workers+=("$n")
+  fi
+done
+
+ORDER=("${workers[@]}" "${stalwart_workers[@]}" "$CP_NAME")
+
+if [ -n "$ONLY" ]; then
+  ORDER=("$ONLY")
+elif [ -n "$START_FROM" ]; then
+  new=()
+  skip=1
+  for n in "${ORDER[@]}"; do
+    [ "$n" = "$START_FROM" ] && skip=0
+    [ $skip -eq 0 ] && new+=("$n")
+  done
+  ORDER=("${new[@]}")
+fi
+
+log "[plan] ordered nodes (${#ORDER[@]}): ${ORDER[*]}"
+printf '%s\n' "${ORDER[@]}" > "$LOG_DIR/plan.txt"
+
+if [ "$DRY_RUN" -eq 1 ]; then
+  log "DRY-RUN — plan written, no node touched. Exiting."
+  exit 0
+fi
+
+# --- 4. per-node loop ---------------------------------------------------------
+updated=()
+skipped=()
+for n in "${ORDER[@]}"; do
+  log "===================================================================="
+  log "==> updating $n"
+  log "===================================================================="
+  NODE_LOG="$LOG_DIR/${n}.log"
+  set +e
+  "$SCRIPT_DIR/update-node.sh" "$n" 2>&1 | tee "$NODE_LOG"
+  rc=${PIPESTATUS[0]}
+  set -e
+  case $rc in
+    0) updated+=("$n") ;;
+    3) skipped+=("$n:drain-blocked") ;;
+    *) die "update-node.sh failed for $n (rc=$rc). Cycle halted. See $NODE_LOG" ;;
+  esac
+done
+
+# --- 5. finalization ----------------------------------------------------------
+log "===================================================================="
+log "==> finalization"
+log "===================================================================="
+
+log "[final] cluster health"
+"$SCRIPT_DIR/cluster-health.sh" | tee "$LOG_DIR/final-health.log" || \
+  die "final cluster health failed after cycle. Do NOT declare success."
+
+log "[final] apt history digest"
+{
+  for n in "${updated[@]}"; do
+    echo "=== $n ==="
+    # resolve ssh target via a mini-eval of node_ssh_target-equivalent
+    case "$n" in
+      k3s-cp-1)           t="root@178.105.17.239" ;;
+      k3s-worker-1)       t="-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
+      k3s-worker-2)       t="-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
+      k3s-worker-3)       t="root@167.233.121.121" ;;
+      k3s-worker-4)       t="root@128.140.3.80" ;;
+      k3s-worker-5)       t="root@167.233.192.86" ;;
+      k3s-update-runner)  t="root@167.233.79.65" ;;
+      *)                  echo "  (unknown ssh target)"; continue ;;
+    esac
+    ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new $t \
+      'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -40' 2>/dev/null \
+      || echo "  (could not read apt history)"
+  done
+} | tee "$LOG_DIR/apt-history.log"
+
+log "=== summary ==="
+log "updated (${#updated[@]}): ${updated[*]:-none}"
+log "skipped (${#skipped[@]}): ${skipped[*]:-none}"
+log "logs:   $LOG_DIR"
+
+if [ ${#skipped[@]} -gt 0 ]; then
+  log "cycle finished with skipped nodes — return code 4 so the caller can escalate"
+  exit 4
+fi
+
+log "cycle complete."
--- a/infrastructure/scripts/os-update/update-node.sh
+++ b/infrastructure/scripts/os-update/update-node.sh
@ -0,0 +1,183 @@
+#!/bin/bash
+# update-node.sh — drain, apt-update, reboot-if-needed, wait, uncordon a single node.
+#
+# Usage:
+#   update-node.sh <node-name>
+#
+# Runs from an operator machine (or from the control plane); needs kubectl and
+# ssh access to root@<node-ssh-target>. Node → SSH target resolution is in
+# `node_ssh_target` below; edit that mapping when you add nodes.
+#
+# NEVER touches k3s config, k3s services, containerd, or any manifest.
+# Only fixes it will attempt: apt/dpkg recovery on the same node (see step 3).
+#
+# Environment overrides:
+#   DRAIN_TIMEOUT_SECONDS (default 600)
+#   REBOOT_MAX_WAIT_SECONDS (default 600)
+#   POST_UNCORDON_WAIT_SECONDS (default 180)
+#   CONTROL_PLANE_HOST (default 178.105.17.239)
+#   SSH_OPTS (default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new")
+#   ASSUME_YES=1 to skip interactive confirmations
+set -euo pipefail
+
+NODE="${1:-}"
+if [ -z "$NODE" ]; then
+  echo "usage: $0 <node-name>" >&2
+  exit 2
+fi
+
+DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}"
+REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}"
+POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}"
+CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
+SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}"
+
+log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*"; }
+die() { log "FATAL: $*"; exit 1; }
+
+# node -> ssh target. Private-IP workers are reached via ProxyJump through the CP.
+node_ssh_target() {
+  case "$1" in
+    k3s-cp-1)            echo "root@178.105.17.239" ;;
+    k3s-worker-1)        echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
+    k3s-worker-2)        echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
+    k3s-worker-3)        echo "root@167.233.121.121" ;;
+    k3s-worker-4)        echo "root@128.140.3.80" ;;
+    k3s-worker-5)        echo "root@167.233.192.86" ;;
+    k3s-update-runner)   echo "root@167.233.79.65" ;;
+    *)                   die "unknown node $1 — update node_ssh_target() in $0" ;;
+  esac
+}
+
+SSH_TARGET=$(node_ssh_target "$NODE")
+
+log "=== update-node.sh $NODE ==="
+log "ssh target: $SSH_TARGET"
+
+# --- 1. pre-check --------------------------------------------------------------
+log "[1/7] pre-check"
+kubectl get node "$NODE" >/dev/null || die "node $NODE not found in cluster"
+READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
+[ "$READY" = "True" ] || die "node $NODE is not Ready before we start (Ready=$READY)"
+
+for cond in DiskPressure MemoryPressure PIDPressure; do
+  v=$(kubectl get node "$NODE" -o jsonpath="{.status.conditions[?(@.type==\"$cond\")].status}")
+  [ "$v" = "True" ] && die "node $NODE has $cond=True — refuse to update"
+done
+
+# --- 2. cordon + drain --------------------------------------------------------
+log "[2/7] cordon + drain (timeout ${DRAIN_TIMEOUT_SECONDS}s)"
+kubectl cordon "$NODE"
+
+set +e
+kubectl drain "$NODE" \
+  --ignore-daemonsets \
+  --delete-emptydir-data \
+  --timeout="${DRAIN_TIMEOUT_SECONDS}s"
+DRAIN_RC=$?
+set -e
+
+if [ $DRAIN_RC -ne 0 ]; then
+  log "drain FAILED (rc=$DRAIN_RC). Never force. Uncordoning $NODE and marking SKIPPED."
+  kubectl uncordon "$NODE" || true
+  echo "SKIPPED_DRAIN_FAILED $NODE"
+  exit 3
+fi
+
+# --- 3. apt update on the node ------------------------------------------------
+log "[3/7] apt update/upgrade on $NODE"
+REMOTE_APT=$(cat <<'REMOTE'
+set -euo pipefail
+export DEBIAN_FRONTEND=noninteractive
+APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
+
+# Recover from any half-finished dpkg state before touching apt.
+if ! dpkg --audit | grep -qE .; then
+  :
+else
+  echo "dpkg audit reported issues, running dpkg --configure -a"
+  dpkg --configure -a || true
+fi
+
+apt-get update
+
+# Try upgrade; on broken deps, one attempt at apt-get -f install then retry.
+if ! apt-get $APT_OPTS upgrade; then
+  echo "upgrade failed, attempting apt-get -f install"
+  apt-get $APT_OPTS -f install
+  apt-get $APT_OPTS upgrade
+fi
+
+apt-get $APT_OPTS dist-upgrade
+apt-get $APT_OPTS autoremove --purge
+apt-get clean
+
+if [ -f /var/run/reboot-required ]; then
+  echo "REBOOT_REQUIRED=yes"
+  echo "REBOOT_REASON<<EOF"
+  cat /var/run/reboot-required.pkgs 2>/dev/null || echo "(no package list)"
+  echo "EOF"
+else
+  echo "REBOOT_REQUIRED=no"
+fi
+REMOTE
+)
+
+APT_OUT=$(ssh $SSH_OPTS $SSH_TARGET "bash -s" <<< "$REMOTE_APT")
+echo "$APT_OUT" | sed 's/^/    /'
+
+if echo "$APT_OUT" | grep -q '^REBOOT_REQUIRED=yes'; then
+  REBOOT=1
+else
+  REBOOT=0
+fi
+
+# --- 4. reboot if required ----------------------------------------------------
+if [ "$REBOOT" -eq 1 ]; then
+  log "[4/7] reboot required — rebooting $NODE"
+  ssh $SSH_OPTS $SSH_TARGET 'systemctl reboot' || true
+
+  # Give SSH a moment to actually drop before we start polling.
+  sleep 15
+
+  deadline=$(( $(date +%s) + REBOOT_MAX_WAIT_SECONDS ))
+  while [ $(date +%s) -lt $deadline ]; do
+    if ssh $SSH_OPTS -o ConnectTimeout=5 $SSH_TARGET 'uptime' >/dev/null 2>&1; then
+      log "  $NODE ssh is back"
+      break
+    fi
+    sleep 10
+  done
+
+  if ! ssh $SSH_OPTS -o ConnectTimeout=5 $SSH_TARGET 'uptime' >/dev/null 2>&1; then
+    die "node $NODE did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate"
+  fi
+else
+  log "[4/7] no reboot needed"
+fi
+
+# --- 5. wait for kubelet Ready ------------------------------------------------
+log "[5/7] wait for kubelet Ready on $NODE"
+deadline=$(( $(date +%s) + 300 ))
+while [ $(date +%s) -lt $deadline ]; do
+  READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
+  [ "$READY" = "True" ] && break
+  sleep 5
+done
+[ "$READY" = "True" ] || die "kubelet on $NODE never returned Ready — escalate (do NOT change k3s config)"
+log "  Ready=True"
+
+# --- 6. uncordon --------------------------------------------------------------
+log "[6/7] uncordon $NODE"
+kubectl uncordon "$NODE"
+
+# --- 7. post-node settle ------------------------------------------------------
+log "[7/7] post-node settle (${POST_UNCORDON_WAIT_SECONDS}s) + health check"
+sleep "$POST_UNCORDON_WAIT_SECONDS"
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+if RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh"; then
+  log "=== $NODE update: OK ==="
+else
+  die "cluster health failed after updating $NODE — STOP the cycle, do NOT continue"
+fi