Add four hard rules to the OS_UPDATE_PROCEDURE:
1. Pre-plan drain order — no drain that evicts >3 StatefulSets at once
2. cp-1 must have swap before the cycle finishes (currently 0 swap, 3.7 GiB RAM)
3. Halt cycle if `kubectl get nodes` from cp-1 exceeds 5 s (kine slowness leading indicator)
4. cp-1 OS update is a separate design task, not part of standard os-update.sh cycle
Root cause reference: DEV-495 (worker-3 drain 2026-08-16 caused kine SQLite cascade
+ taint-eviction storm + near-OOM on cp-1; cluster self-recovered without operator
action after ~80 min).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update
procedure (drain -> apt -> reboot -> verify -> uncordon -> health ->
next). Explicit MUST NOT list around k3s config, PVs, and manifests.
- infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip
routine (Sun 03:00 Europe/Berlin) that fires this procedure.
- infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh,
os-update.sh, README. Enforces the same guardrails in code:
workers-first-then-CP, one node at a time, no --force drains, halts on
reboot/kubelet/health failure, never touches k3s config.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
