Since DEV-510 landed HA control plane (2026-08-22), the cluster runs cp-1
plus cp-2/cp-3. The previous exclusion in os-update.sh matched `k3s-cp-1`
by hard-coded name only, which would have caused cp-2 and cp-3 to be
treated as regular fsn1 workers and drained/rebooted without the
CP-specific procedure.
Fix: select the CP list from `kubectl get nodes -l
node-role.kubernetes.io/control-plane` and skip any of those nodes. This
covers all present and future CPs automatically.
Also updated OS_UPDATE_PROCEDURE.md topology table and order rule to
document that all three CPs exist and are excluded from the weekly
cycle. The HA-aware CP OS-update procedure is a separate follow-up.
Verified on the current cluster:
[plan] EXCLUDING control-plane nodes: k3s-cp-1 k3s-cp-2 k3s-cp-3
[plan] ordered nodes (6): k3s-worker-4 k3s-update-runner k3s-worker-1
k3s-worker-2 k3s-worker-3 k3s-worker-5
Co-Authored-By: Paperclip <noreply@paperclip.ing>
cp-1 is excluded from os-update.sh because a one-shot drain would trigger
the kine cascade documented in DEV-495. This adds:
- CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful
eviction (Phase C), drain+apt (Phase D), reboot with external livez
monitor (Phase E), uncordon+verify (Phase F), and rollback paths.
- scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the
same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports
--dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot,
--finalize, --run.
- os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1
script; kine thundering-herd guardrails preserved.
- OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure.
Execution requires separate board approval; this change is design +
dry-run artifact only.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update
procedure (drain -> apt -> reboot -> verify -> uncordon -> health ->
next). Explicit MUST NOT list around k3s config, PVs, and manifests.
- infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip
routine (Sun 03:00 Europe/Berlin) that fires this procedure.
- infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh,
os-update.sh, README. Enforces the same guardrails in code:
workers-first-then-CP, one node at a time, no --force drains, halts on
reboot/kubelet/health failure, never touches k3s config.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
