From 34350a03bfae35bb4e51adc81601d008b9fd77b6 Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 23 Aug 2026 01:39:25 +0000 Subject: [PATCH] os-update: exclude all control-plane nodes by role, not just cp-1 (DEV-513) Since DEV-510 landed HA control plane (2026-08-22), the cluster runs cp-1 plus cp-2/cp-3. The previous exclusion in os-update.sh matched `k3s-cp-1` by hard-coded name only, which would have caused cp-2 and cp-3 to be treated as regular fsn1 workers and drained/rebooted without the CP-specific procedure. Fix: select the CP list from `kubectl get nodes -l node-role.kubernetes.io/control-plane` and skip any of those nodes. This covers all present and future CPs automatically. Also updated OS_UPDATE_PROCEDURE.md topology table and order rule to document that all three CPs exist and are excluded from the weekly cycle. The HA-aware CP OS-update procedure is a separate follow-up. Verified on the current cluster: [plan] EXCLUDING control-plane nodes: k3s-cp-1 k3s-cp-2 k3s-cp-3 [plan] ordered nodes (6): k3s-worker-4 k3s-update-runner k3s-worker-1 k3s-worker-2 k3s-worker-3 k3s-worker-5 Co-Authored-By: Paperclip --- infrastructure/OS_UPDATE_PROCEDURE.md | 8 ++++-- infrastructure/scripts/os-update/os-update.sh | 28 +++++++++++++------ 2 files changed, 25 insertions(+), 11 deletions(-) diff --git a/infrastructure/OS_UPDATE_PROCEDURE.md b/infrastructure/OS_UPDATE_PROCEDURE.md index ffb3809..fc6b755 100644 --- a/infrastructure/OS_UPDATE_PROCEDURE.md +++ b/infrastructure/OS_UPDATE_PROCEDURE.md @@ -22,7 +22,9 @@ | Role | Node | Private IP | Public IP | Datacenter | Notes | |------|------|-----------|-----------|------------|-------| -| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | update LAST | +| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | EXCLUDED from `os-update.sh` — see CP1_UPDATE_PROCEDURE.md | +| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | EXCLUDED — joined 2026-08-22 (DEV-510); needs HA-aware CP OS-update procedure | +| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | EXCLUDED — joined 2026-08-22 (DEV-510); needs HA-aware CP OS-update procedure | | worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | | | worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | | | worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | | @@ -36,11 +38,11 @@ Always re-derive the live list before running — nodes may have been added/remo ssh root@178.105.17.239 'kubectl get nodes -o wide' ``` -**Order rule:** update ALL workers first, control plane LAST. Within workers, update in this order to protect stateful workloads: +**Order rule:** update workers only. Within workers, update in this order to protect stateful workloads: 1. runner + workers that host no PVs (safest — lowest disruption) 2. remaining workers 3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time. -4. **k3s-cp-1 excluded** — single control plane; has its own dedicated procedure and script. See `CP1_UPDATE_PROCEDURE.md` and `scripts/os-update/update-cp-1.sh`. +4. **ALL control-plane nodes excluded** — `os-update.sh` selects nodes without the `node-role.kubernetes.io/control-plane` label, so `k3s-cp-1`, `k3s-cp-2`, and `k3s-cp-3` are all skipped automatically. CPs need their own procedure (`CP1_UPDATE_PROCEDURE.md` covers cp-1 today; an HA-aware successor is a follow-up). **Concurrency:** exactly one node at a time. Never in parallel. diff --git a/infrastructure/scripts/os-update/os-update.sh b/infrastructure/scripts/os-update/os-update.sh index 9d0adbf..6013910 100755 --- a/infrastructure/scripts/os-update/os-update.sh +++ b/infrastructure/scripts/os-update/os-update.sh @@ -70,19 +70,31 @@ fi # Ordering rule: # - workers only # - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last -# - k3s-cp-1 is EXCLUDED and never updated by this script — see CP1_UPDATE_PROCEDURE.md -# and scripts/os-update/update-cp-1.sh. The rationale is the kine thundering-herd -# guardrails documented in OS_UPDATE_PROCEDURE.md (added after DEV-495). +# - ALL control-plane nodes are EXCLUDED and never updated by this script. Rationale: +# * kine/etcd write-path is sensitive to concurrent drains (see kine thundering-herd +# guardrails in OS_UPDATE_PROCEDURE.md, added after DEV-495). +# * rebooting a CP removes one apiserver — needs external liveness monitoring. +# * CP nodes host their own StatefulSet workloads that need batched eviction. +# Since DEV-510 (2026-08-22) the cluster runs HA control plane (cp-1/cp-2/cp-3). The +# exclusion here is role-based so ALL current and future CPs are covered automatically. +# To update a CP, use `scripts/os-update/update-cp-1.sh` (currently cp-1-only; will be +# generalized to any CP as part of the HA-aware CP OS-update procedure follow-up). STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true) # If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers. -CP_NAME="k3s-cp-1" +CP_NODES=$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[*].metadata.name}' 2>/dev/null | tr ' ' '\n' || true) ALL_NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n') +is_cp() { + local n="$1" + for c in $CP_NODES; do [ "$c" = "$n" ] && return 0; done + return 1 +} + workers=() stalwart_workers=() for n in $ALL_NODES; do - [ "$n" = "$CP_NAME" ] && continue + is_cp "$n" && continue if [ -n "$STALWART_NODE" ] && [ "$n" = "$STALWART_NODE" ]; then stalwart_workers+=("$n") continue @@ -97,9 +109,9 @@ for n in $ALL_NODES; do done ORDER=("${workers[@]}" "${stalwart_workers[@]}") -# NOTE: cp-1 intentionally excluded. To update cp-1, run `scripts/os-update/update-cp-1.sh`. -if printf '%s\n' "$ALL_NODES" | grep -qx "$CP_NAME"; then - log "[plan] EXCLUDING $CP_NAME — use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)" +# NOTE: CP nodes intentionally excluded. To update a CP, see CP1_UPDATE_PROCEDURE.md. +if [ -n "$CP_NODES" ]; then + log "[plan] EXCLUDING control-plane nodes: $(echo $CP_NODES | tr '\n' ' ')— use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)" fi if [ -n "$ONLY" ]; then