From a30274d1eb3e0a73abb8eb417ffaf1aaa249d2ed Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 30 Aug 2026 09:41:22 +0000 Subject: [PATCH] feat(os-update): integrate control planes into weekly rolling cycle (DEV-553) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit os-update.sh now walks all three CPs after workers, one at a time, etcd followers first and leader last. Each CP is delegated to update-cp-node.sh --run with ASSUME_YES=1; a health gate (cluster-health.sh + etcdctl endpoint status) runs between CPs. Adds --skip-cps and --only-cps flags; --only and --start-from now route to the correct list automatically. OS_UPDATE_PROCEDURE.md updated: CPs are no longer excluded, §8 renamed to "Control plane handling" and describes the leader-last cycle + delegation. Co-Authored-By: Paperclip --- infrastructure/OS_UPDATE_PROCEDURE.md | 41 ++-- infrastructure/scripts/os-update/os-update.sh | 188 +++++++++++++++--- 2 files changed, 188 insertions(+), 41 deletions(-) diff --git a/infrastructure/OS_UPDATE_PROCEDURE.md b/infrastructure/OS_UPDATE_PROCEDURE.md index bc4459f..42b63f3 100644 --- a/infrastructure/OS_UPDATE_PROCEDURE.md +++ b/infrastructure/OS_UPDATE_PROCEDURE.md @@ -1,6 +1,6 @@ # Weekly Rolling Ubuntu OS Update Procedure -**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node. +**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node. Since [DEV-553](/DEV/issues/DEV-553) (2026-08-30) this covers **workers AND control-plane nodes** in a single cycle; workers run first, then CPs one at a time with etcd leader last. **Audience:** Agents (currently CTO, optionally a dedicated ClusterOps agent). Also usable manually by an operator. @@ -22,9 +22,9 @@ | Role | Node | Private IP | Public IP | Datacenter | Notes | |------|------|-----------|-----------|------------|-------| -| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | EXCLUDED from `os-update.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) | -| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) | -| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) | +| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) | +| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) | +| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) | | worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | | | worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | | | worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | | @@ -38,11 +38,19 @@ Always re-derive the live list before running — nodes may have been added/remo ssh root@178.105.17.239 'kubectl get nodes -o wide' ``` -**Order rule:** update workers only. Within workers, update in this order to protect stateful workloads: +**Order rule:** update workers first, then control-plane nodes. Within each group, update in this order to protect stateful workloads: + +Workers: 1. runner + workers that host no PVs (safest — lowest disruption) 2. remaining workers 3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time. -4. **ALL control-plane nodes excluded** — `os-update.sh` selects nodes without the `node-role.kubernetes.io/control-plane` label, so `k3s-cp-1`, `k3s-cp-2`, and `k3s-cp-3` are all skipped automatically. CPs have their own procedure ([`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) — HA-aware, one CP at a time, leader last). + +Control planes (integrated with the weekly cycle since [DEV-553](/DEV/issues/DEV-553); previously excluded and run manually via `CP_UPDATE_PROCEDURE.md`): +1. `os-update.sh` queries the current etcd leader before entering the CP phase. +2. Followers are updated first (in any order), leader last. Rationale: draining a follower is a no-op for raft leadership; draining the leader forces a re-election. Doing followers first minimises leader flapping. +3. Exactly one CP is drained or rebooting at a time — two CPs down means etcd quorum loss and a write outage across the cluster. +4. Between CPs `os-update.sh` re-runs `cluster-health.sh` and re-checks `etcdctl endpoint status --cluster`; any yellow signal halts the cycle at that CP. Settle wait between CPs is `POST_CP_WAIT_SECONDS` (default 180 s). +5. Each CP goes through `update-cp-node.sh --run` with `ASSUME_YES=1`: Phase A (add-swap, idempotent) → B preflight → C drain → D apt → E reboot with `/livez` polling and etcd rejoin check → F uncordon + cluster health. **Concurrency:** exactly one node at a time. Never in parallel. @@ -249,15 +257,19 @@ Only proceed to the next node when ALL of the above are green. If not: - Give it another 3 minutes and re-check (workloads with large images may still be pulling). - If still not green: STOP the cycle. Uncordon everything, leave the cluster in a stable state, and open a follow-up issue with the failing workloads. Do NOT proceed to update more nodes. -### 8. Control plane special handling +### 8. Control plane handling -All three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) are **NOT** updated by the weekly `os-update.sh` cycle — they have their own dedicated procedure and script: [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) and `scripts/os-update/update-cp-node.sh`. Reasons: +Since [DEV-553](/DEV/issues/DEV-553) (2026-08-30) the weekly `os-update.sh` cycle **includes** the three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) after all workers finish. The CP steps are delegated to `scripts/os-update/update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) for the full design. Key properties: -- CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. One CP at a time, leader last. -- All three CPs currently have zero swap. The CP procedure adds a durable 4 GiB swapfile per CP before draining. -- The CP procedure polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined. +- CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. `os-update.sh` walks CPs sequentially with a health gate between each. +- `os-update.sh` queries `etcdctl endpoint status --cluster` before the CP phase to identify the current leader, then updates followers first and leader last. +- `update-cp-node.sh` polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined etcd as `started`. +- Phase A (`--add-swap`) is idempotent: creates `/swapfile` (4 GiB default) with `vm.swappiness=10` on any CP without swap, and installs `etcd-client` if `etcdctl` is missing. Re-runs are no-ops when swap is already active. -`os-update.sh` excludes any node carrying the `node-role.kubernetes.io/control-plane` label (see `--dry-run` output) and prints a pointer to the CP procedure. +Skip flags on `os-update.sh`: +- `--skip-cps` — workers only (legacy behavior for narrow patch cycles). +- `--only-cps` — skip workers and only walk the CP phase. +- `--only ` — single node, routed automatically to worker or CP. --- @@ -318,8 +330,8 @@ If a node was skipped or errored → mark `blocked` with the unblock action, or ## Automation entry points -- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from `. Use `--dry-run` to print the plan without touching anything. Explicitly skips `k3s-cp-1`. -- `infrastructure/scripts/os-update/update-node.sh ` — single-node update (all 7 per-node steps). Callable standalone for retry. Not for `k3s-cp-1`. +- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → worker loop → CP loop → finalization). Idempotent, resumable via `--start-from ` (works for workers or CPs). Use `--dry-run` to print the plan without touching anything. Use `--skip-cps` / `--only-cps` to run only one half of the cycle. +- `infrastructure/scripts/os-update/update-node.sh ` — single-node update for **workers** (all 7 per-node steps). Callable standalone for retry. Never call this for a CP. - `infrastructure/scripts/os-update/update-cp-node.sh ` — HA-aware CP update flow (add swap, preflight, drain, apt, reboot, finalize). Accepts `k3s-cp-1`, `k3s-cp-2`, or `k3s-cp-3`. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md). - `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compat. - `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure. @@ -341,3 +353,4 @@ A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this proce | 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) | | 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) | | 2026-08-23 | Downgraded the kine cascade section for the HA-etcd era (DEV-510 migrated cp-1 from kine SQLite to embedded etcd, cp-2/cp-3 joined as full members). Pointed §8 and the "Automation entry points" list at the new HA-aware `CP_UPDATE_PROCEDURE.md` / `update-cp-node.sh`. Old `CP1_UPDATE_PROCEDURE.md` and `update-cp-1.sh` are now redirect stubs. | CTO agent (DEV-515) | +| 2026-08-30 | Integrated the three CPs into the weekly `os-update.sh` cycle (leader-last ordering, health gate between CPs, delegates each CP to `update-cp-node.sh --run` with `ASSUME_YES=1`). Added `--skip-cps` / `--only-cps` flags. `--only ` and `--start-from ` now route between worker/CP lists automatically. | CTO agent (DEV-553) | diff --git a/infrastructure/scripts/os-update/os-update.sh b/infrastructure/scripts/os-update/os-update.sh index 6013910..042e578 100755 --- a/infrastructure/scripts/os-update/os-update.sh +++ b/infrastructure/scripts/os-update/os-update.sh @@ -4,32 +4,47 @@ # Behavior: # 1. preflight cluster health (fail-closed) # 2. take an etcd snapshot -# 3. compute node order (workers first, control plane last; -# Stalwart-hosting fsn1 workers moved to end of workers group) -# 4. call update-node.sh for each node, halting on any failure +# 3. compute node order: +# workers (non-Stalwart first, Stalwart-hosting fsn1 workers last) +# then control-plane nodes (etcd followers first, leader last). +# 4. call update-node.sh for each worker and update-cp-node.sh for each CP, +# halting on any failure # 5. finalization: health snapshot + apt history digest # # Usage: # os-update.sh [--dry-run] [--start-from ] [--only ] +# [--skip-cps] [--only-cps] +# +# Flags: +# --skip-cps run workers only (legacy behavior, useful for narrow patches). +# --only-cps skip workers, only walk the CP phase. +# --only update a single node (worker or CP, routed automatically). # # Environment: # CONTROL_PLANE_HOST (default 178.105.17.239) -# All env vars honored by update-node.sh are honored here as well. +# POST_CP_WAIT_SECONDS (default 180) — settle between CPs after uncordon. +# All env vars honored by update-node.sh / update-cp-node.sh are honored here. # # Read OS_UPDATE_PROCEDURE.md alongside this script; the script mirrors it -# step-for-step and the doc is the authoritative reference. +# step-for-step and the doc is the authoritative reference. CP handling is +# delegated to update-cp-node.sh; see CP_UPDATE_PROCEDURE.md. set -euo pipefail CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}" +POST_CP_WAIT_SECONDS="${POST_CP_WAIT_SECONDS:-180}" DRY_RUN=0 START_FROM="" ONLY="" +SKIP_CPS=0 +ONLY_CPS=0 while [ $# -gt 0 ]; do case "$1" in --dry-run) DRY_RUN=1; shift ;; --start-from) START_FROM="$2"; shift 2 ;; --only) ONLY="$2"; shift 2 ;; + --skip-cps) SKIP_CPS=1; shift ;; + --only-cps) ONLY_CPS=1; shift ;; -h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "unknown arg: $1" >&2; exit 2 ;; esac @@ -68,17 +83,17 @@ fi # --- 3. node ordering --------------------------------------------------------- # Ordering rule: -# - workers only -# - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last -# - ALL control-plane nodes are EXCLUDED and never updated by this script. Rationale: -# * kine/etcd write-path is sensitive to concurrent drains (see kine thundering-herd -# guardrails in OS_UPDATE_PROCEDURE.md, added after DEV-495). -# * rebooting a CP removes one apiserver — needs external liveness monitoring. -# * CP nodes host their own StatefulSet workloads that need batched eviction. -# Since DEV-510 (2026-08-22) the cluster runs HA control plane (cp-1/cp-2/cp-3). The -# exclusion here is role-based so ALL current and future CPs are covered automatically. -# To update a CP, use `scripts/os-update/update-cp-1.sh` (currently cp-1-only; will be -# generalized to any CP as part of the HA-aware CP OS-update procedure follow-up). +# - workers first (non-Stalwart, then Stalwart-hosting fsn1 last) +# - control-plane nodes AFTER workers, one at a time, etcd followers first, +# leader last. Rationale (see CP_UPDATE_PROCEDURE.md): +# * two of three CPs down = etcd quorum loss = write outage. Only one CP +# cordoned/rebooting at a time. +# * draining a follower is a no-op for raft leadership; draining the leader +# forces a re-election, so doing followers first minimises leader flapping. +# Since DEV-510 (2026-08-22) the cluster runs HA embedded etcd across +# cp-1/cp-2/cp-3 with 3-node quorum. This script delegates all CP steps to +# update-cp-node.sh, which handles swap (idempotent), preflight, drain, apt, +# reboot with /livez polling, etcd rejoin check, and finalize. STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true) # If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers. @@ -108,38 +123,111 @@ for n in $ALL_NODES; do fi done -ORDER=("${workers[@]}" "${stalwart_workers[@]}") -# NOTE: CP nodes intentionally excluded. To update a CP, see CP1_UPDATE_PROCEDURE.md. +WORKER_ORDER=("${workers[@]}" "${stalwart_workers[@]}") + +# --- 3b. control-plane ordering (leader last) -------------------------------- +# CP-node → public IP map. Kept out-of-cluster so a full-cluster incident +# doesn't hide etcd from the operator machine. +declare -A CP_HOST=( + [k3s-cp-1]="178.105.17.239" + [k3s-cp-2]="188.245.85.199" + [k3s-cp-3]="49.13.92.162" +) + +etcd_leader_ip() { + # Query etcd leader endpoint from the operator's control-plane host. Uses + # the same TLS material as update-cp-node.sh. Prints "IP" on success, + # nothing on failure. + ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new -o BatchMode=yes \ + root@"$CONTROL_PLANE_HOST" \ + 'ETCDCTL_API=3 etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \ + --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \ + --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \ + endpoint status --cluster -w simple 2>/dev/null' \ + | awk -F, '$5 ~ /true/ {print $1}' \ + | sed -E 's|https?://([^:]+):.*|\1|' +} + +CP_ORDER=() if [ -n "$CP_NODES" ]; then - log "[plan] EXCLUDING control-plane nodes: $(echo $CP_NODES | tr '\n' ' ')— use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)" + LEADER_IP="$(etcd_leader_ip || true)" + leader_node="" + followers=() + for n in $CP_NODES; do + ip="${CP_HOST[$n]:-}" + if [ -n "$ip" ] && [ "$ip" = "$LEADER_IP" ]; then + leader_node="$n" + else + followers+=("$n") + fi + done + if [ -n "$leader_node" ]; then + CP_ORDER=("${followers[@]}" "$leader_node") + log "[plan] etcd leader is $leader_node ($LEADER_IP) — updating followers first, leader last" + else + # Couldn't identify leader — refuse to guess. Fall back to no CPs. + CP_ORDER=() + log "[plan] WARN could not identify etcd leader — skipping CP phase this cycle" + fi fi +# Route --only automatically to worker or CP list. if [ -n "$ONLY" ]; then - ORDER=("$ONLY") + if is_cp "$ONLY"; then + CP_ORDER=("$ONLY") + WORKER_ORDER=() + else + WORKER_ORDER=("$ONLY") + CP_ORDER=() + fi elif [ -n "$START_FROM" ]; then new=() skip=1 - for n in "${ORDER[@]}"; do + for n in "${WORKER_ORDER[@]}"; do [ "$n" = "$START_FROM" ] && skip=0 [ $skip -eq 0 ] && new+=("$n") done - ORDER=("${new[@]}") + WORKER_ORDER=("${new[@]}") + # If START_FROM is a CP, trim CP_ORDER instead. + if is_cp "$START_FROM"; then + new_cp=() + skip=1 + for n in "${CP_ORDER[@]:-}"; do + [ -z "$n" ] && continue + [ "$n" = "$START_FROM" ] && skip=0 + [ $skip -eq 0 ] && new_cp+=("$n") + done + CP_ORDER=("${new_cp[@]}") + WORKER_ORDER=() + fi fi -log "[plan] ordered nodes (${#ORDER[@]}): ${ORDER[*]}" -printf '%s\n' "${ORDER[@]}" > "$LOG_DIR/plan.txt" +[ "$SKIP_CPS" -eq 1 ] && CP_ORDER=() +[ "$ONLY_CPS" -eq 1 ] && WORKER_ORDER=() + +log "[plan] workers (${#WORKER_ORDER[@]}): ${WORKER_ORDER[*]:-none}" +log "[plan] control-planes (${#CP_ORDER[@]}): ${CP_ORDER[*]:-none}" +{ + echo "# workers" + printf '%s\n' "${WORKER_ORDER[@]:-}" + echo "# control-planes" + printf '%s\n' "${CP_ORDER[@]:-}" +} > "$LOG_DIR/plan.txt" if [ "$DRY_RUN" -eq 1 ]; then log "DRY-RUN — plan written, no node touched. Exiting." exit 0 fi -# --- 4. per-node loop --------------------------------------------------------- +# --- 4. worker loop ---------------------------------------------------------- updated=() skipped=() -for n in "${ORDER[@]}"; do +for n in "${WORKER_ORDER[@]}"; do + [ -z "$n" ] && continue log "====================================================================" - log "==> updating $n" + log "==> updating worker $n" log "====================================================================" NODE_LOG="$LOG_DIR/${n}.log" set +e @@ -153,6 +241,50 @@ for n in "${ORDER[@]}"; do esac done +# --- 4b. control-plane loop -------------------------------------------------- +# One CP at a time, followers first, leader last. Delegates to update-cp-node.sh +# --run (Phase A add-swap is idempotent, Phase B..F run with ASSUME_YES=1). +# Between CPs we run cluster-health.sh and re-verify etcd cluster status; halt +# on any yellow signal. +if [ ${#CP_ORDER[@]} -gt 0 ]; then + log "====================================================================" + log "==> control-plane phase (${#CP_ORDER[@]} nodes): ${CP_ORDER[*]}" + log "====================================================================" + first_cp=1 + for cp in "${CP_ORDER[@]}"; do + [ -z "$cp" ] && continue + if [ $first_cp -eq 0 ]; then + log "-- health gate between CPs (settle ${POST_CP_WAIT_SECONDS}s, then re-check)" + sleep "$POST_CP_WAIT_SECONDS" + if ! RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh" 2>&1 | tee "$LOG_DIR/inter-cp-health-before-${cp}.log"; then + die "cluster health degraded between CPs — halting before $cp. See $LOG_DIR/" + fi + # Re-check etcd cluster status; every member must still be started. + ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new root@"$CONTROL_PLANE_HOST" \ + 'ETCDCTL_API=3 etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \ + --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \ + --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \ + endpoint status --cluster -w table' 2>&1 | tee -a "$LOG_DIR/inter-cp-health-before-${cp}.log" || \ + die "etcd endpoint status failed between CPs — halting before $cp" + fi + log "====================================================================" + log "==> updating control-plane $cp" + log "====================================================================" + CP_LOG="$LOG_DIR/${cp}.log" + set +e + ASSUME_YES=1 "$SCRIPT_DIR/update-cp-node.sh" "$cp" --run 2>&1 | tee "$CP_LOG" + rc=${PIPESTATUS[0]} + set -e + case $rc in + 0) updated+=("$cp") ;; + *) die "update-cp-node.sh failed for $cp (rc=$rc). Cycle halted. See $CP_LOG" ;; + esac + first_cp=0 + done +fi + # --- 5. finalization ---------------------------------------------------------- log "====================================================================" log "==> finalization" @@ -169,6 +301,8 @@ log "[final] apt history digest" # resolve ssh target via a mini-eval of node_ssh_target-equivalent case "$n" in k3s-cp-1) t="root@178.105.17.239" ;; + k3s-cp-2) t="root@188.245.85.199" ;; + k3s-cp-3) t="root@49.13.92.162" ;; k3s-worker-1) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;; k3s-worker-2) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;; k3s-worker-3) t="root@167.233.121.121" ;;