feat(os-update): integrate control planes into weekly rolling cycle (DEV-553)

os-update.sh now walks all three CPs after workers, one at a time, etcd
followers first and leader last. Each CP is delegated to update-cp-node.sh
--run with ASSUME_YES=1; a health gate (cluster-health.sh +
etcdctl endpoint status) runs between CPs. Adds --skip-cps and --only-cps
flags; --only <node> and --start-from <node> now route to the correct list
automatically.

OS_UPDATE_PROCEDURE.md updated: CPs are no longer excluded, §8 renamed to
"Control plane handling" and describes the leader-last cycle + delegation.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-08-30 09:41:22 +00:00
parent c97a2d218e
commit a30274d1eb
2 changed files with 188 additions and 41 deletions

View file

@ -1,6 +1,6 @@
# Weekly Rolling Ubuntu OS Update Procedure # Weekly Rolling Ubuntu OS Update Procedure
**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node. **Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node. Since [DEV-553](/DEV/issues/DEV-553) (2026-08-30) this covers **workers AND control-plane nodes** in a single cycle; workers run first, then CPs one at a time with etcd leader last.
**Audience:** Agents (currently CTO, optionally a dedicated ClusterOps agent). Also usable manually by an operator. **Audience:** Agents (currently CTO, optionally a dedicated ClusterOps agent). Also usable manually by an operator.
@ -22,9 +22,9 @@
| Role | Node | Private IP | Public IP | Datacenter | Notes | | Role | Node | Private IP | Public IP | Datacenter | Notes |
|------|------|-----------|-----------|------------|-------| |------|------|-----------|-----------|------------|-------|
| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | EXCLUDED from `os-update.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) | | control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) | | control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) | | control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
| worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | | | worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | |
| worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | | | worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | |
| worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | | | worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | |
@ -38,11 +38,19 @@ Always re-derive the live list before running — nodes may have been added/remo
ssh root@178.105.17.239 'kubectl get nodes -o wide' ssh root@178.105.17.239 'kubectl get nodes -o wide'
``` ```
**Order rule:** update workers only. Within workers, update in this order to protect stateful workloads: **Order rule:** update workers first, then control-plane nodes. Within each group, update in this order to protect stateful workloads:
Workers:
1. runner + workers that host no PVs (safest — lowest disruption) 1. runner + workers that host no PVs (safest — lowest disruption)
2. remaining workers 2. remaining workers
3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time. 3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
4. **ALL control-plane nodes excluded** — `os-update.sh` selects nodes without the `node-role.kubernetes.io/control-plane` label, so `k3s-cp-1`, `k3s-cp-2`, and `k3s-cp-3` are all skipped automatically. CPs have their own procedure ([`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) — HA-aware, one CP at a time, leader last).
Control planes (integrated with the weekly cycle since [DEV-553](/DEV/issues/DEV-553); previously excluded and run manually via `CP_UPDATE_PROCEDURE.md`):
1. `os-update.sh` queries the current etcd leader before entering the CP phase.
2. Followers are updated first (in any order), leader last. Rationale: draining a follower is a no-op for raft leadership; draining the leader forces a re-election. Doing followers first minimises leader flapping.
3. Exactly one CP is drained or rebooting at a time — two CPs down means etcd quorum loss and a write outage across the cluster.
4. Between CPs `os-update.sh` re-runs `cluster-health.sh` and re-checks `etcdctl endpoint status --cluster`; any yellow signal halts the cycle at that CP. Settle wait between CPs is `POST_CP_WAIT_SECONDS` (default 180 s).
5. Each CP goes through `update-cp-node.sh <cp> --run` with `ASSUME_YES=1`: Phase A (add-swap, idempotent) → B preflight → C drain → D apt → E reboot with `/livez` polling and etcd rejoin check → F uncordon + cluster health.
**Concurrency:** exactly one node at a time. Never in parallel. **Concurrency:** exactly one node at a time. Never in parallel.
@ -249,15 +257,19 @@ Only proceed to the next node when ALL of the above are green. If not:
- Give it another 3 minutes and re-check (workloads with large images may still be pulling). - Give it another 3 minutes and re-check (workloads with large images may still be pulling).
- If still not green: STOP the cycle. Uncordon everything, leave the cluster in a stable state, and open a follow-up issue with the failing workloads. Do NOT proceed to update more nodes. - If still not green: STOP the cycle. Uncordon everything, leave the cluster in a stable state, and open a follow-up issue with the failing workloads. Do NOT proceed to update more nodes.
### 8. Control plane special handling ### 8. Control plane handling
All three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) are **NOT** updated by the weekly `os-update.sh` cycle — they have their own dedicated procedure and script: [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) and `scripts/os-update/update-cp-node.sh`. Reasons: Since [DEV-553](/DEV/issues/DEV-553) (2026-08-30) the weekly `os-update.sh` cycle **includes** the three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) after all workers finish. The CP steps are delegated to `scripts/os-update/update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) for the full design. Key properties:
- CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. One CP at a time, leader last. - CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. `os-update.sh` walks CPs sequentially with a health gate between each.
- All three CPs currently have zero swap. The CP procedure adds a durable 4 GiB swapfile per CP before draining. - `os-update.sh` queries `etcdctl endpoint status --cluster` before the CP phase to identify the current leader, then updates followers first and leader last.
- The CP procedure polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined. - `update-cp-node.sh` polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined etcd as `started`.
- Phase A (`--add-swap`) is idempotent: creates `/swapfile` (4 GiB default) with `vm.swappiness=10` on any CP without swap, and installs `etcd-client` if `etcdctl` is missing. Re-runs are no-ops when swap is already active.
`os-update.sh` excludes any node carrying the `node-role.kubernetes.io/control-plane` label (see `--dry-run` output) and prints a pointer to the CP procedure. Skip flags on `os-update.sh`:
- `--skip-cps` — workers only (legacy behavior for narrow patch cycles).
- `--only-cps` — skip workers and only walk the CP phase.
- `--only <node>` — single node, routed automatically to worker or CP.
--- ---
@ -318,8 +330,8 @@ If a node was skipped or errored → mark `blocked` with the unblock action, or
## Automation entry points ## Automation entry points
- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything. Explicitly skips `k3s-cp-1`. - `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → worker loop → CP loop → finalization). Idempotent, resumable via `--start-from <node>` (works for workers or CPs). Use `--dry-run` to print the plan without touching anything. Use `--skip-cps` / `--only-cps` to run only one half of the cycle.
- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry. Not for `k3s-cp-1`. - `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update for **workers** (all 7 per-node steps). Callable standalone for retry. Never call this for a CP.
- `infrastructure/scripts/os-update/update-cp-node.sh <node>` — HA-aware CP update flow (add swap, preflight, drain, apt, reboot, finalize). Accepts `k3s-cp-1`, `k3s-cp-2`, or `k3s-cp-3`. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md). - `infrastructure/scripts/os-update/update-cp-node.sh <node>` — HA-aware CP update flow (add swap, preflight, drain, apt, reboot, finalize). Accepts `k3s-cp-1`, `k3s-cp-2`, or `k3s-cp-3`. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md).
- `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compat. - `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compat.
- `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure. - `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure.
@ -341,3 +353,4 @@ A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this proce
| 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) | | 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) |
| 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) | | 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) |
| 2026-08-23 | Downgraded the kine cascade section for the HA-etcd era (DEV-510 migrated cp-1 from kine SQLite to embedded etcd, cp-2/cp-3 joined as full members). Pointed §8 and the "Automation entry points" list at the new HA-aware `CP_UPDATE_PROCEDURE.md` / `update-cp-node.sh`. Old `CP1_UPDATE_PROCEDURE.md` and `update-cp-1.sh` are now redirect stubs. | CTO agent (DEV-515) | | 2026-08-23 | Downgraded the kine cascade section for the HA-etcd era (DEV-510 migrated cp-1 from kine SQLite to embedded etcd, cp-2/cp-3 joined as full members). Pointed §8 and the "Automation entry points" list at the new HA-aware `CP_UPDATE_PROCEDURE.md` / `update-cp-node.sh`. Old `CP1_UPDATE_PROCEDURE.md` and `update-cp-1.sh` are now redirect stubs. | CTO agent (DEV-515) |
| 2026-08-30 | Integrated the three CPs into the weekly `os-update.sh` cycle (leader-last ordering, health gate between CPs, delegates each CP to `update-cp-node.sh --run` with `ASSUME_YES=1`). Added `--skip-cps` / `--only-cps` flags. `--only <node>` and `--start-from <node>` now route between worker/CP lists automatically. | CTO agent (DEV-553) |

View file

@ -4,32 +4,47 @@
# Behavior: # Behavior:
# 1. preflight cluster health (fail-closed) # 1. preflight cluster health (fail-closed)
# 2. take an etcd snapshot # 2. take an etcd snapshot
# 3. compute node order (workers first, control plane last; # 3. compute node order:
# Stalwart-hosting fsn1 workers moved to end of workers group) # workers (non-Stalwart first, Stalwart-hosting fsn1 workers last)
# 4. call update-node.sh for each node, halting on any failure # then control-plane nodes (etcd followers first, leader last).
# 4. call update-node.sh for each worker and update-cp-node.sh for each CP,
# halting on any failure
# 5. finalization: health snapshot + apt history digest # 5. finalization: health snapshot + apt history digest
# #
# Usage: # Usage:
# os-update.sh [--dry-run] [--start-from <node>] [--only <node>] # os-update.sh [--dry-run] [--start-from <node>] [--only <node>]
# [--skip-cps] [--only-cps]
#
# Flags:
# --skip-cps run workers only (legacy behavior, useful for narrow patches).
# --only-cps skip workers, only walk the CP phase.
# --only <n> update a single node (worker or CP, routed automatically).
# #
# Environment: # Environment:
# CONTROL_PLANE_HOST (default 178.105.17.239) # CONTROL_PLANE_HOST (default 178.105.17.239)
# All env vars honored by update-node.sh are honored here as well. # POST_CP_WAIT_SECONDS (default 180) — settle between CPs after uncordon.
# All env vars honored by update-node.sh / update-cp-node.sh are honored here.
# #
# Read OS_UPDATE_PROCEDURE.md alongside this script; the script mirrors it # Read OS_UPDATE_PROCEDURE.md alongside this script; the script mirrors it
# step-for-step and the doc is the authoritative reference. # step-for-step and the doc is the authoritative reference. CP handling is
# delegated to update-cp-node.sh; see CP_UPDATE_PROCEDURE.md.
set -euo pipefail set -euo pipefail
CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}" CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
POST_CP_WAIT_SECONDS="${POST_CP_WAIT_SECONDS:-180}"
DRY_RUN=0 DRY_RUN=0
START_FROM="" START_FROM=""
ONLY="" ONLY=""
SKIP_CPS=0
ONLY_CPS=0
while [ $# -gt 0 ]; do while [ $# -gt 0 ]; do
case "$1" in case "$1" in
--dry-run) DRY_RUN=1; shift ;; --dry-run) DRY_RUN=1; shift ;;
--start-from) START_FROM="$2"; shift 2 ;; --start-from) START_FROM="$2"; shift 2 ;;
--only) ONLY="$2"; shift 2 ;; --only) ONLY="$2"; shift 2 ;;
--skip-cps) SKIP_CPS=1; shift ;;
--only-cps) ONLY_CPS=1; shift ;;
-h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; -h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown arg: $1" >&2; exit 2 ;; *) echo "unknown arg: $1" >&2; exit 2 ;;
esac esac
@ -68,17 +83,17 @@ fi
# --- 3. node ordering --------------------------------------------------------- # --- 3. node ordering ---------------------------------------------------------
# Ordering rule: # Ordering rule:
# - workers only # - workers first (non-Stalwart, then Stalwart-hosting fsn1 last)
# - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last # - control-plane nodes AFTER workers, one at a time, etcd followers first,
# - ALL control-plane nodes are EXCLUDED and never updated by this script. Rationale: # leader last. Rationale (see CP_UPDATE_PROCEDURE.md):
# * kine/etcd write-path is sensitive to concurrent drains (see kine thundering-herd # * two of three CPs down = etcd quorum loss = write outage. Only one CP
# guardrails in OS_UPDATE_PROCEDURE.md, added after DEV-495). # cordoned/rebooting at a time.
# * rebooting a CP removes one apiserver — needs external liveness monitoring. # * draining a follower is a no-op for raft leadership; draining the leader
# * CP nodes host their own StatefulSet workloads that need batched eviction. # forces a re-election, so doing followers first minimises leader flapping.
# Since DEV-510 (2026-08-22) the cluster runs HA control plane (cp-1/cp-2/cp-3). The # Since DEV-510 (2026-08-22) the cluster runs HA embedded etcd across
# exclusion here is role-based so ALL current and future CPs are covered automatically. # cp-1/cp-2/cp-3 with 3-node quorum. This script delegates all CP steps to
# To update a CP, use `scripts/os-update/update-cp-1.sh` (currently cp-1-only; will be # update-cp-node.sh, which handles swap (idempotent), preflight, drain, apt,
# generalized to any CP as part of the HA-aware CP OS-update procedure follow-up). # reboot with /livez polling, etcd rejoin check, and finalize.
STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true) STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true)
# If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers. # If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers.
@ -108,38 +123,111 @@ for n in $ALL_NODES; do
fi fi
done done
ORDER=("${workers[@]}" "${stalwart_workers[@]}") WORKER_ORDER=("${workers[@]}" "${stalwart_workers[@]}")
# NOTE: CP nodes intentionally excluded. To update a CP, see CP1_UPDATE_PROCEDURE.md.
# --- 3b. control-plane ordering (leader last) --------------------------------
# CP-node → public IP map. Kept out-of-cluster so a full-cluster incident
# doesn't hide etcd from the operator machine.
declare -A CP_HOST=(
[k3s-cp-1]="178.105.17.239"
[k3s-cp-2]="188.245.85.199"
[k3s-cp-3]="49.13.92.162"
)
etcd_leader_ip() {
# Query etcd leader endpoint from the operator's control-plane host. Uses
# the same TLS material as update-cp-node.sh. Prints "IP" on success,
# nothing on failure.
ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new -o BatchMode=yes \
root@"$CONTROL_PLANE_HOST" \
'ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
endpoint status --cluster -w simple 2>/dev/null' \
| awk -F, '$5 ~ /true/ {print $1}' \
| sed -E 's|https?://([^:]+):.*|\1|'
}
CP_ORDER=()
if [ -n "$CP_NODES" ]; then if [ -n "$CP_NODES" ]; then
log "[plan] EXCLUDING control-plane nodes: $(echo $CP_NODES | tr '\n' ' ')— use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)" LEADER_IP="$(etcd_leader_ip || true)"
leader_node=""
followers=()
for n in $CP_NODES; do
ip="${CP_HOST[$n]:-}"
if [ -n "$ip" ] && [ "$ip" = "$LEADER_IP" ]; then
leader_node="$n"
else
followers+=("$n")
fi
done
if [ -n "$leader_node" ]; then
CP_ORDER=("${followers[@]}" "$leader_node")
log "[plan] etcd leader is $leader_node ($LEADER_IP) — updating followers first, leader last"
else
# Couldn't identify leader — refuse to guess. Fall back to no CPs.
CP_ORDER=()
log "[plan] WARN could not identify etcd leader — skipping CP phase this cycle"
fi
fi fi
# Route --only <node> automatically to worker or CP list.
if [ -n "$ONLY" ]; then if [ -n "$ONLY" ]; then
ORDER=("$ONLY") if is_cp "$ONLY"; then
CP_ORDER=("$ONLY")
WORKER_ORDER=()
else
WORKER_ORDER=("$ONLY")
CP_ORDER=()
fi
elif [ -n "$START_FROM" ]; then elif [ -n "$START_FROM" ]; then
new=() new=()
skip=1 skip=1
for n in "${ORDER[@]}"; do for n in "${WORKER_ORDER[@]}"; do
[ "$n" = "$START_FROM" ] && skip=0 [ "$n" = "$START_FROM" ] && skip=0
[ $skip -eq 0 ] && new+=("$n") [ $skip -eq 0 ] && new+=("$n")
done done
ORDER=("${new[@]}") WORKER_ORDER=("${new[@]}")
# If START_FROM is a CP, trim CP_ORDER instead.
if is_cp "$START_FROM"; then
new_cp=()
skip=1
for n in "${CP_ORDER[@]:-}"; do
[ -z "$n" ] && continue
[ "$n" = "$START_FROM" ] && skip=0
[ $skip -eq 0 ] && new_cp+=("$n")
done
CP_ORDER=("${new_cp[@]}")
WORKER_ORDER=()
fi
fi fi
log "[plan] ordered nodes (${#ORDER[@]}): ${ORDER[*]}" [ "$SKIP_CPS" -eq 1 ] && CP_ORDER=()
printf '%s\n' "${ORDER[@]}" > "$LOG_DIR/plan.txt" [ "$ONLY_CPS" -eq 1 ] && WORKER_ORDER=()
log "[plan] workers (${#WORKER_ORDER[@]}): ${WORKER_ORDER[*]:-none}"
log "[plan] control-planes (${#CP_ORDER[@]}): ${CP_ORDER[*]:-none}"
{
echo "# workers"
printf '%s\n' "${WORKER_ORDER[@]:-}"
echo "# control-planes"
printf '%s\n' "${CP_ORDER[@]:-}"
} > "$LOG_DIR/plan.txt"
if [ "$DRY_RUN" -eq 1 ]; then if [ "$DRY_RUN" -eq 1 ]; then
log "DRY-RUN — plan written, no node touched. Exiting." log "DRY-RUN — plan written, no node touched. Exiting."
exit 0 exit 0
fi fi
# --- 4. per-node loop --------------------------------------------------------- # --- 4. worker loop ----------------------------------------------------------
updated=() updated=()
skipped=() skipped=()
for n in "${ORDER[@]}"; do for n in "${WORKER_ORDER[@]}"; do
[ -z "$n" ] && continue
log "====================================================================" log "===================================================================="
log "==> updating $n" log "==> updating worker $n"
log "====================================================================" log "===================================================================="
NODE_LOG="$LOG_DIR/${n}.log" NODE_LOG="$LOG_DIR/${n}.log"
set +e set +e
@ -153,6 +241,50 @@ for n in "${ORDER[@]}"; do
esac esac
done done
# --- 4b. control-plane loop --------------------------------------------------
# One CP at a time, followers first, leader last. Delegates to update-cp-node.sh
# --run (Phase A add-swap is idempotent, Phase B..F run with ASSUME_YES=1).
# Between CPs we run cluster-health.sh and re-verify etcd cluster status; halt
# on any yellow signal.
if [ ${#CP_ORDER[@]} -gt 0 ]; then
log "===================================================================="
log "==> control-plane phase (${#CP_ORDER[@]} nodes): ${CP_ORDER[*]}"
log "===================================================================="
first_cp=1
for cp in "${CP_ORDER[@]}"; do
[ -z "$cp" ] && continue
if [ $first_cp -eq 0 ]; then
log "-- health gate between CPs (settle ${POST_CP_WAIT_SECONDS}s, then re-check)"
sleep "$POST_CP_WAIT_SECONDS"
if ! RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh" 2>&1 | tee "$LOG_DIR/inter-cp-health-before-${cp}.log"; then
die "cluster health degraded between CPs — halting before $cp. See $LOG_DIR/"
fi
# Re-check etcd cluster status; every member must still be started.
ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new root@"$CONTROL_PLANE_HOST" \
'ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
endpoint status --cluster -w table' 2>&1 | tee -a "$LOG_DIR/inter-cp-health-before-${cp}.log" || \
die "etcd endpoint status failed between CPs — halting before $cp"
fi
log "===================================================================="
log "==> updating control-plane $cp"
log "===================================================================="
CP_LOG="$LOG_DIR/${cp}.log"
set +e
ASSUME_YES=1 "$SCRIPT_DIR/update-cp-node.sh" "$cp" --run 2>&1 | tee "$CP_LOG"
rc=${PIPESTATUS[0]}
set -e
case $rc in
0) updated+=("$cp") ;;
*) die "update-cp-node.sh failed for $cp (rc=$rc). Cycle halted. See $CP_LOG" ;;
esac
first_cp=0
done
fi
# --- 5. finalization ---------------------------------------------------------- # --- 5. finalization ----------------------------------------------------------
log "====================================================================" log "===================================================================="
log "==> finalization" log "==> finalization"
@ -169,6 +301,8 @@ log "[final] apt history digest"
# resolve ssh target via a mini-eval of node_ssh_target-equivalent # resolve ssh target via a mini-eval of node_ssh_target-equivalent
case "$n" in case "$n" in
k3s-cp-1) t="root@178.105.17.239" ;; k3s-cp-1) t="root@178.105.17.239" ;;
k3s-cp-2) t="root@188.245.85.199" ;;
k3s-cp-3) t="root@49.13.92.162" ;;
k3s-worker-1) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;; k3s-worker-1) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
k3s-worker-2) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;; k3s-worker-2) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
k3s-worker-3) t="root@167.233.121.121" ;; k3s-worker-3) t="root@167.233.121.121" ;;