feat(os-update): integrate control planes into weekly rolling cycle (DEV-553)
os-update.sh now walks all three CPs after workers, one at a time, etcd followers first and leader last. Each CP is delegated to update-cp-node.sh --run with ASSUME_YES=1; a health gate (cluster-health.sh + etcdctl endpoint status) runs between CPs. Adds --skip-cps and --only-cps flags; --only <node> and --start-from <node> now route to the correct list automatically. OS_UPDATE_PROCEDURE.md updated: CPs are no longer excluded, §8 renamed to "Control plane handling" and describes the leader-last cycle + delegation. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
c97a2d218e
commit
a30274d1eb
2 changed files with 188 additions and 41 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Weekly Rolling Ubuntu OS Update Procedure
|
||||
|
||||
**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node.
|
||||
**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node. Since [DEV-553](/DEV/issues/DEV-553) (2026-08-30) this covers **workers AND control-plane nodes** in a single cycle; workers run first, then CPs one at a time with etcd leader last.
|
||||
|
||||
**Audience:** Agents (currently CTO, optionally a dedicated ClusterOps agent). Also usable manually by an operator.
|
||||
|
||||
|
|
@ -22,9 +22,9 @@
|
|||
|
||||
| Role | Node | Private IP | Public IP | Datacenter | Notes |
|
||||
|------|------|-----------|-----------|------------|-------|
|
||||
| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | EXCLUDED from `os-update.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
||||
| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
||||
| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
||||
| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
||||
| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
||||
| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
||||
| worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | |
|
||||
| worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | |
|
||||
| worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | |
|
||||
|
|
@ -38,11 +38,19 @@ Always re-derive the live list before running — nodes may have been added/remo
|
|||
ssh root@178.105.17.239 'kubectl get nodes -o wide'
|
||||
```
|
||||
|
||||
**Order rule:** update workers only. Within workers, update in this order to protect stateful workloads:
|
||||
**Order rule:** update workers first, then control-plane nodes. Within each group, update in this order to protect stateful workloads:
|
||||
|
||||
Workers:
|
||||
1. runner + workers that host no PVs (safest — lowest disruption)
|
||||
2. remaining workers
|
||||
3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
|
||||
4. **ALL control-plane nodes excluded** — `os-update.sh` selects nodes without the `node-role.kubernetes.io/control-plane` label, so `k3s-cp-1`, `k3s-cp-2`, and `k3s-cp-3` are all skipped automatically. CPs have their own procedure ([`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) — HA-aware, one CP at a time, leader last).
|
||||
|
||||
Control planes (integrated with the weekly cycle since [DEV-553](/DEV/issues/DEV-553); previously excluded and run manually via `CP_UPDATE_PROCEDURE.md`):
|
||||
1. `os-update.sh` queries the current etcd leader before entering the CP phase.
|
||||
2. Followers are updated first (in any order), leader last. Rationale: draining a follower is a no-op for raft leadership; draining the leader forces a re-election. Doing followers first minimises leader flapping.
|
||||
3. Exactly one CP is drained or rebooting at a time — two CPs down means etcd quorum loss and a write outage across the cluster.
|
||||
4. Between CPs `os-update.sh` re-runs `cluster-health.sh` and re-checks `etcdctl endpoint status --cluster`; any yellow signal halts the cycle at that CP. Settle wait between CPs is `POST_CP_WAIT_SECONDS` (default 180 s).
|
||||
5. Each CP goes through `update-cp-node.sh <cp> --run` with `ASSUME_YES=1`: Phase A (add-swap, idempotent) → B preflight → C drain → D apt → E reboot with `/livez` polling and etcd rejoin check → F uncordon + cluster health.
|
||||
|
||||
**Concurrency:** exactly one node at a time. Never in parallel.
|
||||
|
||||
|
|
@ -249,15 +257,19 @@ Only proceed to the next node when ALL of the above are green. If not:
|
|||
- Give it another 3 minutes and re-check (workloads with large images may still be pulling).
|
||||
- If still not green: STOP the cycle. Uncordon everything, leave the cluster in a stable state, and open a follow-up issue with the failing workloads. Do NOT proceed to update more nodes.
|
||||
|
||||
### 8. Control plane special handling
|
||||
### 8. Control plane handling
|
||||
|
||||
All three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) are **NOT** updated by the weekly `os-update.sh` cycle — they have their own dedicated procedure and script: [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) and `scripts/os-update/update-cp-node.sh`. Reasons:
|
||||
Since [DEV-553](/DEV/issues/DEV-553) (2026-08-30) the weekly `os-update.sh` cycle **includes** the three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) after all workers finish. The CP steps are delegated to `scripts/os-update/update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) for the full design. Key properties:
|
||||
|
||||
- CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. One CP at a time, leader last.
|
||||
- All three CPs currently have zero swap. The CP procedure adds a durable 4 GiB swapfile per CP before draining.
|
||||
- The CP procedure polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined.
|
||||
- CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. `os-update.sh` walks CPs sequentially with a health gate between each.
|
||||
- `os-update.sh` queries `etcdctl endpoint status --cluster` before the CP phase to identify the current leader, then updates followers first and leader last.
|
||||
- `update-cp-node.sh` polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined etcd as `started`.
|
||||
- Phase A (`--add-swap`) is idempotent: creates `/swapfile` (4 GiB default) with `vm.swappiness=10` on any CP without swap, and installs `etcd-client` if `etcdctl` is missing. Re-runs are no-ops when swap is already active.
|
||||
|
||||
`os-update.sh` excludes any node carrying the `node-role.kubernetes.io/control-plane` label (see `--dry-run` output) and prints a pointer to the CP procedure.
|
||||
Skip flags on `os-update.sh`:
|
||||
- `--skip-cps` — workers only (legacy behavior for narrow patch cycles).
|
||||
- `--only-cps` — skip workers and only walk the CP phase.
|
||||
- `--only <node>` — single node, routed automatically to worker or CP.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -318,8 +330,8 @@ If a node was skipped or errored → mark `blocked` with the unblock action, or
|
|||
|
||||
## Automation entry points
|
||||
|
||||
- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything. Explicitly skips `k3s-cp-1`.
|
||||
- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry. Not for `k3s-cp-1`.
|
||||
- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → worker loop → CP loop → finalization). Idempotent, resumable via `--start-from <node>` (works for workers or CPs). Use `--dry-run` to print the plan without touching anything. Use `--skip-cps` / `--only-cps` to run only one half of the cycle.
|
||||
- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update for **workers** (all 7 per-node steps). Callable standalone for retry. Never call this for a CP.
|
||||
- `infrastructure/scripts/os-update/update-cp-node.sh <node>` — HA-aware CP update flow (add swap, preflight, drain, apt, reboot, finalize). Accepts `k3s-cp-1`, `k3s-cp-2`, or `k3s-cp-3`. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md).
|
||||
- `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compat.
|
||||
- `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure.
|
||||
|
|
@ -341,3 +353,4 @@ A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this proce
|
|||
| 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) |
|
||||
| 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) |
|
||||
| 2026-08-23 | Downgraded the kine cascade section for the HA-etcd era (DEV-510 migrated cp-1 from kine SQLite to embedded etcd, cp-2/cp-3 joined as full members). Pointed §8 and the "Automation entry points" list at the new HA-aware `CP_UPDATE_PROCEDURE.md` / `update-cp-node.sh`. Old `CP1_UPDATE_PROCEDURE.md` and `update-cp-1.sh` are now redirect stubs. | CTO agent (DEV-515) |
|
||||
| 2026-08-30 | Integrated the three CPs into the weekly `os-update.sh` cycle (leader-last ordering, health gate between CPs, delegates each CP to `update-cp-node.sh --run` with `ASSUME_YES=1`). Added `--skip-cps` / `--only-cps` flags. `--only <node>` and `--start-from <node>` now route between worker/CP lists automatically. | CTO agent (DEV-553) |
|
||||
|
|
|
|||
|
|
@ -4,32 +4,47 @@
|
|||
# Behavior:
|
||||
# 1. preflight cluster health (fail-closed)
|
||||
# 2. take an etcd snapshot
|
||||
# 3. compute node order (workers first, control plane last;
|
||||
# Stalwart-hosting fsn1 workers moved to end of workers group)
|
||||
# 4. call update-node.sh for each node, halting on any failure
|
||||
# 3. compute node order:
|
||||
# workers (non-Stalwart first, Stalwart-hosting fsn1 workers last)
|
||||
# then control-plane nodes (etcd followers first, leader last).
|
||||
# 4. call update-node.sh for each worker and update-cp-node.sh for each CP,
|
||||
# halting on any failure
|
||||
# 5. finalization: health snapshot + apt history digest
|
||||
#
|
||||
# Usage:
|
||||
# os-update.sh [--dry-run] [--start-from <node>] [--only <node>]
|
||||
# [--skip-cps] [--only-cps]
|
||||
#
|
||||
# Flags:
|
||||
# --skip-cps run workers only (legacy behavior, useful for narrow patches).
|
||||
# --only-cps skip workers, only walk the CP phase.
|
||||
# --only <n> update a single node (worker or CP, routed automatically).
|
||||
#
|
||||
# Environment:
|
||||
# CONTROL_PLANE_HOST (default 178.105.17.239)
|
||||
# All env vars honored by update-node.sh are honored here as well.
|
||||
# POST_CP_WAIT_SECONDS (default 180) — settle between CPs after uncordon.
|
||||
# All env vars honored by update-node.sh / update-cp-node.sh are honored here.
|
||||
#
|
||||
# Read OS_UPDATE_PROCEDURE.md alongside this script; the script mirrors it
|
||||
# step-for-step and the doc is the authoritative reference.
|
||||
# step-for-step and the doc is the authoritative reference. CP handling is
|
||||
# delegated to update-cp-node.sh; see CP_UPDATE_PROCEDURE.md.
|
||||
set -euo pipefail
|
||||
|
||||
CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
|
||||
POST_CP_WAIT_SECONDS="${POST_CP_WAIT_SECONDS:-180}"
|
||||
DRY_RUN=0
|
||||
START_FROM=""
|
||||
ONLY=""
|
||||
SKIP_CPS=0
|
||||
ONLY_CPS=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
--start-from) START_FROM="$2"; shift 2 ;;
|
||||
--only) ONLY="$2"; shift 2 ;;
|
||||
--skip-cps) SKIP_CPS=1; shift ;;
|
||||
--only-cps) ONLY_CPS=1; shift ;;
|
||||
-h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
|
|
@ -68,17 +83,17 @@ fi
|
|||
|
||||
# --- 3. node ordering ---------------------------------------------------------
|
||||
# Ordering rule:
|
||||
# - workers only
|
||||
# - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last
|
||||
# - ALL control-plane nodes are EXCLUDED and never updated by this script. Rationale:
|
||||
# * kine/etcd write-path is sensitive to concurrent drains (see kine thundering-herd
|
||||
# guardrails in OS_UPDATE_PROCEDURE.md, added after DEV-495).
|
||||
# * rebooting a CP removes one apiserver — needs external liveness monitoring.
|
||||
# * CP nodes host their own StatefulSet workloads that need batched eviction.
|
||||
# Since DEV-510 (2026-08-22) the cluster runs HA control plane (cp-1/cp-2/cp-3). The
|
||||
# exclusion here is role-based so ALL current and future CPs are covered automatically.
|
||||
# To update a CP, use `scripts/os-update/update-cp-1.sh` (currently cp-1-only; will be
|
||||
# generalized to any CP as part of the HA-aware CP OS-update procedure follow-up).
|
||||
# - workers first (non-Stalwart, then Stalwart-hosting fsn1 last)
|
||||
# - control-plane nodes AFTER workers, one at a time, etcd followers first,
|
||||
# leader last. Rationale (see CP_UPDATE_PROCEDURE.md):
|
||||
# * two of three CPs down = etcd quorum loss = write outage. Only one CP
|
||||
# cordoned/rebooting at a time.
|
||||
# * draining a follower is a no-op for raft leadership; draining the leader
|
||||
# forces a re-election, so doing followers first minimises leader flapping.
|
||||
# Since DEV-510 (2026-08-22) the cluster runs HA embedded etcd across
|
||||
# cp-1/cp-2/cp-3 with 3-node quorum. This script delegates all CP steps to
|
||||
# update-cp-node.sh, which handles swap (idempotent), preflight, drain, apt,
|
||||
# reboot with /livez polling, etcd rejoin check, and finalize.
|
||||
STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true)
|
||||
# If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers.
|
||||
|
||||
|
|
@ -108,38 +123,111 @@ for n in $ALL_NODES; do
|
|||
fi
|
||||
done
|
||||
|
||||
ORDER=("${workers[@]}" "${stalwart_workers[@]}")
|
||||
# NOTE: CP nodes intentionally excluded. To update a CP, see CP1_UPDATE_PROCEDURE.md.
|
||||
WORKER_ORDER=("${workers[@]}" "${stalwart_workers[@]}")
|
||||
|
||||
# --- 3b. control-plane ordering (leader last) --------------------------------
|
||||
# CP-node → public IP map. Kept out-of-cluster so a full-cluster incident
|
||||
# doesn't hide etcd from the operator machine.
|
||||
declare -A CP_HOST=(
|
||||
[k3s-cp-1]="178.105.17.239"
|
||||
[k3s-cp-2]="188.245.85.199"
|
||||
[k3s-cp-3]="49.13.92.162"
|
||||
)
|
||||
|
||||
etcd_leader_ip() {
|
||||
# Query etcd leader endpoint from the operator's control-plane host. Uses
|
||||
# the same TLS material as update-cp-node.sh. Prints "IP" on success,
|
||||
# nothing on failure.
|
||||
ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new -o BatchMode=yes \
|
||||
root@"$CONTROL_PLANE_HOST" \
|
||||
'ETCDCTL_API=3 etcdctl \
|
||||
--endpoints=https://127.0.0.1:2379 \
|
||||
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
|
||||
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
|
||||
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
|
||||
endpoint status --cluster -w simple 2>/dev/null' \
|
||||
| awk -F, '$5 ~ /true/ {print $1}' \
|
||||
| sed -E 's|https?://([^:]+):.*|\1|'
|
||||
}
|
||||
|
||||
CP_ORDER=()
|
||||
if [ -n "$CP_NODES" ]; then
|
||||
log "[plan] EXCLUDING control-plane nodes: $(echo $CP_NODES | tr '\n' ' ')— use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)"
|
||||
LEADER_IP="$(etcd_leader_ip || true)"
|
||||
leader_node=""
|
||||
followers=()
|
||||
for n in $CP_NODES; do
|
||||
ip="${CP_HOST[$n]:-}"
|
||||
if [ -n "$ip" ] && [ "$ip" = "$LEADER_IP" ]; then
|
||||
leader_node="$n"
|
||||
else
|
||||
followers+=("$n")
|
||||
fi
|
||||
done
|
||||
if [ -n "$leader_node" ]; then
|
||||
CP_ORDER=("${followers[@]}" "$leader_node")
|
||||
log "[plan] etcd leader is $leader_node ($LEADER_IP) — updating followers first, leader last"
|
||||
else
|
||||
# Couldn't identify leader — refuse to guess. Fall back to no CPs.
|
||||
CP_ORDER=()
|
||||
log "[plan] WARN could not identify etcd leader — skipping CP phase this cycle"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Route --only <node> automatically to worker or CP list.
|
||||
if [ -n "$ONLY" ]; then
|
||||
ORDER=("$ONLY")
|
||||
if is_cp "$ONLY"; then
|
||||
CP_ORDER=("$ONLY")
|
||||
WORKER_ORDER=()
|
||||
else
|
||||
WORKER_ORDER=("$ONLY")
|
||||
CP_ORDER=()
|
||||
fi
|
||||
elif [ -n "$START_FROM" ]; then
|
||||
new=()
|
||||
skip=1
|
||||
for n in "${ORDER[@]}"; do
|
||||
for n in "${WORKER_ORDER[@]}"; do
|
||||
[ "$n" = "$START_FROM" ] && skip=0
|
||||
[ $skip -eq 0 ] && new+=("$n")
|
||||
done
|
||||
ORDER=("${new[@]}")
|
||||
WORKER_ORDER=("${new[@]}")
|
||||
# If START_FROM is a CP, trim CP_ORDER instead.
|
||||
if is_cp "$START_FROM"; then
|
||||
new_cp=()
|
||||
skip=1
|
||||
for n in "${CP_ORDER[@]:-}"; do
|
||||
[ -z "$n" ] && continue
|
||||
[ "$n" = "$START_FROM" ] && skip=0
|
||||
[ $skip -eq 0 ] && new_cp+=("$n")
|
||||
done
|
||||
CP_ORDER=("${new_cp[@]}")
|
||||
WORKER_ORDER=()
|
||||
fi
|
||||
fi
|
||||
|
||||
log "[plan] ordered nodes (${#ORDER[@]}): ${ORDER[*]}"
|
||||
printf '%s\n' "${ORDER[@]}" > "$LOG_DIR/plan.txt"
|
||||
[ "$SKIP_CPS" -eq 1 ] && CP_ORDER=()
|
||||
[ "$ONLY_CPS" -eq 1 ] && WORKER_ORDER=()
|
||||
|
||||
log "[plan] workers (${#WORKER_ORDER[@]}): ${WORKER_ORDER[*]:-none}"
|
||||
log "[plan] control-planes (${#CP_ORDER[@]}): ${CP_ORDER[*]:-none}"
|
||||
{
|
||||
echo "# workers"
|
||||
printf '%s\n' "${WORKER_ORDER[@]:-}"
|
||||
echo "# control-planes"
|
||||
printf '%s\n' "${CP_ORDER[@]:-}"
|
||||
} > "$LOG_DIR/plan.txt"
|
||||
|
||||
if [ "$DRY_RUN" -eq 1 ]; then
|
||||
log "DRY-RUN — plan written, no node touched. Exiting."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# --- 4. per-node loop ---------------------------------------------------------
|
||||
# --- 4. worker loop ----------------------------------------------------------
|
||||
updated=()
|
||||
skipped=()
|
||||
for n in "${ORDER[@]}"; do
|
||||
for n in "${WORKER_ORDER[@]}"; do
|
||||
[ -z "$n" ] && continue
|
||||
log "===================================================================="
|
||||
log "==> updating $n"
|
||||
log "==> updating worker $n"
|
||||
log "===================================================================="
|
||||
NODE_LOG="$LOG_DIR/${n}.log"
|
||||
set +e
|
||||
|
|
@ -153,6 +241,50 @@ for n in "${ORDER[@]}"; do
|
|||
esac
|
||||
done
|
||||
|
||||
# --- 4b. control-plane loop --------------------------------------------------
|
||||
# One CP at a time, followers first, leader last. Delegates to update-cp-node.sh
|
||||
# --run (Phase A add-swap is idempotent, Phase B..F run with ASSUME_YES=1).
|
||||
# Between CPs we run cluster-health.sh and re-verify etcd cluster status; halt
|
||||
# on any yellow signal.
|
||||
if [ ${#CP_ORDER[@]} -gt 0 ]; then
|
||||
log "===================================================================="
|
||||
log "==> control-plane phase (${#CP_ORDER[@]} nodes): ${CP_ORDER[*]}"
|
||||
log "===================================================================="
|
||||
first_cp=1
|
||||
for cp in "${CP_ORDER[@]}"; do
|
||||
[ -z "$cp" ] && continue
|
||||
if [ $first_cp -eq 0 ]; then
|
||||
log "-- health gate between CPs (settle ${POST_CP_WAIT_SECONDS}s, then re-check)"
|
||||
sleep "$POST_CP_WAIT_SECONDS"
|
||||
if ! RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh" 2>&1 | tee "$LOG_DIR/inter-cp-health-before-${cp}.log"; then
|
||||
die "cluster health degraded between CPs — halting before $cp. See $LOG_DIR/"
|
||||
fi
|
||||
# Re-check etcd cluster status; every member must still be started.
|
||||
ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new root@"$CONTROL_PLANE_HOST" \
|
||||
'ETCDCTL_API=3 etcdctl \
|
||||
--endpoints=https://127.0.0.1:2379 \
|
||||
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
|
||||
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
|
||||
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
|
||||
endpoint status --cluster -w table' 2>&1 | tee -a "$LOG_DIR/inter-cp-health-before-${cp}.log" || \
|
||||
die "etcd endpoint status failed between CPs — halting before $cp"
|
||||
fi
|
||||
log "===================================================================="
|
||||
log "==> updating control-plane $cp"
|
||||
log "===================================================================="
|
||||
CP_LOG="$LOG_DIR/${cp}.log"
|
||||
set +e
|
||||
ASSUME_YES=1 "$SCRIPT_DIR/update-cp-node.sh" "$cp" --run 2>&1 | tee "$CP_LOG"
|
||||
rc=${PIPESTATUS[0]}
|
||||
set -e
|
||||
case $rc in
|
||||
0) updated+=("$cp") ;;
|
||||
*) die "update-cp-node.sh failed for $cp (rc=$rc). Cycle halted. See $CP_LOG" ;;
|
||||
esac
|
||||
first_cp=0
|
||||
done
|
||||
fi
|
||||
|
||||
# --- 5. finalization ----------------------------------------------------------
|
||||
log "===================================================================="
|
||||
log "==> finalization"
|
||||
|
|
@ -169,6 +301,8 @@ log "[final] apt history digest"
|
|||
# resolve ssh target via a mini-eval of node_ssh_target-equivalent
|
||||
case "$n" in
|
||||
k3s-cp-1) t="root@178.105.17.239" ;;
|
||||
k3s-cp-2) t="root@188.245.85.199" ;;
|
||||
k3s-cp-3) t="root@49.13.92.162" ;;
|
||||
k3s-worker-1) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
|
||||
k3s-worker-2) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
|
||||
k3s-worker-3) t="root@167.233.121.121" ;;
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue