feat(os-update): integrate control planes into weekly rolling cycle (DEV-553)
os-update.sh now walks all three CPs after workers, one at a time, etcd followers first and leader last. Each CP is delegated to update-cp-node.sh --run with ASSUME_YES=1; a health gate (cluster-health.sh + etcdctl endpoint status) runs between CPs. Adds --skip-cps and --only-cps flags; --only <node> and --start-from <node> now route to the correct list automatically. OS_UPDATE_PROCEDURE.md updated: CPs are no longer excluded, §8 renamed to "Control plane handling" and describes the leader-last cycle + delegation. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
parent
c97a2d218e
commit
a30274d1eb
2 changed files with 188 additions and 41 deletions
|
|
@ -1,6 +1,6 @@
|
||||||
# Weekly Rolling Ubuntu OS Update Procedure
|
# Weekly Rolling Ubuntu OS Update Procedure
|
||||||
|
|
||||||
**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node.
|
**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node. Since [DEV-553](/DEV/issues/DEV-553) (2026-08-30) this covers **workers AND control-plane nodes** in a single cycle; workers run first, then CPs one at a time with etcd leader last.
|
||||||
|
|
||||||
**Audience:** Agents (currently CTO, optionally a dedicated ClusterOps agent). Also usable manually by an operator.
|
**Audience:** Agents (currently CTO, optionally a dedicated ClusterOps agent). Also usable manually by an operator.
|
||||||
|
|
||||||
|
|
@ -22,9 +22,9 @@
|
||||||
|
|
||||||
| Role | Node | Private IP | Public IP | Datacenter | Notes |
|
| Role | Node | Private IP | Public IP | Datacenter | Notes |
|
||||||
|------|------|-----------|-----------|------------|-------|
|
|------|------|-----------|-----------|------------|-------|
|
||||||
| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | EXCLUDED from `os-update.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
||||||
| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
||||||
| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | included via `update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
|
||||||
| worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | |
|
| worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | |
|
||||||
| worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | |
|
| worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | |
|
||||||
| worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | |
|
| worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | |
|
||||||
|
|
@ -38,11 +38,19 @@ Always re-derive the live list before running — nodes may have been added/remo
|
||||||
ssh root@178.105.17.239 'kubectl get nodes -o wide'
|
ssh root@178.105.17.239 'kubectl get nodes -o wide'
|
||||||
```
|
```
|
||||||
|
|
||||||
**Order rule:** update workers only. Within workers, update in this order to protect stateful workloads:
|
**Order rule:** update workers first, then control-plane nodes. Within each group, update in this order to protect stateful workloads:
|
||||||
|
|
||||||
|
Workers:
|
||||||
1. runner + workers that host no PVs (safest — lowest disruption)
|
1. runner + workers that host no PVs (safest — lowest disruption)
|
||||||
2. remaining workers
|
2. remaining workers
|
||||||
3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
|
3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
|
||||||
4. **ALL control-plane nodes excluded** — `os-update.sh` selects nodes without the `node-role.kubernetes.io/control-plane` label, so `k3s-cp-1`, `k3s-cp-2`, and `k3s-cp-3` are all skipped automatically. CPs have their own procedure ([`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) — HA-aware, one CP at a time, leader last).
|
|
||||||
|
Control planes (integrated with the weekly cycle since [DEV-553](/DEV/issues/DEV-553); previously excluded and run manually via `CP_UPDATE_PROCEDURE.md`):
|
||||||
|
1. `os-update.sh` queries the current etcd leader before entering the CP phase.
|
||||||
|
2. Followers are updated first (in any order), leader last. Rationale: draining a follower is a no-op for raft leadership; draining the leader forces a re-election. Doing followers first minimises leader flapping.
|
||||||
|
3. Exactly one CP is drained or rebooting at a time — two CPs down means etcd quorum loss and a write outage across the cluster.
|
||||||
|
4. Between CPs `os-update.sh` re-runs `cluster-health.sh` and re-checks `etcdctl endpoint status --cluster`; any yellow signal halts the cycle at that CP. Settle wait between CPs is `POST_CP_WAIT_SECONDS` (default 180 s).
|
||||||
|
5. Each CP goes through `update-cp-node.sh <cp> --run` with `ASSUME_YES=1`: Phase A (add-swap, idempotent) → B preflight → C drain → D apt → E reboot with `/livez` polling and etcd rejoin check → F uncordon + cluster health.
|
||||||
|
|
||||||
**Concurrency:** exactly one node at a time. Never in parallel.
|
**Concurrency:** exactly one node at a time. Never in parallel.
|
||||||
|
|
||||||
|
|
@ -249,15 +257,19 @@ Only proceed to the next node when ALL of the above are green. If not:
|
||||||
- Give it another 3 minutes and re-check (workloads with large images may still be pulling).
|
- Give it another 3 minutes and re-check (workloads with large images may still be pulling).
|
||||||
- If still not green: STOP the cycle. Uncordon everything, leave the cluster in a stable state, and open a follow-up issue with the failing workloads. Do NOT proceed to update more nodes.
|
- If still not green: STOP the cycle. Uncordon everything, leave the cluster in a stable state, and open a follow-up issue with the failing workloads. Do NOT proceed to update more nodes.
|
||||||
|
|
||||||
### 8. Control plane special handling
|
### 8. Control plane handling
|
||||||
|
|
||||||
All three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) are **NOT** updated by the weekly `os-update.sh` cycle — they have their own dedicated procedure and script: [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) and `scripts/os-update/update-cp-node.sh`. Reasons:
|
Since [DEV-553](/DEV/issues/DEV-553) (2026-08-30) the weekly `os-update.sh` cycle **includes** the three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) after all workers finish. The CP steps are delegated to `scripts/os-update/update-cp-node.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) for the full design. Key properties:
|
||||||
|
|
||||||
- CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. One CP at a time, leader last.
|
- CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. `os-update.sh` walks CPs sequentially with a health gate between each.
|
||||||
- All three CPs currently have zero swap. The CP procedure adds a durable 4 GiB swapfile per CP before draining.
|
- `os-update.sh` queries `etcdctl endpoint status --cluster` before the CP phase to identify the current leader, then updates followers first and leader last.
|
||||||
- The CP procedure polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined.
|
- `update-cp-node.sh` polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined etcd as `started`.
|
||||||
|
- Phase A (`--add-swap`) is idempotent: creates `/swapfile` (4 GiB default) with `vm.swappiness=10` on any CP without swap, and installs `etcd-client` if `etcdctl` is missing. Re-runs are no-ops when swap is already active.
|
||||||
|
|
||||||
`os-update.sh` excludes any node carrying the `node-role.kubernetes.io/control-plane` label (see `--dry-run` output) and prints a pointer to the CP procedure.
|
Skip flags on `os-update.sh`:
|
||||||
|
- `--skip-cps` — workers only (legacy behavior for narrow patch cycles).
|
||||||
|
- `--only-cps` — skip workers and only walk the CP phase.
|
||||||
|
- `--only <node>` — single node, routed automatically to worker or CP.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -318,8 +330,8 @@ If a node was skipped or errored → mark `blocked` with the unblock action, or
|
||||||
|
|
||||||
## Automation entry points
|
## Automation entry points
|
||||||
|
|
||||||
- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything. Explicitly skips `k3s-cp-1`.
|
- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → worker loop → CP loop → finalization). Idempotent, resumable via `--start-from <node>` (works for workers or CPs). Use `--dry-run` to print the plan without touching anything. Use `--skip-cps` / `--only-cps` to run only one half of the cycle.
|
||||||
- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry. Not for `k3s-cp-1`.
|
- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update for **workers** (all 7 per-node steps). Callable standalone for retry. Never call this for a CP.
|
||||||
- `infrastructure/scripts/os-update/update-cp-node.sh <node>` — HA-aware CP update flow (add swap, preflight, drain, apt, reboot, finalize). Accepts `k3s-cp-1`, `k3s-cp-2`, or `k3s-cp-3`. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md).
|
- `infrastructure/scripts/os-update/update-cp-node.sh <node>` — HA-aware CP update flow (add swap, preflight, drain, apt, reboot, finalize). Accepts `k3s-cp-1`, `k3s-cp-2`, or `k3s-cp-3`. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md).
|
||||||
- `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compat.
|
- `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compat.
|
||||||
- `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure.
|
- `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure.
|
||||||
|
|
@ -341,3 +353,4 @@ A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this proce
|
||||||
| 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) |
|
| 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) |
|
||||||
| 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) |
|
| 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) |
|
||||||
| 2026-08-23 | Downgraded the kine cascade section for the HA-etcd era (DEV-510 migrated cp-1 from kine SQLite to embedded etcd, cp-2/cp-3 joined as full members). Pointed §8 and the "Automation entry points" list at the new HA-aware `CP_UPDATE_PROCEDURE.md` / `update-cp-node.sh`. Old `CP1_UPDATE_PROCEDURE.md` and `update-cp-1.sh` are now redirect stubs. | CTO agent (DEV-515) |
|
| 2026-08-23 | Downgraded the kine cascade section for the HA-etcd era (DEV-510 migrated cp-1 from kine SQLite to embedded etcd, cp-2/cp-3 joined as full members). Pointed §8 and the "Automation entry points" list at the new HA-aware `CP_UPDATE_PROCEDURE.md` / `update-cp-node.sh`. Old `CP1_UPDATE_PROCEDURE.md` and `update-cp-1.sh` are now redirect stubs. | CTO agent (DEV-515) |
|
||||||
|
| 2026-08-30 | Integrated the three CPs into the weekly `os-update.sh` cycle (leader-last ordering, health gate between CPs, delegates each CP to `update-cp-node.sh --run` with `ASSUME_YES=1`). Added `--skip-cps` / `--only-cps` flags. `--only <node>` and `--start-from <node>` now route between worker/CP lists automatically. | CTO agent (DEV-553) |
|
||||||
|
|
|
||||||
|
|
@ -4,32 +4,47 @@
|
||||||
# Behavior:
|
# Behavior:
|
||||||
# 1. preflight cluster health (fail-closed)
|
# 1. preflight cluster health (fail-closed)
|
||||||
# 2. take an etcd snapshot
|
# 2. take an etcd snapshot
|
||||||
# 3. compute node order (workers first, control plane last;
|
# 3. compute node order:
|
||||||
# Stalwart-hosting fsn1 workers moved to end of workers group)
|
# workers (non-Stalwart first, Stalwart-hosting fsn1 workers last)
|
||||||
# 4. call update-node.sh for each node, halting on any failure
|
# then control-plane nodes (etcd followers first, leader last).
|
||||||
|
# 4. call update-node.sh for each worker and update-cp-node.sh for each CP,
|
||||||
|
# halting on any failure
|
||||||
# 5. finalization: health snapshot + apt history digest
|
# 5. finalization: health snapshot + apt history digest
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# os-update.sh [--dry-run] [--start-from <node>] [--only <node>]
|
# os-update.sh [--dry-run] [--start-from <node>] [--only <node>]
|
||||||
|
# [--skip-cps] [--only-cps]
|
||||||
|
#
|
||||||
|
# Flags:
|
||||||
|
# --skip-cps run workers only (legacy behavior, useful for narrow patches).
|
||||||
|
# --only-cps skip workers, only walk the CP phase.
|
||||||
|
# --only <n> update a single node (worker or CP, routed automatically).
|
||||||
#
|
#
|
||||||
# Environment:
|
# Environment:
|
||||||
# CONTROL_PLANE_HOST (default 178.105.17.239)
|
# CONTROL_PLANE_HOST (default 178.105.17.239)
|
||||||
# All env vars honored by update-node.sh are honored here as well.
|
# POST_CP_WAIT_SECONDS (default 180) — settle between CPs after uncordon.
|
||||||
|
# All env vars honored by update-node.sh / update-cp-node.sh are honored here.
|
||||||
#
|
#
|
||||||
# Read OS_UPDATE_PROCEDURE.md alongside this script; the script mirrors it
|
# Read OS_UPDATE_PROCEDURE.md alongside this script; the script mirrors it
|
||||||
# step-for-step and the doc is the authoritative reference.
|
# step-for-step and the doc is the authoritative reference. CP handling is
|
||||||
|
# delegated to update-cp-node.sh; see CP_UPDATE_PROCEDURE.md.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
|
CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
|
||||||
|
POST_CP_WAIT_SECONDS="${POST_CP_WAIT_SECONDS:-180}"
|
||||||
DRY_RUN=0
|
DRY_RUN=0
|
||||||
START_FROM=""
|
START_FROM=""
|
||||||
ONLY=""
|
ONLY=""
|
||||||
|
SKIP_CPS=0
|
||||||
|
ONLY_CPS=0
|
||||||
|
|
||||||
while [ $# -gt 0 ]; do
|
while [ $# -gt 0 ]; do
|
||||||
case "$1" in
|
case "$1" in
|
||||||
--dry-run) DRY_RUN=1; shift ;;
|
--dry-run) DRY_RUN=1; shift ;;
|
||||||
--start-from) START_FROM="$2"; shift 2 ;;
|
--start-from) START_FROM="$2"; shift 2 ;;
|
||||||
--only) ONLY="$2"; shift 2 ;;
|
--only) ONLY="$2"; shift 2 ;;
|
||||||
|
--skip-cps) SKIP_CPS=1; shift ;;
|
||||||
|
--only-cps) ONLY_CPS=1; shift ;;
|
||||||
-h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
-h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
||||||
esac
|
esac
|
||||||
|
|
@ -68,17 +83,17 @@ fi
|
||||||
|
|
||||||
# --- 3. node ordering ---------------------------------------------------------
|
# --- 3. node ordering ---------------------------------------------------------
|
||||||
# Ordering rule:
|
# Ordering rule:
|
||||||
# - workers only
|
# - workers first (non-Stalwart, then Stalwart-hosting fsn1 last)
|
||||||
# - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last
|
# - control-plane nodes AFTER workers, one at a time, etcd followers first,
|
||||||
# - ALL control-plane nodes are EXCLUDED and never updated by this script. Rationale:
|
# leader last. Rationale (see CP_UPDATE_PROCEDURE.md):
|
||||||
# * kine/etcd write-path is sensitive to concurrent drains (see kine thundering-herd
|
# * two of three CPs down = etcd quorum loss = write outage. Only one CP
|
||||||
# guardrails in OS_UPDATE_PROCEDURE.md, added after DEV-495).
|
# cordoned/rebooting at a time.
|
||||||
# * rebooting a CP removes one apiserver — needs external liveness monitoring.
|
# * draining a follower is a no-op for raft leadership; draining the leader
|
||||||
# * CP nodes host their own StatefulSet workloads that need batched eviction.
|
# forces a re-election, so doing followers first minimises leader flapping.
|
||||||
# Since DEV-510 (2026-08-22) the cluster runs HA control plane (cp-1/cp-2/cp-3). The
|
# Since DEV-510 (2026-08-22) the cluster runs HA embedded etcd across
|
||||||
# exclusion here is role-based so ALL current and future CPs are covered automatically.
|
# cp-1/cp-2/cp-3 with 3-node quorum. This script delegates all CP steps to
|
||||||
# To update a CP, use `scripts/os-update/update-cp-1.sh` (currently cp-1-only; will be
|
# update-cp-node.sh, which handles swap (idempotent), preflight, drain, apt,
|
||||||
# generalized to any CP as part of the HA-aware CP OS-update procedure follow-up).
|
# reboot with /livez polling, etcd rejoin check, and finalize.
|
||||||
STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true)
|
STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true)
|
||||||
# If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers.
|
# If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers.
|
||||||
|
|
||||||
|
|
@ -108,38 +123,111 @@ for n in $ALL_NODES; do
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
|
|
||||||
ORDER=("${workers[@]}" "${stalwart_workers[@]}")
|
WORKER_ORDER=("${workers[@]}" "${stalwart_workers[@]}")
|
||||||
# NOTE: CP nodes intentionally excluded. To update a CP, see CP1_UPDATE_PROCEDURE.md.
|
|
||||||
|
# --- 3b. control-plane ordering (leader last) --------------------------------
|
||||||
|
# CP-node → public IP map. Kept out-of-cluster so a full-cluster incident
|
||||||
|
# doesn't hide etcd from the operator machine.
|
||||||
|
declare -A CP_HOST=(
|
||||||
|
[k3s-cp-1]="178.105.17.239"
|
||||||
|
[k3s-cp-2]="188.245.85.199"
|
||||||
|
[k3s-cp-3]="49.13.92.162"
|
||||||
|
)
|
||||||
|
|
||||||
|
etcd_leader_ip() {
|
||||||
|
# Query etcd leader endpoint from the operator's control-plane host. Uses
|
||||||
|
# the same TLS material as update-cp-node.sh. Prints "IP" on success,
|
||||||
|
# nothing on failure.
|
||||||
|
ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new -o BatchMode=yes \
|
||||||
|
root@"$CONTROL_PLANE_HOST" \
|
||||||
|
'ETCDCTL_API=3 etcdctl \
|
||||||
|
--endpoints=https://127.0.0.1:2379 \
|
||||||
|
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
|
||||||
|
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
|
||||||
|
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
|
||||||
|
endpoint status --cluster -w simple 2>/dev/null' \
|
||||||
|
| awk -F, '$5 ~ /true/ {print $1}' \
|
||||||
|
| sed -E 's|https?://([^:]+):.*|\1|'
|
||||||
|
}
|
||||||
|
|
||||||
|
CP_ORDER=()
|
||||||
if [ -n "$CP_NODES" ]; then
|
if [ -n "$CP_NODES" ]; then
|
||||||
log "[plan] EXCLUDING control-plane nodes: $(echo $CP_NODES | tr '\n' ' ')— use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)"
|
LEADER_IP="$(etcd_leader_ip || true)"
|
||||||
|
leader_node=""
|
||||||
|
followers=()
|
||||||
|
for n in $CP_NODES; do
|
||||||
|
ip="${CP_HOST[$n]:-}"
|
||||||
|
if [ -n "$ip" ] && [ "$ip" = "$LEADER_IP" ]; then
|
||||||
|
leader_node="$n"
|
||||||
|
else
|
||||||
|
followers+=("$n")
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [ -n "$leader_node" ]; then
|
||||||
|
CP_ORDER=("${followers[@]}" "$leader_node")
|
||||||
|
log "[plan] etcd leader is $leader_node ($LEADER_IP) — updating followers first, leader last"
|
||||||
|
else
|
||||||
|
# Couldn't identify leader — refuse to guess. Fall back to no CPs.
|
||||||
|
CP_ORDER=()
|
||||||
|
log "[plan] WARN could not identify etcd leader — skipping CP phase this cycle"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Route --only <node> automatically to worker or CP list.
|
||||||
if [ -n "$ONLY" ]; then
|
if [ -n "$ONLY" ]; then
|
||||||
ORDER=("$ONLY")
|
if is_cp "$ONLY"; then
|
||||||
|
CP_ORDER=("$ONLY")
|
||||||
|
WORKER_ORDER=()
|
||||||
|
else
|
||||||
|
WORKER_ORDER=("$ONLY")
|
||||||
|
CP_ORDER=()
|
||||||
|
fi
|
||||||
elif [ -n "$START_FROM" ]; then
|
elif [ -n "$START_FROM" ]; then
|
||||||
new=()
|
new=()
|
||||||
skip=1
|
skip=1
|
||||||
for n in "${ORDER[@]}"; do
|
for n in "${WORKER_ORDER[@]}"; do
|
||||||
[ "$n" = "$START_FROM" ] && skip=0
|
[ "$n" = "$START_FROM" ] && skip=0
|
||||||
[ $skip -eq 0 ] && new+=("$n")
|
[ $skip -eq 0 ] && new+=("$n")
|
||||||
done
|
done
|
||||||
ORDER=("${new[@]}")
|
WORKER_ORDER=("${new[@]}")
|
||||||
|
# If START_FROM is a CP, trim CP_ORDER instead.
|
||||||
|
if is_cp "$START_FROM"; then
|
||||||
|
new_cp=()
|
||||||
|
skip=1
|
||||||
|
for n in "${CP_ORDER[@]:-}"; do
|
||||||
|
[ -z "$n" ] && continue
|
||||||
|
[ "$n" = "$START_FROM" ] && skip=0
|
||||||
|
[ $skip -eq 0 ] && new_cp+=("$n")
|
||||||
|
done
|
||||||
|
CP_ORDER=("${new_cp[@]}")
|
||||||
|
WORKER_ORDER=()
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
log "[plan] ordered nodes (${#ORDER[@]}): ${ORDER[*]}"
|
[ "$SKIP_CPS" -eq 1 ] && CP_ORDER=()
|
||||||
printf '%s\n' "${ORDER[@]}" > "$LOG_DIR/plan.txt"
|
[ "$ONLY_CPS" -eq 1 ] && WORKER_ORDER=()
|
||||||
|
|
||||||
|
log "[plan] workers (${#WORKER_ORDER[@]}): ${WORKER_ORDER[*]:-none}"
|
||||||
|
log "[plan] control-planes (${#CP_ORDER[@]}): ${CP_ORDER[*]:-none}"
|
||||||
|
{
|
||||||
|
echo "# workers"
|
||||||
|
printf '%s\n' "${WORKER_ORDER[@]:-}"
|
||||||
|
echo "# control-planes"
|
||||||
|
printf '%s\n' "${CP_ORDER[@]:-}"
|
||||||
|
} > "$LOG_DIR/plan.txt"
|
||||||
|
|
||||||
if [ "$DRY_RUN" -eq 1 ]; then
|
if [ "$DRY_RUN" -eq 1 ]; then
|
||||||
log "DRY-RUN — plan written, no node touched. Exiting."
|
log "DRY-RUN — plan written, no node touched. Exiting."
|
||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# --- 4. per-node loop ---------------------------------------------------------
|
# --- 4. worker loop ----------------------------------------------------------
|
||||||
updated=()
|
updated=()
|
||||||
skipped=()
|
skipped=()
|
||||||
for n in "${ORDER[@]}"; do
|
for n in "${WORKER_ORDER[@]}"; do
|
||||||
|
[ -z "$n" ] && continue
|
||||||
log "===================================================================="
|
log "===================================================================="
|
||||||
log "==> updating $n"
|
log "==> updating worker $n"
|
||||||
log "===================================================================="
|
log "===================================================================="
|
||||||
NODE_LOG="$LOG_DIR/${n}.log"
|
NODE_LOG="$LOG_DIR/${n}.log"
|
||||||
set +e
|
set +e
|
||||||
|
|
@ -153,6 +241,50 @@ for n in "${ORDER[@]}"; do
|
||||||
esac
|
esac
|
||||||
done
|
done
|
||||||
|
|
||||||
|
# --- 4b. control-plane loop --------------------------------------------------
|
||||||
|
# One CP at a time, followers first, leader last. Delegates to update-cp-node.sh
|
||||||
|
# --run (Phase A add-swap is idempotent, Phase B..F run with ASSUME_YES=1).
|
||||||
|
# Between CPs we run cluster-health.sh and re-verify etcd cluster status; halt
|
||||||
|
# on any yellow signal.
|
||||||
|
if [ ${#CP_ORDER[@]} -gt 0 ]; then
|
||||||
|
log "===================================================================="
|
||||||
|
log "==> control-plane phase (${#CP_ORDER[@]} nodes): ${CP_ORDER[*]}"
|
||||||
|
log "===================================================================="
|
||||||
|
first_cp=1
|
||||||
|
for cp in "${CP_ORDER[@]}"; do
|
||||||
|
[ -z "$cp" ] && continue
|
||||||
|
if [ $first_cp -eq 0 ]; then
|
||||||
|
log "-- health gate between CPs (settle ${POST_CP_WAIT_SECONDS}s, then re-check)"
|
||||||
|
sleep "$POST_CP_WAIT_SECONDS"
|
||||||
|
if ! RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh" 2>&1 | tee "$LOG_DIR/inter-cp-health-before-${cp}.log"; then
|
||||||
|
die "cluster health degraded between CPs — halting before $cp. See $LOG_DIR/"
|
||||||
|
fi
|
||||||
|
# Re-check etcd cluster status; every member must still be started.
|
||||||
|
ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new root@"$CONTROL_PLANE_HOST" \
|
||||||
|
'ETCDCTL_API=3 etcdctl \
|
||||||
|
--endpoints=https://127.0.0.1:2379 \
|
||||||
|
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
|
||||||
|
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
|
||||||
|
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
|
||||||
|
endpoint status --cluster -w table' 2>&1 | tee -a "$LOG_DIR/inter-cp-health-before-${cp}.log" || \
|
||||||
|
die "etcd endpoint status failed between CPs — halting before $cp"
|
||||||
|
fi
|
||||||
|
log "===================================================================="
|
||||||
|
log "==> updating control-plane $cp"
|
||||||
|
log "===================================================================="
|
||||||
|
CP_LOG="$LOG_DIR/${cp}.log"
|
||||||
|
set +e
|
||||||
|
ASSUME_YES=1 "$SCRIPT_DIR/update-cp-node.sh" "$cp" --run 2>&1 | tee "$CP_LOG"
|
||||||
|
rc=${PIPESTATUS[0]}
|
||||||
|
set -e
|
||||||
|
case $rc in
|
||||||
|
0) updated+=("$cp") ;;
|
||||||
|
*) die "update-cp-node.sh failed for $cp (rc=$rc). Cycle halted. See $CP_LOG" ;;
|
||||||
|
esac
|
||||||
|
first_cp=0
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
# --- 5. finalization ----------------------------------------------------------
|
# --- 5. finalization ----------------------------------------------------------
|
||||||
log "===================================================================="
|
log "===================================================================="
|
||||||
log "==> finalization"
|
log "==> finalization"
|
||||||
|
|
@ -169,6 +301,8 @@ log "[final] apt history digest"
|
||||||
# resolve ssh target via a mini-eval of node_ssh_target-equivalent
|
# resolve ssh target via a mini-eval of node_ssh_target-equivalent
|
||||||
case "$n" in
|
case "$n" in
|
||||||
k3s-cp-1) t="root@178.105.17.239" ;;
|
k3s-cp-1) t="root@178.105.17.239" ;;
|
||||||
|
k3s-cp-2) t="root@188.245.85.199" ;;
|
||||||
|
k3s-cp-3) t="root@49.13.92.162" ;;
|
||||||
k3s-worker-1) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
|
k3s-worker-1) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
|
||||||
k3s-worker-2) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
|
k3s-worker-2) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
|
||||||
k3s-worker-3) t="root@167.233.121.121" ;;
|
k3s-worker-3) t="root@167.233.121.121" ;;
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue