From 05000b3e06adece332ffec096f34b6b59d739248 Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 16 Aug 2026 19:28:46 +0000 Subject: [PATCH] docs(infra): design cp-1 OS update procedure (DEV-496) cp-1 is excluded from os-update.sh because a one-shot drain would trigger the kine cascade documented in DEV-495. This adds: - CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful eviction (Phase C), drain+apt (Phase D), reboot with external livez monitor (Phase E), uncordon+verify (Phase F), and rollback paths. - scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the same /tmp/os-update-cp-1-.log contract as update-node.sh; supports --dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot, --finalize, --run. - os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1 script; kine thundering-herd guardrails preserved. - OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure. Execution requires separate board approval; this change is design + dry-run artifact only. Co-Authored-By: Paperclip --- infrastructure/CP1_UPDATE_PROCEDURE.md | 382 ++++++++++++ infrastructure/OS_UPDATE_PROCEDURE.md | 16 +- infrastructure/scripts/os-update/os-update.sh | 12 +- .../scripts/os-update/update-cp-1.sh | 577 ++++++++++++++++++ 4 files changed, 978 insertions(+), 9 deletions(-) create mode 100644 infrastructure/CP1_UPDATE_PROCEDURE.md create mode 100755 infrastructure/scripts/os-update/update-cp-1.sh diff --git a/infrastructure/CP1_UPDATE_PROCEDURE.md b/infrastructure/CP1_UPDATE_PROCEDURE.md new file mode 100644 index 0000000..55fa313 --- /dev/null +++ b/infrastructure/CP1_UPDATE_PROCEDURE.md @@ -0,0 +1,382 @@ +# k3s-cp-1 OS Update Procedure + +**Purpose:** apply the weekly Ubuntu OS updates to `k3s-cp-1`, the **sole** control-plane node of the k3s cluster, without triggering the kine/SQLite thundering-herd cascade documented in [DEV-495](/DEV/issues/DEV-495) and without losing api-server access longer than a normal reboot. + +**Audience:** the CTO agent, or an operator with root SSH to the cluster. Execution is board-approval-gated — see `## Approval gates` below. + +**Why cp-1 is not covered by the standard [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md):** + +- cp-1 has 3.7 GiB RAM and **zero swap** — during the 2026-08-16 worker-3 drain, `MemAvailable` dropped to 43 MiB while kine backed up on writes. cp-1's own drain evicts **more** state at once than any worker drain. +- cp-1 currently hosts 5 StatefulSets (`harbor-database`, `harbor-redis`, `nextcloud-redis-replicas`, `stalwart`, `stalwart-postgres`) plus 8 single-replica Deployments — all backed by Hetzner-CSI RWO volumes pinned to the fsn1 datacenter. A one-shot `kubectl drain` reschedules >13 pods simultaneously → guaranteed kine cascade. +- Rebooting cp-1 removes the entire kube-apiserver for the reboot window (~90–180 s expected). There is no fallback control-plane. + +The four kine thundering-herd guardrails from [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md) still apply here. This procedure implements them for cp-1. + +--- + +## Scope + +**In scope** + +- Add a durable ≥ 2 GiB swapfile on cp-1 (idempotent one-off — `--add-swap`). +- Pre-drain redistribution of stateful pods off cp-1, one at a time. +- k3s SQLite datastore snapshot as the restore point. +- `apt-get update/upgrade/dist-upgrade/autoremove` on cp-1. +- Controlled reboot with external liveness monitoring. +- Uncordon + cluster health verify. + +**Out of scope — do NOT do here** + +- Any change to `/etc/rancher/k3s/*`, `/etc/systemd/system/k3s*.service*`, or the k3s binary version. k3s upgrades go through system-upgrade-controller (see `K3S_OPERATIONS.md`). +- Any change to manifests under `apps/`, `infrastructure/`, or applied via ArgoCD. +- Deleting PVs / PVCs. StatefulSets that get rescheduled off cp-1 stay on their new node — do NOT try to move them back. +- Fixing application-level problems ("harbor-core is CrashLoopBackOff after apt") — that's an app problem, escalate. +- Rebuilding the node. If cp-1 does not return after reboot, escalate; the rebuild path is `ADD_WORKER_NODE.md` plus board approval, not this document. + +--- + +## Approval gates + +This document has **two** independent gates. Neither happens without explicit board approval on the corresponding Paperclip issue: + +1. **Add swap (Phase A).** Non-invasive, non-state-mutating, kubelet already runs with `failSwapOn=false`. Requires board approval because it modifies the CP node. +2. **Full OS update (Phases B–F).** Requires board approval because it drains cp-1 and reboots the sole api-server. Do NOT execute without an explicit `request_board_approval` acceptance on the execution ticket. + +Both gates are independent — swap can (and should) be added first, in a quiet window, before the full update is scheduled. + +--- + +## Current cp-1 workload (snapshot 2026-08-16) + +Re-derive live before any execution — this table is a design reference only. + +| Kind | Namespace/Name | Notes | +|------|-----------------|-------| +| StatefulSet | `harbor/harbor-database-0` | Harbor Postgres, RWO 10 GiB fsn1 | +| StatefulSet | `harbor/harbor-redis-0` | Harbor cache, RWO 10 GiB fsn1 | +| StatefulSet | `nextcloud/nextcloud-redis-replicas-0` | Nextcloud Redis replica, RWO 10 GiB fsn1 | +| StatefulSet | `stalwart/stalwart-0` | Mail server, RWO 20 GiB fsn1 — **critical, move last** | +| StatefulSet | `stalwart/stalwart-postgres-0` | Mail metadata DB, RWO 10 GiB fsn1 | +| Deployment | `harbor/harbor-core` | Registry frontend, single replica | +| Deployment | `harbor/harbor-jobservice` | Registry job worker, single replica | +| Deployment | `monitoring/alertmanager` | Alertmanager, single replica | +| Deployment | `monitoring/loki` | Loki (single-binary), single replica | +| Deployment | `monitoring/prometheus` | Prometheus, single replica | +| Deployment | `observability/blackbox-exporter-*` | Blackbox exporter | +| Deployment | `opencloud/tika` | Apache Tika, single replica | +| Deployment | `passbolt/passbolt` | Passbolt web, single replica | +| DaemonSet | `kube-system/hcloud-csi-node-*` | Ignored by drain | +| DaemonSet | `kube-system/svclb-*` | Ignored by drain | +| DaemonSet | `observability/*-node-exporter-*` | Ignored by drain | +| DaemonSet | `observability/loki-stack-promtail-*` | Ignored by drain | + +**Absorption capacity (fsn1 workers only — nbg1 worker-4 cannot receive fsn1 RWO volumes):** + +- `k3s-worker-5`: ~3.0 GiB free, 6 pods scheduled — primary target for the two heaviest stateful pods. +- `k3s-worker-2`: ~2.0 GiB free, 14 pods. +- `k3s-worker-3`: ~2.3 GiB free, 15 pods, 2 sts. +- `k3s-worker-1`: ~1.9 GiB free, 23 pods, 6 sts — **avoid piling more onto this one**. + +--- + +## Phase A — Add swap (one-off, idempotent) + +**Goal:** eliminate the "3.7 GiB, no swap" underlying constraint before we ever try to drain cp-1. + +**Preconditions:** + +- kubelet in this k3s already runs with `failSwapOn=false` (confirmed via `/api/v1/nodes/k3s-cp-1/proxy/configz`) — enabling swap does NOT break the kubelet. +- cp-1 `/` has ≥ 10 GiB free (currently 21 GiB free of 75 GiB). +- Board approval on the swap-add ticket. + +**Sizing:** default **4 GiB** swap. Rationale — cp-1 baseline (k3s + hosted apps) already sits at ~3.1 GiB used; 4 GiB swap gives us headroom for the drain-eviction transient without inflating disk usage past ~10 % of `/`. Minimum acceptable per guardrail: 2 GiB. + +**Location:** `/swapfile` (root filesystem). Not a separate partition — reversible, no LVM changes, no ext4/xfs migration. + +**Steps (encoded in `update-cp-1.sh --add-swap`):** + +```bash +# On cp-1 as root: +SWAPFILE=/swapfile +SIZE_MB=4096 + +# Idempotency: skip if a swapfile of the target size already exists and is on. +if swapon --show=NAME | grep -qx "$SWAPFILE"; then + echo "swap already on at $SWAPFILE" + exit 0 +fi + +# Create the file with fallocate; fall back to dd for filesystems without fallocate support. +fallocate -l "${SIZE_MB}M" "$SWAPFILE" || dd if=/dev/zero of="$SWAPFILE" bs=1M count="$SIZE_MB" status=progress +chmod 600 "$SWAPFILE" +mkswap "$SWAPFILE" +swapon "$SWAPFILE" + +# Persist across reboot. Guard against duplicate fstab entries. +grep -q "^$SWAPFILE " /etc/fstab || echo "$SWAPFILE none swap sw 0 0" >> /etc/fstab + +# Moderate swappiness — we want swap as a safety net, not aggressive paging. +sysctl -w vm.swappiness=10 +grep -q '^vm.swappiness' /etc/sysctl.d/99-k3s-swap.conf 2>/dev/null || { + echo 'vm.swappiness=10' > /etc/sysctl.d/99-k3s-swap.conf +} + +# Verify. +free -h +swapon --show +``` + +**Rollback for Phase A:** `swapoff /swapfile && rm /swapfile` and remove the fstab line. This is safe at any time — swap is a soft resource. + +**Verification after Phase A:** + +- `free -h` shows `Swap: 4.0Gi` used ≈ 0. +- `swapon --show` shows `/swapfile 4G`. +- `sysctl vm.swappiness` returns `10`. +- kubelet still Ready (`kubectl get node k3s-cp-1`). +- No new `MemoryPressure` condition. + +--- + +## Phase B — Preflight for the full OS update + +Everything from here on runs from the CTO's operator machine (not from cp-1 itself, because we lose kubectl during the reboot). SSH to cp-1 is fine — kubectl calls issued from cp-1 during the pre-drain phase are fine and get logged into `/tmp/os-update-cp-1-.log`. + +**Log directory contract** (same as `update-node.sh`): every command's stdout+stderr goes to `/tmp/os-update-cp-1-.log` on cp-1. Attach that log to the Paperclip execution ticket at the end. + +Run `update-cp-1.sh --preflight`: + +1. Cluster is currently healthy: `cluster-health.sh` returns 0. +2. All 5 fsn1 workers are `Ready` (worker-1, worker-2, worker-3, worker-5, cp-1). worker-4 is nbg1 and irrelevant here. +3. Each fsn1 worker has ≥ 500 MiB `MemAvailable`. +4. `time kubectl get nodes` returns in ≤ 2 s (kine is not already stressed). +5. cp-1 has swap on — abort if `free -h` shows `Swap: 0B`. +6. **k3s datastore snapshot.** cp-1 runs k3s in embedded-SQLite mode (`--datastore-endpoint` is unset). `k3s etcd-snapshot save --name pre-cp1-os-update-` produces a copy of the SQLite file under `/var/lib/rancher/k3s/server/db/snapshots/`. This is our restore point. Snapshots older than 30 days are pruned in Phase F. + +If any preflight check fails: STOP. Do not proceed. Do not add pods to cp-1 to "rebalance later" — that's a separate task. + +--- + +## Phase C — Move stateful pods off cp-1 (batched) + +**Rule** ([[k3s-drain-kine-thundering-herd]] guardrail #1): no drain that reschedules > 3 StatefulSets at once. So we DO NOT `kubectl drain k3s-cp-1` while stateful pods still live on it. We move them one at a time first. + +**Cordon cp-1 immediately.** Cordon only prevents *new* scheduling — existing pods stay put. Cordoning first ensures that when we delete a stateful pod, the StatefulSet controller cannot re-create it back on cp-1. + +```bash +kubectl cordon k3s-cp-1 +``` + +**Order of eviction (idle → active, tiny → heavy):** + +| # | Pod | Why in this position | +|---|-----|----------------------| +| 1 | `harbor/harbor-redis-0` | Idle cache, cold restart is instant, low write pressure | +| 2 | `nextcloud/nextcloud-redis-replicas-0` | Replica-1 of a Redis replicaset — non-primary, safe to bounce | +| 3 | `harbor/harbor-database-0` | Registry Postgres, mostly idle (registry pulls, not writes) | +| 4 | `stalwart/stalwart-postgres-0` | Mail metadata DB — active but recoverable; move before the mail server itself | +| 5 | `stalwart/stalwart-0` | Mail server, most important. Save for last so mail keeps flowing until the very end | + +For each pod: + +```bash +# 1. Delete the pod — StatefulSet controller will re-create it on a schedulable fsn1 worker. +kubectl -n "$NS" delete pod "$POD" --wait=false + +# 2. Wait for the *new* pod (same name — StatefulSets keep identities) to be scheduled +# somewhere OTHER than cp-1 and reach Ready=True. +deadline=$(( $(date +%s) + 300 )) +while [ $(date +%s) -lt $deadline ]; do + new_node=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.spec.nodeName}' 2>/dev/null || true) + ready=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true) + if [ -n "$new_node" ] && [ "$new_node" != "k3s-cp-1" ] && [ "$ready" = "True" ]; then + echo " $POD -> $new_node OK" + break + fi + sleep 5 +done +[ "$ready" = "True" ] || die "pod $NS/$POD did not reach Ready on a non-cp-1 node in 300s — HALT" + +# 3. Kine health probe — abort the batch if the api server is getting slow. +t=$( { time kubectl get nodes >/dev/null; } 2>&1 | awk '/real/{print $2}' ) +# t is like "0m1.234s" — if the m field is >0 or the s field is >5, halt. +# See update-cp-1.sh for the exact parse; > 5 s -> halt the whole cycle. + +# 4. Free-memory probe on cp-1. +avail=$(ssh root@cp-1 "free -m | awk '/Mem:/{print \$7}'") +[ "$avail" -lt 200 ] && die "cp-1 MemAvailable dropped below 200 MiB — HALT" + +# 5. Settle pause before the next eviction — kine needs to catch up on the write +# burst from the just-attached PV and the just-scheduled pod. +sleep 60 +``` + +**Halt conditions during Phase C** (any one → STOP, do NOT proceed to Phase D): + +- `kubectl get nodes` takes > 5 s. +- cp-1 `MemAvailable` drops below 200 MiB. +- Any target worker enters `MemoryPressure=True` or `DiskPressure=True`. +- Any stateful pod does not reach Ready on a new node within 5 minutes (may indicate PV-attach or PDB issue). + +If we halt: leave cp-1 **cordoned** but do not reboot. The pods that have already moved stay where they are; the pods that haven't will still be on cp-1. Escalate on the execution ticket with `blocked` and name the halt condition. + +--- + +## Phase D — Drain remaining pods + apt + +Once all 5 stateful pods are off cp-1 and cluster health is green: + +```bash +# Drain everything else — only Deployment pods left (single-replica each, +# no PV attach, so reschedule is fast). +kubectl drain k3s-cp-1 \ + --ignore-daemonsets \ + --delete-emptydir-data \ + --timeout="${DRAIN_TIMEOUT_SECONDS:-600}s" +``` + +If drain reports a PDB block: do NOT `--force`. Uncordon cp-1, mark the run as `blocked` on the PDB, and escalate. This is a design bug in the workload's PDB — fix it separately. + +**apt on cp-1** (mirrors `update-node.sh` step 3; `update-cp-1.sh --apt` runs this via `ssh root@cp-1 bash -s`): + +```bash +export DEBIAN_FRONTEND=noninteractive +APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold' +uname -r > /root/pre-apt-kernel # for rollback reference +dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv + +if dpkg --audit | grep -qE .; then dpkg --configure -a || true; fi +apt-get update +apt-get $APT_OPTS upgrade || { apt-get $APT_OPTS -f install; apt-get $APT_OPTS upgrade; } +apt-get $APT_OPTS dist-upgrade +apt-get $APT_OPTS autoremove --purge +apt-get clean +[ -f /var/run/reboot-required ] && echo REBOOT_REQUIRED=yes || echo REBOOT_REQUIRED=no +``` + +`pre-apt-kernel` and `pre-apt-packages.tsv` are the local rollback references — see Rollback below. + +--- + +## Phase E — Reboot handling (api-server unavailability) + +**Duration expectation:** 90–180 s of api-server unavailability. The operator machine will get `Unable to connect to the server` from kubectl during this window — that is expected, not an alarm. + +**Escalation trigger:** api-server not back on `/livez` after **10 minutes** → escalate. First check Hetzner console via `hcloud server describe k3s-cp-1` for boot state; if kernel-panic / initramfs, use grub previous-kernel path (see Rollback). Do NOT rebuild the node. + +**Steps (executed by `update-cp-1.sh --reboot`):** + +```bash +# 1. Tell cp-1 to reboot. This SSH will hang up mid-command — that's fine. +ssh $SSH_OPTS root@cp-1 'systemctl reboot' || true + +# 2. Deliberate 15 s pause — SSH needs to actually drop, don't race the poll. +sleep 15 + +# 3. Poll the api-server livez from the OPERATOR machine (not from cp-1). +# Using --insecure (`-k`) because the server cert is self-signed by k3s. +deadline=$(( $(date +%s) + ${REBOOT_MAX_WAIT_SECONDS:-600} )) +while [ $(date +%s) -lt $deadline ]; do + code=$(curl -sk -o /dev/null -w '%{http_code}' https://178.105.17.239:6443/livez 2>/dev/null || echo 000) + if [ "$code" = "200" ]; then + echo " api-server /livez=200" + break + fi + sleep 5 +done +[ "$code" = "200" ] || die "api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate; check hcloud console" + +# 4. Wait for kubelet Ready on cp-1 from the api-server view. +deadline=$(( $(date +%s) + 300 )) +while [ $(date +%s) -lt $deadline ]; do + ready=$(kubectl get node k3s-cp-1 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown) + [ "$ready" = "True" ] && break + sleep 5 +done +[ "$ready" = "True" ] || die "kubelet on cp-1 never returned Ready — escalate (do NOT change k3s config)" +``` + +**What to do if the api-server takes 3–10 minutes:** it may be normal on this box (large SQLite → replay). Wait it out. Do NOT restart k3s to "help" — kine SQLite replay is stateful and interrupting it can corrupt the DB. See [[k3s-drain-kine-thundering-herd]] recovery: doing nothing is a valid path. + +**What to do if it takes > 10 minutes:** escalate; run the Hetzner-console diagnosis; if the console shows a bootable Ubuntu but the k3s service is failing, that's the boundary — this procedure stops here. Follow `K3S_OPERATIONS.md` for the k3s recovery path. + +--- + +## Phase F — Uncordon + verify + finalize + +```bash +# Uncordon cp-1. StatefulSet pods will NOT be moved back — that's correct +# behaviour, they were moved for a reason. Rebalancing is a separate task. +kubectl uncordon k3s-cp-1 + +# Settle wait — kine needs to process the flood of "node schedulable again" events. +sleep "${POST_UNCORDON_WAIT_SECONDS:-180}" + +# Full cluster health. +RETRY_ON_TRANSIENT=1 infrastructure/scripts/os-update/cluster-health.sh + +# Print the apt history for audit. +ssh root@cp-1 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60' + +# Prune snapshots older than 30 days. +ssh root@cp-1 'find /var/lib/rancher/k3s/server/db/snapshots/ -type f -mtime +30 -name "pre-*" -print' +# (list only — actual deletion is a manual review after the run) +``` + +Attach the `/tmp/os-update-cp-1-.log` and the ordered ledger of moved pods to the execution ticket. Mark the ticket `done` on green health, or `blocked` naming the specific residual issue on red. + +--- + +## Rollback / recovery + +### If apt broke a package + +- On cp-1, `dpkg --audit` to find half-configured packages. +- `dpkg --configure -a`, then `apt-get -f install`. +- If a specific package broke and you know the previous version from `/root/pre-apt-packages.tsv`, `apt-get install =`. + +### If the new kernel does not boot + +- Hetzner cloud console: send `hcloud server request-console k3s-cp-1` → get VNC URL, watch boot. +- If grub is up, select the previous-kernel entry. Ubuntu keeps ≥ 1 old kernel installed by default (we verified `6.8.0-137` current; the previous `6.8.0-124` was in use during the DEV-478 cycle). +- Once booted on the old kernel, `apt-get remove` the broken kernel and pin the working one: + ```bash + apt-mark hold linux-image- linux-headers- + ``` +- Escalate on the ticket regardless — a kernel rollback is a follow-up investigation, not a "done" outcome. + +### If the node does not return at all + +- Do NOT `hcloud server delete`. Do NOT re-provision. +- Escalate to the board with the Hetzner console output and the `/tmp/os-update-cp-1-*.log`. +- The `--cluster-reset --cluster-reset-restore-path=` recovery path exists (see `K3S_OPERATIONS.md`) but requires board approval per stateful-service safety rules. The pre-flight snapshot from Phase B is the restore point. + +### If Phase C halted mid-eviction + +- cp-1 is cordoned, some stateful pods have moved, some haven't. Cluster is functional. +- Uncordon cp-1 (`kubectl uncordon k3s-cp-1`) — StatefulSets that stayed on cp-1 keep running, moved ones stay where they went. +- Do NOT proceed to Phase D. Open a follow-up ticket with the halt condition. Retry in the next maintenance window after fixing the halt condition. + +--- + +## Automation entry points + +- `infrastructure/scripts/os-update/update-cp-1.sh` — this whole flow, with subcommands: + - `--add-swap` — Phase A only, idempotent, safe standalone. + - `--dry-run` — walk Phases B/C/D/E/F printing exactly what would be done without touching anything. Safe to run any time; used for design review. + - `--preflight` — Phase B only. + - `--drain-stateful` — Phase C only (cordon + batched stateful moves). + - `--apt` — Phase D apt commands only (requires cp-1 already fully drained). + - `--reboot` — Phase E only (requires apt already done). + - `--finalize` — Phase F only (uncordon + verify). + - `--run` — do all phases in order, with a confirmation prompt between each unless `ASSUME_YES=1`. + +The script follows the same log-dir contract as `update-node.sh` (`/tmp/os-update-cp-1-.log`). + +--- + +## Change history + +| Date | Change | By | +|------|--------|-----| +| 2026-08-16 | Initial cp-1 update design (DEV-496) | CTO agent | diff --git a/infrastructure/OS_UPDATE_PROCEDURE.md b/infrastructure/OS_UPDATE_PROCEDURE.md index b158109..f50c670 100644 --- a/infrastructure/OS_UPDATE_PROCEDURE.md +++ b/infrastructure/OS_UPDATE_PROCEDURE.md @@ -237,10 +237,13 @@ Only proceed to the next node when ALL of the above are green. If not: ### 8. Control plane special handling -`k3s-cp-1` is the single control plane node. During its reboot: -- The kube-apiserver is unavailable — kubectl commands from the operator machine will error out. -- The `kubectl` waits in step 5/7 must run from a machine that is NOT the control plane, or must be scheduled after the control plane's SSH is back and `curl -k https://localhost:6443/healthz` returns `ok`. -- Skip the drain for DaemonSet pods on the control plane (`--ignore-daemonsets` covers that), but hosted apps that scheduled onto CP (rare — verify with `kubectl get pods -A --field-selector spec.nodeName=k3s-cp-1`) will be evicted. +`k3s-cp-1` is the single control plane node and is **NOT** updated by the weekly `os-update.sh` cycle — it has its own dedicated procedure and script. See `CP1_UPDATE_PROCEDURE.md` and `scripts/os-update/update-cp-1.sh`. Reasons: + +- cp-1 hosts many stateful pods (currently 5 StatefulSets + 8 single-replica Deployments) — a one-shot `kubectl drain` would trigger the kine/SQLite cascade documented in the "Kine thundering-herd guardrails" section above (DEV-495). +- cp-1 has 3.7 GiB RAM and (until Phase A of the cp-1 procedure is done) zero swap. The cp-1 procedure adds a durable 4 GiB swapfile before draining. +- Rebooting cp-1 removes the entire kube-apiserver — the cp-1 procedure polls `/livez` from an external operator machine, not from cp-1 itself. + +`os-update.sh` will `SKIP` cp-1 and print a pointer to `CP1_UPDATE_PROCEDURE.md`. --- @@ -301,8 +304,9 @@ If a node was skipped or errored → mark `blocked` with the unblock action, or ## Automation entry points -- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from `. Use `--dry-run` to print the plan without touching anything. -- `infrastructure/scripts/os-update/update-node.sh ` — single-node update (all 7 per-node steps). Callable standalone for retry. +- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from `. Use `--dry-run` to print the plan without touching anything. Explicitly skips `k3s-cp-1`. +- `infrastructure/scripts/os-update/update-node.sh ` — single-node update (all 7 per-node steps). Callable standalone for retry. Not for `k3s-cp-1`. +- `infrastructure/scripts/os-update/update-cp-1.sh` — dedicated cp-1 update flow (add swap, batched stateful eviction, apt, reboot with external liveness monitor). See `CP1_UPDATE_PROCEDURE.md`. - `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure. Read the script sources for the exact behavior before running them. They mirror this procedure step for step. diff --git a/infrastructure/scripts/os-update/os-update.sh b/infrastructure/scripts/os-update/os-update.sh index ef1e19d..9d0adbf 100755 --- a/infrastructure/scripts/os-update/os-update.sh +++ b/infrastructure/scripts/os-update/os-update.sh @@ -68,9 +68,11 @@ fi # --- 3. node ordering --------------------------------------------------------- # Ordering rule: -# - workers before control plane +# - workers only # - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last -# - k3s-cp-1 always last +# - k3s-cp-1 is EXCLUDED and never updated by this script — see CP1_UPDATE_PROCEDURE.md +# and scripts/os-update/update-cp-1.sh. The rationale is the kine thundering-herd +# guardrails documented in OS_UPDATE_PROCEDURE.md (added after DEV-495). STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true) # If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers. CP_NAME="k3s-cp-1" @@ -94,7 +96,11 @@ for n in $ALL_NODES; do fi done -ORDER=("${workers[@]}" "${stalwart_workers[@]}" "$CP_NAME") +ORDER=("${workers[@]}" "${stalwart_workers[@]}") +# NOTE: cp-1 intentionally excluded. To update cp-1, run `scripts/os-update/update-cp-1.sh`. +if printf '%s\n' "$ALL_NODES" | grep -qx "$CP_NAME"; then + log "[plan] EXCLUDING $CP_NAME — use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)" +fi if [ -n "$ONLY" ]; then ORDER=("$ONLY") diff --git a/infrastructure/scripts/os-update/update-cp-1.sh b/infrastructure/scripts/os-update/update-cp-1.sh new file mode 100755 index 0000000..37136ea --- /dev/null +++ b/infrastructure/scripts/os-update/update-cp-1.sh @@ -0,0 +1,577 @@ +#!/bin/bash +# update-cp-1.sh — controlled OS update for the k3s control-plane node. +# +# See ../CP1_UPDATE_PROCEDURE.md for the full design rationale. +# +# Usage: +# update-cp-1.sh --dry-run # print what would be done, touch nothing +# update-cp-1.sh --add-swap # Phase A only (idempotent, safe standalone) +# update-cp-1.sh --preflight # Phase B only +# update-cp-1.sh --drain-stateful # Phase C only (cordon + batched sts moves) +# update-cp-1.sh --apt # Phase D only (requires cp-1 fully drained) +# update-cp-1.sh --reboot # Phase E only (requires --apt reported REBOOT_REQUIRED=yes) +# update-cp-1.sh --finalize # Phase F only (uncordon + verify) +# update-cp-1.sh --run # all phases with a confirmation between each (or ASSUME_YES=1) +# +# Environment overrides: +# CP1_HOST default 178.105.17.239 +# SWAP_SIZE_MB default 4096 (>=2048 required) +# SWAP_PATH default /swapfile +# DRAIN_TIMEOUT_SECONDS default 600 +# REBOOT_MAX_WAIT_SECONDS default 600 +# POST_UNCORDON_WAIT_SECONDS default 180 +# STATEFUL_SETTLE_SECONDS default 60 (pause between batched sts moves) +# MIN_WORKER_MEM_MIB default 500 (per-worker MemAvailable floor at preflight) +# MIN_CP1_MEM_MIB default 200 (cp-1 MemAvailable floor mid-drain) +# MAX_KUBECTL_SECONDS default 5 (kine-health guardrail) +# SSH_OPTS default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new" +# ASSUME_YES=1 skip interactive confirmations in --run +# DRY_RUN=1 do not execute state-mutating commands; log-only +# +# Log directory contract (matches update-node.sh): +# Every command's stdout+stderr is tee'd to /tmp/os-update-cp-1-.log on +# the operator machine AND to /tmp/os-update-cp-1-.log on cp-1 (via the +# ssh command wrappers). Attach both to the execution ticket at the end. +# +# NEVER touches k3s config, k3s services, containerd, or any manifest. Only +# fixes it will attempt: apt/dpkg recovery on cp-1 (see Phase D). Any other +# problem → escalate and STOP. + +set -euo pipefail + +# --------------------------------------------------------------------------- # +# Config +# --------------------------------------------------------------------------- # +CP1_HOST="${CP1_HOST:-178.105.17.239}" +SWAP_SIZE_MB="${SWAP_SIZE_MB:-4096}" +SWAP_PATH="${SWAP_PATH:-/swapfile}" +DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}" +REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}" +POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}" +STATEFUL_SETTLE_SECONDS="${STATEFUL_SETTLE_SECONDS:-60}" +MIN_WORKER_MEM_MIB="${MIN_WORKER_MEM_MIB:-500}" +MIN_CP1_MEM_MIB="${MIN_CP1_MEM_MIB:-200}" +MAX_KUBECTL_SECONDS="${MAX_KUBECTL_SECONDS:-5}" +SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}" +DRY_RUN="${DRY_RUN:-0}" +ASSUME_YES="${ASSUME_YES:-0}" + +NODE="k3s-cp-1" +TS="$(date -u +%Y%m%dT%H%M%SZ)" +LOG_LOCAL="/tmp/os-update-cp-1-${TS}.log" +LOG_REMOTE="/tmp/os-update-cp-1-${TS}.log" + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +HEALTH_SCRIPT="$SCRIPT_DIR/cluster-health.sh" + +# --------------------------------------------------------------------------- # +# Stateful eviction ledger — order matters (idle -> active, small -> heavy). +# --------------------------------------------------------------------------- # +# The list is derived live from the cluster in preflight and stored to +# /tmp/cp1-stateful-plan-.txt. This ORDER is the authoritative fallback. +STATEFUL_ORDER=( + "harbor:harbor-redis-0" + "nextcloud:nextcloud-redis-replicas-0" + "harbor:harbor-database-0" + "stalwart:stalwart-postgres-0" + "stalwart:stalwart-0" +) + +# --------------------------------------------------------------------------- # +# Logging + safe-run helpers +# --------------------------------------------------------------------------- # +log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOG_LOCAL"; } +die() { log "FATAL: $*"; exit 1; } +warn() { log "WARN: $*"; } + +# run/ssh_run/kubectl_run: honour DRY_RUN — print, do not execute state changes. +run() { + if [ "$DRY_RUN" = "1" ]; then + log "DRY-RUN would exec: $*" + return 0 + fi + log "exec: $*" + "$@" 2>&1 | tee -a "$LOG_LOCAL" +} + +ssh_run() { + local target="root@${CP1_HOST}" + if [ "$DRY_RUN" = "1" ]; then + log "DRY-RUN would ssh $target: $*" + return 0 + fi + log "ssh $target: $*" + ssh $SSH_OPTS "$target" "$@" 2>&1 | tee -a "$LOG_LOCAL" +} + +# ssh_run_stdin: pipe a heredoc through bash -s on cp-1; used for multi-line remote +# blocks (swap add, apt). +ssh_run_stdin() { + local target="root@${CP1_HOST}" + if [ "$DRY_RUN" = "1" ]; then + log "DRY-RUN would ssh $target with stdin script:" + sed 's/^/ | /' | tee -a "$LOG_LOCAL" + return 0 + fi + log "ssh $target (heredoc)" + ssh $SSH_OPTS "$target" "bash -s" 2>&1 | tee -a "$LOG_LOCAL" +} + +confirm() { + local prompt="$1" + if [ "$ASSUME_YES" = "1" ]; then + log "confirm SKIPPED (ASSUME_YES=1): $prompt" + return 0 + fi + echo -n " >>> $prompt Continue? [y/N] " + read -r a + case "$a" in y|Y|yes|YES) return 0 ;; *) die "aborted by operator" ;; esac +} + +# --------------------------------------------------------------------------- # +# Phase A — Add swap on cp-1 (idempotent) +# --------------------------------------------------------------------------- # +phase_add_swap() { + log "=== Phase A: add swap on $NODE ($SWAP_SIZE_MB MiB at $SWAP_PATH) ===" + [ "$SWAP_SIZE_MB" -ge 2048 ] || die "SWAP_SIZE_MB=$SWAP_SIZE_MB below 2048 MiB guardrail" + + cat </dev/null | grep -qx "\$SWAP_PATH"; then + echo "swap already on at \$SWAP_PATH — skipping" + free -h + exit 0 +fi + +# Root filesystem free space check — abort if less than 2*swap free. +avail_mb=\$(df -m --output=avail / | tail -1 | tr -d ' ') +need_mb=\$(( SIZE_MB * 2 )) +if [ "\$avail_mb" -lt "\$need_mb" ]; then + echo "ERROR: only \${avail_mb} MiB free on /, need \${need_mb} MiB (2x swap for safety)" + exit 1 +fi + +# Create swapfile. fallocate is fast; dd is the fallback. +if ! fallocate -l "\${SIZE_MB}M" "\$SWAP_PATH" 2>/dev/null; then + dd if=/dev/zero of="\$SWAP_PATH" bs=1M count="\$SIZE_MB" status=progress +fi +chmod 600 "\$SWAP_PATH" +mkswap "\$SWAP_PATH" +swapon "\$SWAP_PATH" + +# Persist via fstab (dedup). +if ! grep -q "^\$SWAP_PATH " /etc/fstab; then + echo "\$SWAP_PATH none swap sw 0 0" >> /etc/fstab +fi + +# Moderate swappiness — swap as safety net, not aggressive paging. +sysctl -w vm.swappiness=10 +if [ ! -f /etc/sysctl.d/99-k3s-swap.conf ] || ! grep -q '^vm.swappiness' /etc/sysctl.d/99-k3s-swap.conf; then + echo 'vm.swappiness=10' > /etc/sysctl.d/99-k3s-swap.conf +fi + +echo "--- swap after ---" +free -h +swapon --show +sysctl vm.swappiness +REMOTE + + # Verify from operator view. + if [ "$DRY_RUN" != "1" ]; then + log "verifying kubelet still Ready after swap add" + local ready + ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}') + [ "$ready" = "True" ] || die "kubelet on $NODE not Ready after swap add — halt" + log "kubelet Ready=True — Phase A complete" + fi +} + +# --------------------------------------------------------------------------- # +# Guardrail probes +# --------------------------------------------------------------------------- # +kubectl_latency_s() { + # returns integer seconds elapsed for `kubectl get nodes >/dev/null`. + local start end + start=$(date +%s) + kubectl get nodes >/dev/null 2>&1 || echo "kubectl-error" >&2 + end=$(date +%s) + echo $(( end - start )) +} + +cp1_mem_available_mib() { + # returns integer MiB `MemAvailable` on cp-1. + ssh $SSH_OPTS "root@$CP1_HOST" "awk '/^MemAvailable:/{printf \"%d\n\", \$2/1024}' /proc/meminfo" 2>/dev/null || echo 0 +} + +guard_kine_healthy() { + local s + s=$(kubectl_latency_s) + if [ "$s" -gt "$MAX_KUBECTL_SECONDS" ]; then + die "kubectl get nodes took ${s}s (>${MAX_KUBECTL_SECONDS}s) — kine cascade risk, HALT" + fi + log " kine ok: kubectl get nodes took ${s}s" +} + +guard_cp1_memory() { + local m + m=$(cp1_mem_available_mib) + if [ "$m" -lt "$MIN_CP1_MEM_MIB" ]; then + die "cp-1 MemAvailable=${m} MiB below ${MIN_CP1_MEM_MIB} MiB floor — HALT" + fi + log " cp-1 mem ok: MemAvailable=${m} MiB" +} + +# --------------------------------------------------------------------------- # +# Phase B — Preflight +# --------------------------------------------------------------------------- # +phase_preflight() { + log "=== Phase B: preflight ===" + + log "-- cluster health" + if [ "$DRY_RUN" != "1" ]; then + if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then + die "cluster is not healthy — refuse to start cp-1 update" + fi + else + log "DRY-RUN would run: $HEALTH_SCRIPT" + fi + + log "-- swap on cp-1" + if [ "$DRY_RUN" != "1" ]; then + local swap_total + swap_total=$(ssh $SSH_OPTS "root@$CP1_HOST" "awk '/^SwapTotal:/{print \$2}' /proc/meminfo") + [ "${swap_total:-0}" -ge $((2 * 1024 * 1024)) ] \ + || die "cp-1 SwapTotal=${swap_total} KiB below 2 GiB — run --add-swap first" + log " cp-1 SwapTotal=$(( swap_total / 1024 )) MiB" + fi + + log "-- fsn1 workers Ready and have >=${MIN_WORKER_MEM_MIB} MiB MemAvailable" + # Worker private IPs (worker-4 nbg1 excluded — cannot host fsn1 RWO PVs). + local ips=("10.42.1.2" "10.42.1.3" "10.42.1.5" "10.42.1.7") + local names=("k3s-worker-1" "k3s-worker-2" "k3s-worker-3" "k3s-worker-5") + if [ "$DRY_RUN" != "1" ]; then + for i in "${!ips[@]}"; do + local nm="${names[$i]}" + local ip="${ips[$i]}" + local ready + ready=$(kubectl get node "$nm" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown) + [ "$ready" = "True" ] || die "worker $nm not Ready (Ready=$ready)" + local mem + mem=$(ssh $SSH_OPTS -J "root@$CP1_HOST" "root@$ip" \ + "awk '/^MemAvailable:/{printf \"%d\n\", \$2/1024}' /proc/meminfo" 2>/dev/null || echo 0) + [ "$mem" -ge "$MIN_WORKER_MEM_MIB" ] \ + || die "worker $nm MemAvailable=${mem} MiB below ${MIN_WORKER_MEM_MIB} MiB — HALT" + log " $nm ok: MemAvailable=${mem} MiB" + done + fi + + log "-- kine latency probe" + if [ "$DRY_RUN" != "1" ]; then guard_kine_healthy; fi + + log "-- k3s SQLite datastore snapshot" + # k3s etcd-snapshot in embedded-SQLite mode copies the sqlite file to + # /var/lib/rancher/k3s/server/db/snapshots/. + ssh_run "k3s etcd-snapshot save --name pre-cp1-os-update-${TS}" + ssh_run "ls -la /var/lib/rancher/k3s/server/db/snapshots/ | tail -10" + + log "-- persist eviction plan" + if [ "$DRY_RUN" != "1" ]; then + local plan="/tmp/cp1-stateful-plan-${TS}.txt" + ssh $SSH_OPTS "root@$CP1_HOST" "cat > $plan" < active, small -> heavy. Do NOT reorder without design review. +$(for e in "${STATEFUL_ORDER[@]}"; do echo "$e"; done) +EOF + log " wrote $plan on cp-1" + else + log "DRY-RUN would write /tmp/cp1-stateful-plan-${TS}.txt with STATEFUL_ORDER" + fi + + log "=== Phase B: preflight OK ===" +} + +# --------------------------------------------------------------------------- # +# Phase C — Cordon + move stateful pods off cp-1, one at a time +# --------------------------------------------------------------------------- # +phase_drain_stateful() { + log "=== Phase C: cordon + batched stateful move ===" + + log "-- cordon $NODE (prevents rescheduled pods from landing back on cp-1)" + run kubectl cordon "$NODE" + + local i=0 + for entry in "${STATEFUL_ORDER[@]}"; do + i=$((i + 1)) + local ns="${entry%%:*}" + local pod="${entry##*:}" + + log "--- [${i}/${#STATEFUL_ORDER[@]}] moving $ns/$pod" + + # Confirm the pod actually IS on cp-1 before touching it. + if [ "$DRY_RUN" != "1" ]; then + local on + on=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null || echo "") + if [ -z "$on" ]; then + log " $ns/$pod not found — sts may have changed; skipping" + continue + fi + if [ "$on" != "$NODE" ]; then + log " $ns/$pod already on $on (not cp-1) — skipping" + continue + fi + fi + + # Delete the pod. StatefulSet controller will re-create it; scheduler will + # pick a fsn1 worker because cp-1 is cordoned and worker-4 is nbg1. + run kubectl -n "$ns" delete pod "$pod" --wait=false + + if [ "$DRY_RUN" = "1" ]; then + log " DRY-RUN skipping wait-for-ready" + continue + fi + + log " waiting for $ns/$pod to be Ready on a non-cp-1 node (max 300s)" + local deadline=$(( $(date +%s) + 300 )) + local new_node="" ready="" + while [ $(date +%s) -lt $deadline ]; do + new_node=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null || echo "") + ready=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "") + if [ -n "$new_node" ] && [ "$new_node" != "$NODE" ] && [ "$ready" = "True" ]; then + log " $ns/$pod -> $new_node OK" + break + fi + sleep 5 + done + [ "$ready" = "True" ] && [ "$new_node" != "$NODE" ] \ + || die "$ns/$pod did not reach Ready on a non-cp-1 node in 300s — HALT (leave cp-1 cordoned)" + + log " guardrails after $ns/$pod" + guard_kine_healthy + guard_cp1_memory + + if [ "$i" -lt "${#STATEFUL_ORDER[@]}" ]; then + log " settle pause ${STATEFUL_SETTLE_SECONDS}s" + sleep "$STATEFUL_SETTLE_SECONDS" + fi + done + + log "=== Phase C: all stateful pods evicted from cp-1 ===" +} + +# --------------------------------------------------------------------------- # +# Phase D — Drain remaining pods + apt on cp-1 +# --------------------------------------------------------------------------- # +phase_apt() { + log "=== Phase D: drain (remaining deployments) + apt on $NODE ===" + + # Guardrail: refuse to run if any StatefulSet pod is still on cp-1. + if [ "$DRY_RUN" != "1" ]; then + local sts_on_cp1 + sts_on_cp1=$(kubectl get pods -A -o json --field-selector spec.nodeName=$NODE \ + | jq -r '.items[] | select(.metadata.ownerReferences[0].kind=="StatefulSet") | "\(.metadata.namespace)/\(.metadata.name)"' \ + | wc -l) + [ "$sts_on_cp1" -eq 0 ] \ + || die "$sts_on_cp1 StatefulSet pod(s) still on cp-1 — Phase C incomplete; refuse Phase D" + fi + + log "-- drain (timeout ${DRAIN_TIMEOUT_SECONDS}s)" + set +e + if [ "$DRY_RUN" = "1" ]; then + log "DRY-RUN would run: kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --timeout=${DRAIN_TIMEOUT_SECONDS}s" + local rc=0 + else + kubectl drain "$NODE" \ + --ignore-daemonsets \ + --delete-emptydir-data \ + --timeout="${DRAIN_TIMEOUT_SECONDS}s" 2>&1 | tee -a "$LOG_LOCAL" + local rc=${PIPESTATUS[0]} + fi + set -e + + if [ "$rc" -ne 0 ]; then + log "drain FAILED (rc=$rc). Never force. Uncordoning." + run kubectl uncordon "$NODE" + die "drain failed on $NODE — investigate PDB / orphan pods; do NOT proceed" + fi + + log "-- apt on cp-1" + cat <<'REMOTE' | ssh_run_stdin +set -euo pipefail +export DEBIAN_FRONTEND=noninteractive +APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold' + +# Rollback references (Phase-A didn't need this, but Phase D does). +uname -r > /root/pre-apt-kernel +dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv +echo "pre-apt kernel: $(cat /root/pre-apt-kernel)" + +# Recover from any half-finished dpkg state before touching apt. +if dpkg --audit | grep -qE .; then + echo "dpkg audit reported issues, running dpkg --configure -a" + dpkg --configure -a || true +fi + +apt-get update + +if ! apt-get $APT_OPTS upgrade; then + echo "upgrade failed, attempting apt-get -f install" + apt-get $APT_OPTS -f install + apt-get $APT_OPTS upgrade +fi + +apt-get $APT_OPTS dist-upgrade +apt-get $APT_OPTS autoremove --purge +apt-get clean + +if [ -f /var/run/reboot-required ]; then + echo "REBOOT_REQUIRED=yes" + echo "REBOOT_REASON</dev/null || echo "(no package list)" + echo "EOF" +else + echo "REBOOT_REQUIRED=no" +fi +REMOTE + + log "=== Phase D: apt complete (check REBOOT_REQUIRED in the log) ===" +} + +# --------------------------------------------------------------------------- # +# Phase E — Reboot cp-1 and wait for api-server /livez +# --------------------------------------------------------------------------- # +phase_reboot() { + log "=== Phase E: reboot $NODE ===" + + # Tell cp-1 to reboot. The ssh command will hang up mid-command — that's fine. + if [ "$DRY_RUN" != "1" ]; then + log "issuing 'systemctl reboot' on $NODE (ssh will drop; expected)" + ssh $SSH_OPTS "root@$CP1_HOST" 'systemctl reboot' 2>&1 | tee -a "$LOG_LOCAL" || true + log "waiting 15s for ssh to fully drop before polling api-server" + sleep 15 + else + log "DRY-RUN would ssh root@$CP1_HOST 'systemctl reboot'" + fi + + log "-- poll api-server /livez (timeout ${REBOOT_MAX_WAIT_SECONDS}s)" + if [ "$DRY_RUN" != "1" ]; then + local deadline=$(( $(date +%s) + REBOOT_MAX_WAIT_SECONDS )) + local code=000 + while [ $(date +%s) -lt $deadline ]; do + code=$(curl -sk -o /dev/null -w '%{http_code}' "https://$CP1_HOST:6443/livez" 2>/dev/null || echo 000) + if [ "$code" = "200" ]; then + log " api-server /livez=200 (elapsed $(( REBOOT_MAX_WAIT_SECONDS - (deadline - $(date +%s)) ))s)" + break + fi + sleep 5 + done + [ "$code" = "200" ] || die "api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate; check 'hcloud server describe k3s-cp-1' and Hetzner console" + fi + + log "-- wait for kubelet Ready on $NODE (max 300s)" + if [ "$DRY_RUN" != "1" ]; then + local deadline=$(( $(date +%s) + 300 )) + local ready=Unknown + while [ $(date +%s) -lt $deadline ]; do + ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown) + [ "$ready" = "True" ] && break + sleep 5 + done + [ "$ready" = "True" ] || die "kubelet on $NODE never returned Ready — escalate (do NOT change k3s config)" + log " kubelet Ready=True" + fi + + log "=== Phase E: cp-1 is back ===" +} + +# --------------------------------------------------------------------------- # +# Phase F — Uncordon + verify + finalize +# --------------------------------------------------------------------------- # +phase_finalize() { + log "=== Phase F: uncordon + verify ===" + + run kubectl uncordon "$NODE" + + log "-- settle wait ${POST_UNCORDON_WAIT_SECONDS}s" + [ "$DRY_RUN" = "1" ] || sleep "$POST_UNCORDON_WAIT_SECONDS" + + log "-- cluster health" + if [ "$DRY_RUN" != "1" ]; then + if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then + die "cluster health failed after cp-1 update — escalate, do NOT touch k3s" + fi + fi + + log "-- apt history summary (audit)" + ssh_run 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60' + + log "-- old snapshots (>30d) — listing only, review manually" + ssh_run 'find /var/lib/rancher/k3s/server/db/snapshots/ -type f -mtime +30 -name "pre-*" -print 2>/dev/null || true' + + log "=== cp-1 OS update complete — attach $LOG_LOCAL to the execution ticket ===" +} + +# --------------------------------------------------------------------------- # +# --run — orchestrate all phases with confirmations +# --------------------------------------------------------------------------- # +phase_run_all() { + log "=== full cp-1 update run (log: $LOG_LOCAL) ===" + + confirm "Phase A (add swap) — proceed?" + phase_add_swap + + confirm "Phase B (preflight) — proceed?" + phase_preflight + + confirm "Phase C (cordon + batched stateful move) — proceed?" + phase_drain_stateful + + confirm "Phase D (drain remaining + apt) — proceed?" + phase_apt + + confirm "Phase E (reboot cp-1; api-server unavailable ~90-180s) — proceed?" + phase_reboot + + confirm "Phase F (uncordon + verify) — proceed?" + phase_finalize + + log "=== FULL RUN COMPLETE ===" +} + +# --------------------------------------------------------------------------- # +# Arg parse +# --------------------------------------------------------------------------- # +usage() { grep -E '^# ' "$0" | sed 's/^# \{0,1\}//'; exit 2; } + +[ $# -ge 1 ] || usage + +# Init the local log file up front so tee always has a target. +: > "$LOG_LOCAL" +log "update-cp-1.sh started (DRY_RUN=$DRY_RUN)" +log "log file: $LOG_LOCAL" + +case "$1" in + --dry-run) + DRY_RUN=1 + export DRY_RUN + log "DRY_RUN=1 — walking Phases A..F without touching state" + phase_add_swap + phase_preflight + phase_drain_stateful + phase_apt + phase_reboot + phase_finalize + ;; + --add-swap) phase_add_swap ;; + --preflight) phase_preflight ;; + --drain-stateful) phase_drain_stateful ;; + --apt) phase_apt ;; + --reboot) phase_reboot ;; + --finalize) phase_finalize ;; + --run) phase_run_all ;; + -h|--help) usage ;; + *) echo "unknown arg: $1" >&2; usage ;; +esac