basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515)

Browse source 
Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an
HA-aware procedure that covers all three CPs. Rationale: after DEV-510
the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs
are NoSchedule-tainted, so:

- Kine cascade is no longer the driving risk; embedded 3-node etcd
  tolerates one member down. The kubectl-latency guardrail stays as a
  soft signal.
- CPs host no StatefulSets or single-replica Deployments. The old
  Phase C batched stateful eviction dance is obsolete.
- CP reboots no longer lose the api-server; two peers keep serving.
  /livez is polled from the operator machine and cross-checked with
  etcdctl endpoint status from a peer CP.

New CP ordering rule: one CP at a time, leader last, health gate
between CPs. Refuses to start if another CP is already cordoned.

Automation:
- New scripts/os-update/update-cp-node.sh <node> is the entry point.
  Supports --add-swap / --preflight / --drain / --apt / --reboot /
  --finalize / --run / --dry-run. Phase A also installs etcd-client
  on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the
  target. Peer picker prefers a CP that has etcdctl; refuses cp-1
  target with a targeted error if no other CP has etcdctl yet.
- scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards
  to update-cp-node.sh k3s-cp-1 "$@" for backwards compat.
- CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc.
- OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and
  automation entry point list all point at the new HA-aware artifacts.

Verified: bash -n clean on both scripts; --dry-run walks all six
phases for each of cp-1/cp-2/cp-3; --preflight against live cluster
correctly aborts cp-1 with "install etcdctl on another CP first" and
cp-2/cp-3 with "run --add-swap first" (their current state).

Execution is board-approval-gated; DEV-497 (cp-1-only executor) is
superseded by a successor ticket.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-08-23 01:53:14 +00:00
parent 34350a03bf
commit e848d225cd
5 changed files with 925 additions and 960 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

384
infrastructure/CP1_UPDATE_PROCEDURE.md
View file

@ -1,382 +1,14 @@
# k3s-cp-1 OS Update Procedure
# k3s-cp-1 OS Update Procedure — SUPERSEDED
**Purpose:** apply the weekly Ubuntu OS updates to `k3s-cp-1`, the **sole** control-plane node of the k3s cluster, without triggering the kine/SQLite thundering-herd cascade documented in [DEV-495](/DEV/issues/DEV-495) and without losing api-server access longer than a normal reboot.
**Superseded by:** [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) (DEV-515, 2026-08-23).
**Audience:** the CTO agent, or an operator with root SSH to the cluster. Execution is board-approval-gated — see `## Approval gates` below.
**Why:** this document was written when cp-1 was the **sole** control-plane node and the datastore was embedded SQLite via kine. After [DEV-510](/DEV/issues/DEV-510) the cluster runs HA embedded etcd across `k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`, and all three CPs are `NoSchedule`-tainted (they host no StatefulSets or single-replica Deployments anymore). The single-CP kine cascade risk, the batched stateful eviction dance, and the external-`/livez` polling pattern no longer describe the current cluster.
**Why cp-1 is not covered by the standard [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md):**
The generalized procedure covers cp-1 and cp-2 and cp-3. Use it.
- cp-1 has 3.7 GiB RAM and **zero swap** — during the 2026-08-16 worker-3 drain, `MemAvailable` dropped to 43 MiB while kine backed up on writes. cp-1's own drain evicts **more** state at once than any worker drain.
- cp-1 currently hosts 5 StatefulSets (`harbor-database`, `harbor-redis`, `nextcloud-redis-replicas`, `stalwart`, `stalwart-postgres`) plus 8 single-replica Deployments — all backed by Hetzner-CSI RWO volumes pinned to the fsn1 datacenter. A one-shot `kubectl drain` reschedules >13 pods simultaneously → guaranteed kine cascade.
- Rebooting cp-1 removes the entire kube-apiserver for the reboot window (~90–180 s expected). There is no fallback control-plane.
**Scripts:**
The four kine thundering-herd guardrails from [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md) still apply here. This procedure implements them for cp-1.
- `infrastructure/scripts/os-update/update-cp-node.sh <node>` — the HA-aware entry point.
- `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compatibility with any existing runbook that names it explicitly.
---
## Scope
**In scope**
- Add a durable ≥ 2 GiB swapfile on cp-1 (idempotent one-off — `--add-swap`).
- Pre-drain redistribution of stateful pods off cp-1, one at a time.
- k3s SQLite datastore snapshot as the restore point.
- `apt-get update/upgrade/dist-upgrade/autoremove` on cp-1.
- Controlled reboot with external liveness monitoring.
- Uncordon + cluster health verify.
**Out of scope — do NOT do here**
- Any change to `/etc/rancher/k3s/*`, `/etc/systemd/system/k3s*.service*`, or the k3s binary version. k3s upgrades go through system-upgrade-controller (see `K3S_OPERATIONS.md`).
- Any change to manifests under `apps/`, `infrastructure/`, or applied via ArgoCD.
- Deleting PVs / PVCs. StatefulSets that get rescheduled off cp-1 stay on their new node — do NOT try to move them back.
- Fixing application-level problems ("harbor-core is CrashLoopBackOff after apt") — that's an app problem, escalate.
- Rebuilding the node. If cp-1 does not return after reboot, escalate; the rebuild path is `ADD_WORKER_NODE.md` plus board approval, not this document.
---
## Approval gates
This document has **two** independent gates. Neither happens without explicit board approval on the corresponding Paperclip issue:
1. **Add swap (Phase A).** Non-invasive, non-state-mutating, kubelet already runs with `failSwapOn=false`. Requires board approval because it modifies the CP node.
2. **Full OS update (Phases B–F).** Requires board approval because it drains cp-1 and reboots the sole api-server. Do NOT execute without an explicit `request_board_approval` acceptance on the execution ticket.
Both gates are independent — swap can (and should) be added first, in a quiet window, before the full update is scheduled.
---
## Current cp-1 workload (snapshot 2026-08-16)
Re-derive live before any execution — this table is a design reference only.
| Kind | Namespace/Name | Notes |
|------|-----------------|-------|
| StatefulSet | `harbor/harbor-database-0` | Harbor Postgres, RWO 10 GiB fsn1 |
| StatefulSet | `harbor/harbor-redis-0` | Harbor cache, RWO 10 GiB fsn1 |
| StatefulSet | `nextcloud/nextcloud-redis-replicas-0` | Nextcloud Redis replica, RWO 10 GiB fsn1 |
| StatefulSet | `stalwart/stalwart-0` | Mail server, RWO 20 GiB fsn1 — **critical, move last** |
| StatefulSet | `stalwart/stalwart-postgres-0` | Mail metadata DB, RWO 10 GiB fsn1 |
| Deployment | `harbor/harbor-core` | Registry frontend, single replica |
| Deployment | `harbor/harbor-jobservice` | Registry job worker, single replica |
| Deployment | `monitoring/alertmanager` | Alertmanager, single replica |
| Deployment | `monitoring/loki` | Loki (single-binary), single replica |
| Deployment | `monitoring/prometheus` | Prometheus, single replica |
| Deployment | `observability/blackbox-exporter-*` | Blackbox exporter |
| Deployment | `opencloud/tika` | Apache Tika, single replica |
| Deployment | `passbolt/passbolt` | Passbolt web, single replica |
| DaemonSet | `kube-system/hcloud-csi-node-*` | Ignored by drain |
| DaemonSet | `kube-system/svclb-*` | Ignored by drain |
| DaemonSet | `observability/*-node-exporter-*` | Ignored by drain |
| DaemonSet | `observability/loki-stack-promtail-*` | Ignored by drain |
**Absorption capacity (fsn1 workers only — nbg1 worker-4 cannot receive fsn1 RWO volumes):**
- `k3s-worker-5`: ~3.0 GiB free, 6 pods scheduled — primary target for the two heaviest stateful pods.
- `k3s-worker-2`: ~2.0 GiB free, 14 pods.
- `k3s-worker-3`: ~2.3 GiB free, 15 pods, 2 sts.
- `k3s-worker-1`: ~1.9 GiB free, 23 pods, 6 sts — **avoid piling more onto this one**.
---
## Phase A — Add swap (one-off, idempotent)
**Goal:** eliminate the "3.7 GiB, no swap" underlying constraint before we ever try to drain cp-1.
**Preconditions:**
- kubelet in this k3s already runs with `failSwapOn=false` (confirmed via `/api/v1/nodes/k3s-cp-1/proxy/configz`) — enabling swap does NOT break the kubelet.
- cp-1 `/` has ≥ 10 GiB free (currently 21 GiB free of 75 GiB).
- Board approval on the swap-add ticket.
**Sizing:** default **4 GiB** swap. Rationale — cp-1 baseline (k3s + hosted apps) already sits at ~3.1 GiB used; 4 GiB swap gives us headroom for the drain-eviction transient without inflating disk usage past ~10 % of `/`. Minimum acceptable per guardrail: 2 GiB.
**Location:** `/swapfile` (root filesystem). Not a separate partition — reversible, no LVM changes, no ext4/xfs migration.
**Steps (encoded in `update-cp-1.sh --add-swap`):**
```bash
# On cp-1 as root:
SWAPFILE=/swapfile
SIZE_MB=4096
# Idempotency: skip if a swapfile of the target size already exists and is on.
if swapon --show=NAME | grep -qx "$SWAPFILE"; then
echo "swap already on at $SWAPFILE"
exit 0
fi
# Create the file with fallocate; fall back to dd for filesystems without fallocate support.
fallocate -l "${SIZE_MB}M" "$SWAPFILE" || dd if=/dev/zero of="$SWAPFILE" bs=1M count="$SIZE_MB" status=progress
chmod 600 "$SWAPFILE"
mkswap "$SWAPFILE"
swapon "$SWAPFILE"
# Persist across reboot. Guard against duplicate fstab entries.
grep -q "^$SWAPFILE " /etc/fstab || echo "$SWAPFILE none swap sw 0 0" >> /etc/fstab
# Moderate swappiness — we want swap as a safety net, not aggressive paging.
sysctl -w vm.swappiness=10
grep -q '^vm.swappiness' /etc/sysctl.d/99-k3s-swap.conf 2>/dev/null || {
echo 'vm.swappiness=10' > /etc/sysctl.d/99-k3s-swap.conf
}
# Verify.
free -h
swapon --show
```
**Rollback for Phase A:** `swapoff /swapfile && rm /swapfile` and remove the fstab line. This is safe at any time — swap is a soft resource.
**Verification after Phase A:**
- `free -h` shows `Swap: 4.0Gi` used ≈ 0.
- `swapon --show` shows `/swapfile 4G`.
- `sysctl vm.swappiness` returns `10`.
- kubelet still Ready (`kubectl get node k3s-cp-1`).
- No new `MemoryPressure` condition.
---
## Phase B — Preflight for the full OS update
Everything from here on runs from the CTO's operator machine (not from cp-1 itself, because we lose kubectl during the reboot). SSH to cp-1 is fine — kubectl calls issued from cp-1 during the pre-drain phase are fine and get logged into `/tmp/os-update-cp-1-<ts>.log`.
**Log directory contract** (same as `update-node.sh`): every command's stdout+stderr goes to `/tmp/os-update-cp-1-<UTC-timestamp>.log` on cp-1. Attach that log to the Paperclip execution ticket at the end.
Run `update-cp-1.sh --preflight`:
1. Cluster is currently healthy: `cluster-health.sh` returns 0.
2. All 5 fsn1 workers are `Ready` (worker-1, worker-2, worker-3, worker-5, cp-1). worker-4 is nbg1 and irrelevant here.
3. Each fsn1 worker has ≥ 500 MiB `MemAvailable`.
4. `time kubectl get nodes` returns in ≤ 2 s (kine is not already stressed).
5. cp-1 has swap on — abort if `free -h` shows `Swap: 0B`.
6. **k3s datastore snapshot.** cp-1 runs k3s in embedded-SQLite mode (`--datastore-endpoint` is unset). `k3s etcd-snapshot save --name pre-cp1-os-update-<UTC-date>` produces a copy of the SQLite file under `/var/lib/rancher/k3s/server/db/snapshots/`. This is our restore point. Snapshots older than 30 days are pruned in Phase F.
If any preflight check fails: STOP. Do not proceed. Do not add pods to cp-1 to "rebalance later" — that's a separate task.
---
## Phase C — Move stateful pods off cp-1 (batched)
**Rule** ([[k3s-drain-kine-thundering-herd]] guardrail #1): no drain that reschedules > 3 StatefulSets at once. So we DO NOT `kubectl drain k3s-cp-1` while stateful pods still live on it. We move them one at a time first.
**Cordon cp-1 immediately.** Cordon only prevents *new* scheduling — existing pods stay put. Cordoning first ensures that when we delete a stateful pod, the StatefulSet controller cannot re-create it back on cp-1.
```bash
kubectl cordon k3s-cp-1
```
**Order of eviction (idle → active, tiny → heavy):**
| # | Pod | Why in this position |
|---|-----|----------------------|
| 1 | `harbor/harbor-redis-0` | Idle cache, cold restart is instant, low write pressure |
| 2 | `nextcloud/nextcloud-redis-replicas-0` | Replica-1 of a Redis replicaset — non-primary, safe to bounce |
| 3 | `harbor/harbor-database-0` | Registry Postgres, mostly idle (registry pulls, not writes) |
| 4 | `stalwart/stalwart-postgres-0` | Mail metadata DB — active but recoverable; move before the mail server itself |
| 5 | `stalwart/stalwart-0` | Mail server, most important. Save for last so mail keeps flowing until the very end |
For each pod:
```bash
# 1. Delete the pod — StatefulSet controller will re-create it on a schedulable fsn1 worker.
kubectl -n "$NS" delete pod "$POD" --wait=false
# 2. Wait for the *new* pod (same name — StatefulSets keep identities) to be scheduled
# somewhere OTHER than cp-1 and reach Ready=True.
deadline=$(( $(date +%s) + 300 ))
while [ $(date +%s) -lt $deadline ]; do
new_node=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.spec.nodeName}' 2>/dev/null || true)
ready=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)
if [ -n "$new_node" ] && [ "$new_node" != "k3s-cp-1" ] && [ "$ready" = "True" ]; then
echo " $POD -> $new_node OK"
break
fi
sleep 5
done
[ "$ready" = "True" ] || die "pod $NS/$POD did not reach Ready on a non-cp-1 node in 300s — HALT"
# 3. Kine health probe — abort the batch if the api server is getting slow.
t=$( { time kubectl get nodes >/dev/null; } 2>&1 | awk '/real/{print $2}' )
# t is like "0m1.234s" — if the m field is >0 or the s field is >5, halt.
# See update-cp-1.sh for the exact parse; > 5 s -> halt the whole cycle.
# 4. Free-memory probe on cp-1.
avail=$(ssh root@cp-1 "free -m | awk '/Mem:/{print \$7}'")
[ "$avail" -lt 200 ] && die "cp-1 MemAvailable dropped below 200 MiB — HALT"
# 5. Settle pause before the next eviction — kine needs to catch up on the write
# burst from the just-attached PV and the just-scheduled pod.
sleep 60
```
**Halt conditions during Phase C** (any one → STOP, do NOT proceed to Phase D):
- `kubectl get nodes` takes > 5 s.
- cp-1 `MemAvailable` drops below 200 MiB.
- Any target worker enters `MemoryPressure=True` or `DiskPressure=True`.
- Any stateful pod does not reach Ready on a new node within 5 minutes (may indicate PV-attach or PDB issue).
If we halt: leave cp-1 **cordoned** but do not reboot. The pods that have already moved stay where they are; the pods that haven't will still be on cp-1. Escalate on the execution ticket with `blocked` and name the halt condition.
---
## Phase D — Drain remaining pods + apt
Once all 5 stateful pods are off cp-1 and cluster health is green:
```bash
# Drain everything else — only Deployment pods left (single-replica each,
# no PV attach, so reschedule is fast).
kubectl drain k3s-cp-1 \
--ignore-daemonsets \
--delete-emptydir-data \
--timeout="${DRAIN_TIMEOUT_SECONDS:-600}s"
```
If drain reports a PDB block: do NOT `--force`. Uncordon cp-1, mark the run as `blocked` on the PDB, and escalate. This is a design bug in the workload's PDB — fix it separately.
**apt on cp-1** (mirrors `update-node.sh` step 3; `update-cp-1.sh --apt` runs this via `ssh root@cp-1 bash -s`):
```bash
export DEBIAN_FRONTEND=noninteractive
APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
uname -r > /root/pre-apt-kernel # for rollback reference
dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv
if dpkg --audit | grep -qE .; then dpkg --configure -a || true; fi
apt-get update
apt-get $APT_OPTS upgrade || { apt-get $APT_OPTS -f install; apt-get $APT_OPTS upgrade; }
apt-get $APT_OPTS dist-upgrade
apt-get $APT_OPTS autoremove --purge
apt-get clean
[ -f /var/run/reboot-required ] && echo REBOOT_REQUIRED=yes || echo REBOOT_REQUIRED=no
```
`pre-apt-kernel` and `pre-apt-packages.tsv` are the local rollback references — see Rollback below.
---
## Phase E — Reboot handling (api-server unavailability)
**Duration expectation:** 90–180 s of api-server unavailability. The operator machine will get `Unable to connect to the server` from kubectl during this window — that is expected, not an alarm.
**Escalation trigger:** api-server not back on `/livez` after **10 minutes** → escalate. First check Hetzner console via `hcloud server describe k3s-cp-1` for boot state; if kernel-panic / initramfs, use grub previous-kernel path (see Rollback). Do NOT rebuild the node.
**Steps (executed by `update-cp-1.sh --reboot`):**
```bash
# 1. Tell cp-1 to reboot. This SSH will hang up mid-command — that's fine.
ssh $SSH_OPTS root@cp-1 'systemctl reboot' || true
# 2. Deliberate 15 s pause — SSH needs to actually drop, don't race the poll.
sleep 15
# 3. Poll the api-server livez from the OPERATOR machine (not from cp-1).
# Using --insecure (`-k`) because the server cert is self-signed by k3s.
deadline=$(( $(date +%s) + ${REBOOT_MAX_WAIT_SECONDS:-600} ))
while [ $(date +%s) -lt $deadline ]; do
code=$(curl -sk -o /dev/null -w '%{http_code}' https://178.105.17.239:6443/livez 2>/dev/null || echo 000)
if [ "$code" = "200" ]; then
echo " api-server /livez=200"
break
fi
sleep 5
done
[ "$code" = "200" ] || die "api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate; check hcloud console"
# 4. Wait for kubelet Ready on cp-1 from the api-server view.
deadline=$(( $(date +%s) + 300 ))
while [ $(date +%s) -lt $deadline ]; do
ready=$(kubectl get node k3s-cp-1 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
[ "$ready" = "True" ] && break
sleep 5
done
[ "$ready" = "True" ] || die "kubelet on cp-1 never returned Ready — escalate (do NOT change k3s config)"
```
**What to do if the api-server takes 3–10 minutes:** it may be normal on this box (large SQLite → replay). Wait it out. Do NOT restart k3s to "help" — kine SQLite replay is stateful and interrupting it can corrupt the DB. See [[k3s-drain-kine-thundering-herd]] recovery: doing nothing is a valid path.
**What to do if it takes > 10 minutes:** escalate; run the Hetzner-console diagnosis; if the console shows a bootable Ubuntu but the k3s service is failing, that's the boundary — this procedure stops here. Follow `K3S_OPERATIONS.md` for the k3s recovery path.
---
## Phase F — Uncordon + verify + finalize
```bash
# Uncordon cp-1. StatefulSet pods will NOT be moved back — that's correct
# behaviour, they were moved for a reason. Rebalancing is a separate task.
kubectl uncordon k3s-cp-1
# Settle wait — kine needs to process the flood of "node schedulable again" events.
sleep "${POST_UNCORDON_WAIT_SECONDS:-180}"
# Full cluster health.
RETRY_ON_TRANSIENT=1 infrastructure/scripts/os-update/cluster-health.sh
# Print the apt history for audit.
ssh root@cp-1 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
# Prune snapshots older than 30 days.
ssh root@cp-1 'find /var/lib/rancher/k3s/server/db/snapshots/ -type f -mtime +30 -name "pre-*" -print'
# (list only — actual deletion is a manual review after the run)
```
Attach the `/tmp/os-update-cp-1-<ts>.log` and the ordered ledger of moved pods to the execution ticket. Mark the ticket `done` on green health, or `blocked` naming the specific residual issue on red.
---
## Rollback / recovery
### If apt broke a package
- On cp-1, `dpkg --audit` to find half-configured packages.
- `dpkg --configure -a`, then `apt-get -f install`.
- If a specific package broke and you know the previous version from `/root/pre-apt-packages.tsv`, `apt-get install <pkg>=<old-version>`.
### If the new kernel does not boot
- Hetzner cloud console: send `hcloud server request-console k3s-cp-1` → get VNC URL, watch boot.
- If grub is up, select the previous-kernel entry. Ubuntu keeps ≥ 1 old kernel installed by default (we verified `6.8.0-137` current; the previous `6.8.0-124` was in use during the DEV-478 cycle).
- Once booted on the old kernel, `apt-get remove` the broken kernel and pin the working one:
```bash
apt-mark hold linux-image-<broken-version> linux-headers-<broken-version>
```
- Escalate on the ticket regardless — a kernel rollback is a follow-up investigation, not a "done" outcome.
### If the node does not return at all
- Do NOT `hcloud server delete`. Do NOT re-provision.
- Escalate to the board with the Hetzner console output and the `/tmp/os-update-cp-1-*.log`.
- The `--cluster-reset --cluster-reset-restore-path=<snapshot>` recovery path exists (see `K3S_OPERATIONS.md`) but requires board approval per stateful-service safety rules. The pre-flight snapshot from Phase B is the restore point.
### If Phase C halted mid-eviction
- cp-1 is cordoned, some stateful pods have moved, some haven't. Cluster is functional.
- Uncordon cp-1 (`kubectl uncordon k3s-cp-1`) — StatefulSets that stayed on cp-1 keep running, moved ones stay where they went.
- Do NOT proceed to Phase D. Open a follow-up ticket with the halt condition. Retry in the next maintenance window after fixing the halt condition.
---
## Automation entry points
- `infrastructure/scripts/os-update/update-cp-1.sh` — this whole flow, with subcommands:
- `--add-swap` — Phase A only, idempotent, safe standalone.
- `--dry-run` — walk Phases B/C/D/E/F printing exactly what would be done without touching anything. Safe to run any time; used for design review.
- `--preflight` — Phase B only.
- `--drain-stateful` — Phase C only (cordon + batched stateful moves).
- `--apt` — Phase D apt commands only (requires cp-1 already fully drained).
- `--reboot` — Phase E only (requires apt already done).
- `--finalize` — Phase F only (uncordon + verify).
- `--run` — do all phases in order, with a confirmation prompt between each unless `ASSUME_YES=1`.
The script follows the same log-dir contract as `update-node.sh` (`/tmp/os-update-cp-1-<UTC-timestamp>.log`).
---
## Change history
| Date | Change | By |
|------|--------|-----|
| 2026-08-16 | Initial cp-1 update design (DEV-496) | CTO agent |
**History note:** the original phase design (add swap → preflight → batched stateful eviction → drain → apt → reboot → finalize) is preserved in git history at commit `19af691` and `989033d` for future reference. The new procedure retains Phases A/B/D/E/F essentially unchanged and drops Phase C (batched stateful eviction) as obsolete.

298
infrastructure/CP_UPDATE_PROCEDURE.md Normal file
View file

@ -0,0 +1,298 @@
# HA k3s Control-Plane OS Update Procedure
**Purpose:** apply Ubuntu OS updates (kernel, security, apt) to any control-plane node in the HA k3s cluster (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) without losing etcd quorum and without a longer api-server outage than a normal reboot.
**Audience:** the CTO agent, or an operator with root SSH to the cluster. Execution is board-approval-gated — see `## Approval gates` below.
**Supersedes:** the earlier single-CP `CP1_UPDATE_PROCEDURE.md` (kept as a redirect stub). The cp-1-only procedure was written when cp-1 was the sole api-server and the datastore was embedded SQLite via kine. After [DEV-510](/DEV/issues/DEV-510) (2026-08-22), the cluster runs HA embedded etcd across cp-1/cp-2/cp-3 and all three CPs are `NoSchedule`-tainted, so:
- **The kine cascade is no longer the driving risk.** The datastore is etcd 3.6.12 with 3-node quorum; a single CP drain no longer leaves a lone SQLite writer. The kubectl-latency guardrail stays as a sanity check but is now a much softer signal.
- **CPs host no StatefulSet or single-replica Deployment workloads.** The pre-drain stateful eviction dance from the old cp-1 doc no longer applies — CPs only host DaemonSets + `metrics-server` (currently on cp-2) + occasional CronJob completions.
- **The api-server is not lost on reboot.** During a CP reboot the other two apiservers keep serving. `/livez` is polled from the *other* apiservers, not from an external machine forced to poll the one being rebooted.
---
## Scope
**In scope**
- Add a durable ≥ 2 GiB swapfile on any CP that has zero swap (idempotent one-off — `--add-swap`).
- k3s etcd snapshot as the restore point.
- `kubectl cordon` + `kubectl drain` on the target CP.
- `apt-get update/upgrade/dist-upgrade/autoremove` on the target CP.
- Controlled reboot with etcd-quorum-aware liveness monitoring.
- Uncordon + cluster health verify.
**Out of scope — do NOT do here**
- Any change to `/etc/rancher/k3s/*`, `/etc/systemd/system/k3s*.service*`, or the k3s binary version. k3s upgrades go through system-upgrade-controller (see `K3S_OPERATIONS.md`).
- Any change to manifests under `apps/`, `infrastructure/`, or applied via ArgoCD.
- Deleting PVs / PVCs. Any pod that gets rescheduled off the target CP stays on its new node.
- Fixing application-level problems.
- Rebuilding the node — if a CP does not return after reboot, escalate; the rebuild path is `ADD_WORKER_NODE.md` plus board approval, not this document.
---
## Approval gates
This procedure has **two** independent gates. Neither happens without explicit board approval on the corresponding Paperclip issue:
1. **Add swap (Phase A).** Non-invasive, non-state-mutating, kubelet already runs with `failSwapOn=false`. Requires board approval per stateful-service safety rules because it modifies a CP node.
2. **Full OS update (Phases B–E).** Requires board approval because it drains + reboots one of the etcd members. Do NOT execute without an explicit `request_board_approval` acceptance on the execution ticket.
Both gates are independent — swap can (and should) be added first, in a quiet window, before the full update is scheduled.
---
## CP ordering rule (multiple CPs in one cycle)
- **One at a time. Never two CPs cordoned or draining at once.** Two of three CPs down = etcd quorum loss = api-server unavailability for the whole cluster.
- **Leader last.** Query the current etcd leader before starting; update the two followers first (in any order), then the leader. Rationale: draining a follower is a no-op for the raft leader; draining the leader forces a re-election. Doing followers first minimises leader flapping.
- **Health gate between CPs.** After each CP finishes Phase F (finalize) and cluster health is green, wait at least `POST_UNCORDON_WAIT_SECONDS` (default 180 s) and re-check etcd endpoint status before touching the next CP. This gives etcd time to fully re-sync the just-rebooted member.
- **Halt on any yellow.** Any of the following aborts the cycle at the current CP (do NOT proceed to the next CP): etcd reports a member as not `started`, `kubectl get nodes` > 5 s, any node not `Ready`, any workload deployment/sts under desired replicas.
Query the leader:
```bash
ssh root@$CP1_HOST bash -c '
ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
endpoint status --cluster -w table'
```
---
## Current CP workload (snapshot 2026-08-23)
Re-derive live before any execution — this table is a design reference only.
| Node | Kernel | Swap | StatefulSets | Single-replica Deployments | DaemonSets |
|------|--------|------|--------------|-----------------------------|------------|
| k3s-cp-1 | 6.8.0-137 | **0 B** | 0 | 0 | node-exporter, promtail, hcloud-csi-node, svclb-* |
| k3s-cp-2 | 6.8.0-137 | **0 B** | 0 | metrics-server (evictable) | node-exporter, promtail, hcloud-csi-node, svclb-* |
| k3s-cp-3 | 6.8.0-137 | **0 B** | 0 | 0 | node-exporter, promtail, hcloud-csi-node, svclb-* |
etcd datastore live snapshot: 38 MiB per member, all three `started`, cp-3 is current leader.
Because CPs are `NoSchedule`-tainted, no workload will end up on them via normal scheduling — the pre-drain stateful redistribution loop from the old cp-1 procedure is obsolete.
---
## Phase A — Add swap (one-off, idempotent)
**Goal:** eliminate the "3.7 GiB, no swap" underlying constraint on any CP before ever draining it. All three CPs currently have zero swap; add swap once per CP.
**Preconditions:**
- kubelet in this k3s already runs with `failSwapOn=false` (confirmed on cp-1 via `/api/v1/nodes/k3s-cp-1/proxy/configz`) — enabling swap does NOT break the kubelet.
- Target CP's `/` has ≥ `2 * SWAP_SIZE_MB` free.
- Board approval on the swap-add ticket.
**Sizing:** default **4 GiB** swap. Rationale — cp-1 baseline (k3s + hosted daemons) sits at ~2 GiB used; 4 GiB swap gives us headroom for the drain-eviction transient without inflating disk usage past ~10 % of `/`. Minimum acceptable per guardrail: 2 GiB.
**Location:** `/swapfile` (root filesystem). Not a separate partition — reversible, no LVM changes.
**Steps (encoded in `update-cp-node.sh <node> --add-swap`):** identical to the old cp-1 procedure — `fallocate` → `mkswap` → `swapon` → `/etc/fstab` → `vm.swappiness=10` in `/etc/sysctl.d/99-k3s-swap.conf`. Idempotent: skips if a swapfile of the target size is already active. Full script source in `scripts/os-update/update-cp-node.sh`.
**Phase A also installs `etcd-client`** on the target CP if `etcdctl` is missing. This is a hard requirement for the etcd-quorum probes used in Phase B/E when the CP is later itself the update target and one of the *other* CPs must query etcd cluster status. cp-1 already has `etcdctl` (installed pre-HA); cp-2 and cp-3 pick it up here.
**Rollback for Phase A:** `swapoff /swapfile && rm /swapfile` and remove the fstab line. Safe at any time — swap is a soft resource.
**Verification after Phase A:**
- `free -h` shows `Swap: 4.0Gi` used ≈ 0.
- `swapon --show` shows `/swapfile 4G`.
- `sysctl vm.swappiness` returns `10`.
- kubelet still Ready (`kubectl get node <node>`).
- No new `MemoryPressure` condition.
---
## Phase B — Preflight
Runs from the operator machine (or from any healthy CP). Every command's stdout+stderr goes to `/tmp/os-update-<node>-<UTC-timestamp>.log` on the operator machine. Attach that log to the Paperclip execution ticket at the end.
Run `update-cp-node.sh <node> --preflight`. Checks:
1. **Cluster is currently healthy:** `cluster-health.sh` returns 0.
2. **Target node is a CP:** `kubectl get node <node>` carries the `node-role.kubernetes.io/control-plane` label.
3. **All 3 CPs are healthy etcd members:** `etcdctl endpoint status --cluster` shows every member `started`, no errors.
4. **Only one CP is being updated this cycle:** no other CP is currently cordoned.
5. **Target CP has swap on:** abort if `free -h` shows `Swap: 0B`. Run `--add-swap` first.
6. **kubectl latency probe:** `time kubectl get nodes` returns in ≤ 5 s.
7. **etcd snapshot** (restore point): `k3s etcd-snapshot save --name pre-cp-os-update-<node>-<UTC-timestamp>` executed on the target CP.
8. **Records the current etcd leader** — if the target *is* the leader, prints a warning ("prefer updating a follower first"). Does NOT auto-swap the target; the operator/agent makes the call per the CP ordering rule.
If any preflight check fails: STOP. Do not proceed.
---
## Phase C — Cordon + drain the target CP
CPs are `NoSchedule`-tainted, so cordon is mostly a belt-and-suspenders measure. Drain evicts the small tail of not-DaemonSet workloads (`metrics-server` currently lives on cp-2).
```bash
kubectl cordon <node>
kubectl drain <node> \
--ignore-daemonsets \
--delete-emptydir-data \
--timeout="${DRAIN_TIMEOUT_SECONDS:-600}s"
```
If drain reports a PDB block: do NOT `--force`. Uncordon the node, mark the run `blocked` on the PDB, and escalate. This is a workload PDB bug — fix separately.
**Kine-cascade guardrail (softened for HA etcd):** during drain, `time kubectl get nodes` should still return in ≤ 5 s. Because we now have 3-member etcd (not lone SQLite kine), a single-CP drain does not create the write-amplification hazard from [DEV-495](/DEV/issues/DEV-495). But `> 5 s` still indicates something is wrong (etcd slow disk, leader flapping) — halt and investigate.
---
## Phase D — apt on the target CP
Identical to the worker-node apt block (`update-node.sh` step 3). `update-cp-node.sh <node> --apt` runs this via `ssh root@<node> bash -s`:
```bash
export DEBIAN_FRONTEND=noninteractive
APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
uname -r > /root/pre-apt-kernel
dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv
if dpkg --audit | grep -qE .; then dpkg --configure -a || true; fi
apt-get update
apt-get $APT_OPTS upgrade || { apt-get $APT_OPTS -f install; apt-get $APT_OPTS upgrade; }
apt-get $APT_OPTS dist-upgrade
apt-get $APT_OPTS autoremove --purge
apt-get clean
[ -f /var/run/reboot-required ] && echo REBOOT_REQUIRED=yes || echo REBOOT_REQUIRED=no
```
`pre-apt-kernel` and `pre-apt-packages.tsv` are the local rollback references — see Rollback below.
---
## Phase E — Reboot handling (etcd-quorum-aware)
**Duration expectation:** 90–180 s of *this member's* api-server unavailability. The other two apiservers keep serving; the operator machine's kubectl continues to work via one of them.
**Escalation trigger:** rebooted apiserver not back on `/livez` after **10 minutes** → escalate. First check the Hetzner console via `hcloud server describe <node>` for boot state; if kernel-panic / initramfs, use grub previous-kernel path (see Rollback). Do NOT rebuild the node.
**Steps (executed by `update-cp-node.sh <node> --reboot`):**
```bash
# 1. Reboot the target CP (ssh will hang up mid-command — expected).
ssh $SSH_OPTS root@$TARGET_HOST 'systemctl reboot' || true
sleep 15
# 2. Poll the TARGET's /livez from the operator machine. It is served on port 6443.
# Insecure (`-k`) because the server cert is self-signed by k3s.
deadline=$(( $(date +%s) + ${REBOOT_MAX_WAIT_SECONDS:-600} ))
while [ $(date +%s) -lt $deadline ]; do
code=$(curl -sk -o /dev/null -w '%{http_code}' https://$TARGET_HOST:6443/livez 2>/dev/null || echo 000)
[ "$code" = "200" ] && break
sleep 5
done
# 3. Verify etcd membership from a peer CP — the target should be back as `started`.
ssh $SSH_OPTS root@$PEER_HOST '
ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
endpoint status --cluster -w table'
# 4. Wait for kubelet Ready on the target from the api-server view.
deadline=$(( $(date +%s) + 300 ))
while [ $(date +%s) -lt $deadline ]; do
ready=$(kubectl get node "$TARGET_NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
[ "$ready" = "True" ] && break
sleep 5
done
```
**What to do if the api-server takes 3–10 minutes:** it may be normal for a large etcd on a slow disk. Wait it out. Do NOT restart k3s to "help" — an etcd member catching up on raft log is stateful and interrupting it can slow the whole cluster.
**What to do if it takes > 10 minutes:** escalate; run the Hetzner-console diagnosis; if the console shows a bootable Ubuntu but the k3s service is failing, that's the boundary — this procedure stops here. Follow `K3S_OPERATIONS.md` for the k3s recovery path.
---
## Phase F — Uncordon + verify + finalize
```bash
kubectl uncordon <node>
sleep "${POST_UNCORDON_WAIT_SECONDS:-180}"
RETRY_ON_TRANSIENT=1 infrastructure/scripts/os-update/cluster-health.sh
ssh root@<node> 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
```
Between CPs (when multiple CPs will be updated in the same cycle), also verify:
```bash
ssh root@<peer-cp> '
ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
endpoint status --cluster -w table'
```
All three members must be `started`, no errors, DB sizes within 10 % of each other.
Attach the `/tmp/os-update-<node>-<ts>.log` to the execution ticket. Mark the ticket `done` on green health, or `blocked` naming the specific residual issue.
---
## Rollback / recovery
### If apt broke a package
- On the target CP, `dpkg --audit` to find half-configured packages.
- `dpkg --configure -a`, then `apt-get -f install`.
- If a specific package broke and you know the previous version from `/root/pre-apt-packages.tsv`, `apt-get install <pkg>=<old-version>`.
### If the new kernel does not boot
- Hetzner cloud console: `hcloud server request-console <node>` → get VNC URL, watch boot.
- If grub is up, select the previous-kernel entry. Ubuntu keeps ≥ 1 old kernel installed by default.
- Once booted on the old kernel, `apt-get remove` the broken kernel and pin the working one:
```bash
apt-mark hold linux-image-<broken-version> linux-headers-<broken-version>
```
- Escalate on the ticket regardless — a kernel rollback is a follow-up investigation, not a "done" outcome.
### If the node does not return at all
- Do NOT `hcloud server delete`. Do NOT re-provision.
- Escalate to the board with the Hetzner console output and `/tmp/os-update-<node>-*.log`.
- The `--cluster-reset --cluster-reset-restore-path=<snapshot>` recovery path exists (see `K3S_OPERATIONS.md`) but requires board approval per stateful-service safety rules. The pre-flight snapshot from Phase B is the restore point.
- **HA-etcd advantage:** if one CP is permanently lost but the other two are healthy, you have quorum. Do not rush a `cluster-reset`. Follow the "member remove + re-add" path in the k3s docs first.
### If etcd quorum degrades mid-cycle
- Two of three CPs down = write outage until quorum returns. Do NOT attempt to update the third.
- Uncordon all CPs, restore the just-updated one first, and confirm all three members `started` before doing anything else.
---
## Automation entry points
- `infrastructure/scripts/os-update/update-cp-node.sh <node> [phase-flag]` — the generalized HA-aware flow.
- `--add-swap` — Phase A only (idempotent, safe standalone).
- `--dry-run` — walk all phases printing what would be done, no state change.
- `--preflight` — Phase B only.
- `--drain` — Phase C only (cordon + drain).
- `--apt` — Phase D apt commands only.
- `--reboot` — Phase E only.
- `--finalize` — Phase F only.
- `--run` — all phases with a confirmation between each unless `ASSUME_YES=1`.
- `infrastructure/scripts/os-update/update-cp-1.sh` — retained as a thin wrapper that calls `update-cp-node.sh k3s-cp-1 "$@"`. Historical callers keep working.
The script follows the same log-dir contract as `update-node.sh` (`/tmp/os-update-<node>-<UTC-timestamp>.log`).
---
## Change history
| Date | Change | By |
|------|--------|-----|
| 2026-08-16 | Initial cp-1-only procedure (DEV-496) | CTO agent |
| 2026-08-23 | Generalized to HA (cp-1/cp-2/cp-3); dropped Phase C batched stateful eviction (CPs are NoSchedule-tainted now); softened kine cascade guardrail (embedded etcd 3-member quorum); added CP ordering rule (leader last, one at a time); introduced `update-cp-node.sh` (DEV-515) | CTO agent |

30
infrastructure/OS_UPDATE_PROCEDURE.md
View file

@ -22,9 +22,9 @@
| Role | Node | Private IP | Public IP | Datacenter | Notes |
|------|------|-----------|-----------|------------|-------|
| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | EXCLUDED from `os-update.sh` — see CP1_UPDATE_PROCEDURE.md |
| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | EXCLUDED — joined 2026-08-22 (DEV-510); needs HA-aware CP OS-update procedure |
| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | EXCLUDED — joined 2026-08-22 (DEV-510); needs HA-aware CP OS-update procedure |
| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | EXCLUDED from `os-update.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
| worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | |
| worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | |
| worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | |
@ -42,7 +42,7 @@ ssh root@178.105.17.239 'kubectl get nodes -o wide'
1. runner + workers that host no PVs (safest — lowest disruption)
2. remaining workers
3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
4. **ALL control-plane nodes excluded** — `os-update.sh` selects nodes without the `node-role.kubernetes.io/control-plane` label, so `k3s-cp-1`, `k3s-cp-2`, and `k3s-cp-3` are all skipped automatically. CPs need their own procedure (`CP1_UPDATE_PROCEDURE.md` covers cp-1 today; an HA-aware successor is a follow-up).
4. **ALL control-plane nodes excluded** — `os-update.sh` selects nodes without the `node-role.kubernetes.io/control-plane` label, so `k3s-cp-1`, `k3s-cp-2`, and `k3s-cp-3` are all skipped automatically. CPs have their own procedure ([`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) — HA-aware, one CP at a time, leader last).
**Concurrency:** exactly one node at a time. Never in parallel.
@ -50,14 +50,16 @@ ssh root@178.105.17.239 'kubectl get nodes -o wide'
## Kine thundering-herd guardrails (added after 2026-08-16 incident, see DEV-495)
The k3s control plane uses embedded SQLite (kine) as its datastore. On a single-CP cluster with limited RAM and no swap (cp-1 currently: 3.7 GiB RAM, 0 swap), draining a worker with many StatefulSets triggers a self-amplifying eviction-and-slow-SQL cascade: kine backs up on writes → apiserver hangs → node-lease renewals fail → taint-eviction kicks in on more nodes → more writes → kine falls further behind → OOM risk on cp-1.
**Historical context (single-CP kine era, 2026-08-16).** When the cluster ran a single CP with embedded SQLite via kine, draining a worker with many StatefulSets could trigger a self-amplifying eviction-and-slow-SQL cascade: kine backs up on writes → apiserver hangs → node-lease renewals fail → taint-eviction kicks in on more nodes → more writes → kine falls further behind → OOM risk on cp-1.
**Current state (HA-etcd era, since DEV-510 on 2026-08-22).** The datastore is now embedded etcd 3.6.12 with 3-node quorum across cp-1/cp-2/cp-3. A single-worker drain no longer creates the lone-SQLite writer risk, and etcd tolerates one member being slow or briefly unreachable. Guardrails (1) and (3) below are retained as sanity checks but the failure mode they were named for is much less likely to trigger. Guardrail (2) is largely done (add swap on cp-1/cp-2/cp-3 tracked in the CP update procedure).
All four rules below MUST be observed on every DEV-478 fire. The reference implementation (`os-update.sh`) does not yet enforce (1)/(3) automatically; the operator must actively watch.
1. **Pre-plan drain order for StatefulSets.** Before draining any worker, `kubectl get pods -n <ns> -o wide` against every namespace with StatefulSets and count how many will be evicted from the target node. If a single drain would evict **more than 3 StatefulSets at once**, redistribute first: cordon+delete individual StatefulSet pods one namespace at a time and wait for each reschedule to settle before draining the whole node.
2. **cp-1 must have swap before finishing the cycle.** cp-1 has 0 swap. Add at least 2 GiB of swap on cp-1 before the cp-1 update step (and ideally before draining the last stateful-heavy worker). This is a one-off setup task; once done it is a durable capability.
2. **All CPs must have swap before their update cycle.** cp-1/cp-2/cp-3 currently all have 0 swap. Add at least 2 GiB (default 4 GiB) of swap on each CP before its OS-update run — see Phase A in [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md). This is a one-off setup task per CP; once done it is a durable capability.
3. **Halt on kine slowness.** During any drain, keep a `time kubectl get nodes` running from cp-1. If it exceeds **5 s** in real time, halt the cycle immediately (uncordon the current node, do not proceed), verify cluster health, and escalate. The 5 s threshold is the leading indicator that kine has fallen behind and the taint-eviction cascade is about to start.
4. **cp-1 update is a separate design task.** Draining cp-1's own pods (Stalwart, coredns, harbor-database if there, etc.) plus rebooting the single apiserver is the highest-risk step of the whole cycle. It MUST be planned and approved as a distinct issue before it runs; it is NOT covered by the standard `os-update.sh` cycle. See `CP1_UPDATE_PROCEDURE.md` and `scripts/os-update/update-cp-1.sh` (DEV-496).
4. **CP updates are a separate design task.** With HA etcd across three CPs the risk is now etcd quorum loss (two CPs down simultaneously), not the sole-apiserver reboot. CP updates MUST be planned and board-approved per node; they are NOT covered by the standard `os-update.sh` cycle. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) and `scripts/os-update/update-cp-node.sh` (DEV-515). The old cp-1-only doc `CP1_UPDATE_PROCEDURE.md` is a redirect stub.
---
@ -249,13 +251,13 @@ Only proceed to the next node when ALL of the above are green. If not:
### 8. Control plane special handling
`k3s-cp-1` is the single control plane node and is **NOT** updated by the weekly `os-update.sh` cycle — it has its own dedicated procedure and script. See `CP1_UPDATE_PROCEDURE.md` and `scripts/os-update/update-cp-1.sh`. Reasons:
All three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) are **NOT** updated by the weekly `os-update.sh` cycle — they have their own dedicated procedure and script: [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) and `scripts/os-update/update-cp-node.sh`. Reasons:
- cp-1 hosts many stateful pods (currently 5 StatefulSets + 8 single-replica Deployments) — a one-shot `kubectl drain` would trigger the kine/SQLite cascade documented in the "Kine thundering-herd guardrails" section above (DEV-495).
- cp-1 has 3.7 GiB RAM and (until Phase A of the cp-1 procedure is done) zero swap. The cp-1 procedure adds a durable 4 GiB swapfile before draining.
- Rebooting cp-1 removes the entire kube-apiserver — the cp-1 procedure polls `/livez` from an external operator machine, not from cp-1 itself.
- CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. One CP at a time, leader last.
- All three CPs currently have zero swap. The CP procedure adds a durable 4 GiB swapfile per CP before draining.
- The CP procedure polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined.
`os-update.sh` will `SKIP` cp-1 and print a pointer to `CP1_UPDATE_PROCEDURE.md`.
`os-update.sh` excludes any node carrying the `node-role.kubernetes.io/control-plane` label (see `--dry-run` output) and prints a pointer to the CP procedure.
---
@ -318,7 +320,8 @@ If a node was skipped or errored → mark `blocked` with the unblock action, or
- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything. Explicitly skips `k3s-cp-1`.
- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry. Not for `k3s-cp-1`.
- `infrastructure/scripts/os-update/update-cp-1.sh` — dedicated cp-1 update flow (add swap, batched stateful eviction, apt, reboot with external liveness monitor). See `CP1_UPDATE_PROCEDURE.md`.
- `infrastructure/scripts/os-update/update-cp-node.sh <node>` — HA-aware CP update flow (add swap, preflight, drain, apt, reboot, finalize). Accepts `k3s-cp-1`, `k3s-cp-2`, or `k3s-cp-3`. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md).
- `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compat.
- `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure.
Read the script sources for the exact behavior before running them. They mirror this procedure step for step.
@ -337,3 +340,4 @@ A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this proce
|------|--------|-----|
| 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) |
| 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) |
| 2026-08-23 | Downgraded the kine cascade section for the HA-etcd era (DEV-510 migrated cp-1 from kine SQLite to embedded etcd, cp-2/cp-3 joined as full members). Pointed §8 and the "Automation entry points" list at the new HA-aware `CP_UPDATE_PROCEDURE.md` / `update-cp-node.sh`. Old `CP1_UPDATE_PROCEDURE.md` and `update-cp-1.sh` are now redirect stubs. | CTO agent (DEV-515) |

578
infrastructure/scripts/os-update/update-cp-1.sh
View file

@ -1,577 +1,13 @@
#!/bin/bash
# update-cp-1.sh — controlled OS update for the k3s control-plane node.
# update-cp-1.sh -- SUPERSEDED thin wrapper.
#
# See ../CP1_UPDATE_PROCEDURE.md for the full design rationale.
# Since DEV-515 (2026-08-23) the HA-aware entry point is update-cp-node.sh.
# This wrapper forwards to `update-cp-node.sh k3s-cp-1 "$@"` so historical
# callers keep working (routines, runbook references, board-approval docs).
#
# Usage:
# update-cp-1.sh --dry-run # print what would be done, touch nothing
# update-cp-1.sh --add-swap # Phase A only (idempotent, safe standalone)
# update-cp-1.sh --preflight # Phase B only
# update-cp-1.sh --drain-stateful # Phase C only (cordon + batched sts moves)
# update-cp-1.sh --apt # Phase D only (requires cp-1 fully drained)
# update-cp-1.sh --reboot # Phase E only (requires --apt reported REBOOT_REQUIRED=yes)
# update-cp-1.sh --finalize # Phase F only (uncordon + verify)
# update-cp-1.sh --run # all phases with a confirmation between each (or ASSUME_YES=1)
#
# Environment overrides:
# CP1_HOST default 178.105.17.239
# SWAP_SIZE_MB default 4096 (>=2048 required)
# SWAP_PATH default /swapfile
# DRAIN_TIMEOUT_SECONDS default 600
# REBOOT_MAX_WAIT_SECONDS default 600
# POST_UNCORDON_WAIT_SECONDS default 180
# STATEFUL_SETTLE_SECONDS default 60 (pause between batched sts moves)
# MIN_WORKER_MEM_MIB default 500 (per-worker MemAvailable floor at preflight)
# MIN_CP1_MEM_MIB default 200 (cp-1 MemAvailable floor mid-drain)
# MAX_KUBECTL_SECONDS default 5 (kine-health guardrail)
# SSH_OPTS default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new"
# ASSUME_YES=1 skip interactive confirmations in --run
# DRY_RUN=1 do not execute state-mutating commands; log-only
#
# Log directory contract (matches update-node.sh):
# Every command's stdout+stderr is tee'd to /tmp/os-update-cp-1-<UTC-ts>.log on
# the operator machine AND to /tmp/os-update-cp-1-<UTC-ts>.log on cp-1 (via the
# ssh command wrappers). Attach both to the execution ticket at the end.
#
# NEVER touches k3s config, k3s services, containerd, or any manifest. Only
# fixes it will attempt: apt/dpkg recovery on cp-1 (see Phase D). Any other
# problem → escalate and STOP.
# See ../CP_UPDATE_PROCEDURE.md for the current procedure.
# The old cp-1-only design lives in git history at commit 19af691.
set -euo pipefail
# --------------------------------------------------------------------------- #
# Config
# --------------------------------------------------------------------------- #
CP1_HOST="${CP1_HOST:-178.105.17.239}"
SWAP_SIZE_MB="${SWAP_SIZE_MB:-4096}"
SWAP_PATH="${SWAP_PATH:-/swapfile}"
DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}"
REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}"
POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}"
STATEFUL_SETTLE_SECONDS="${STATEFUL_SETTLE_SECONDS:-60}"
MIN_WORKER_MEM_MIB="${MIN_WORKER_MEM_MIB:-500}"
MIN_CP1_MEM_MIB="${MIN_CP1_MEM_MIB:-200}"
MAX_KUBECTL_SECONDS="${MAX_KUBECTL_SECONDS:-5}"
SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}"
DRY_RUN="${DRY_RUN:-0}"
ASSUME_YES="${ASSUME_YES:-0}"
NODE="k3s-cp-1"
TS="$(date -u +%Y%m%dT%H%M%SZ)"
LOG_LOCAL="/tmp/os-update-cp-1-${TS}.log"
LOG_REMOTE="/tmp/os-update-cp-1-${TS}.log"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
HEALTH_SCRIPT="$SCRIPT_DIR/cluster-health.sh"
# --------------------------------------------------------------------------- #
# Stateful eviction ledger — order matters (idle -> active, small -> heavy).
# --------------------------------------------------------------------------- #
# The list is derived live from the cluster in preflight and stored to
# /tmp/cp1-stateful-plan-<ts>.txt. This ORDER is the authoritative fallback.
STATEFUL_ORDER=(
"harbor:harbor-redis-0"
"nextcloud:nextcloud-redis-replicas-0"
"harbor:harbor-database-0"
"stalwart:stalwart-postgres-0"
"stalwart:stalwart-0"
)
# --------------------------------------------------------------------------- #
# Logging + safe-run helpers
# --------------------------------------------------------------------------- #
log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOG_LOCAL"; }
die() { log "FATAL: $*"; exit 1; }
warn() { log "WARN: $*"; }
# run/ssh_run/kubectl_run: honour DRY_RUN — print, do not execute state changes.
run() {
if [ "$DRY_RUN" = "1" ]; then
log "DRY-RUN would exec: $*"
return 0
fi
log "exec: $*"
"$@" 2>&1 | tee -a "$LOG_LOCAL"
}
ssh_run() {
local target="root@${CP1_HOST}"
if [ "$DRY_RUN" = "1" ]; then
log "DRY-RUN would ssh $target: $*"
return 0
fi
log "ssh $target: $*"
ssh $SSH_OPTS "$target" "$@" 2>&1 | tee -a "$LOG_LOCAL"
}
# ssh_run_stdin: pipe a heredoc through bash -s on cp-1; used for multi-line remote
# blocks (swap add, apt).
ssh_run_stdin() {
local target="root@${CP1_HOST}"
if [ "$DRY_RUN" = "1" ]; then
log "DRY-RUN would ssh $target with stdin script:"
sed 's/^/ | /' | tee -a "$LOG_LOCAL"
return 0
fi
log "ssh $target (heredoc)"
ssh $SSH_OPTS "$target" "bash -s" 2>&1 | tee -a "$LOG_LOCAL"
}
confirm() {
local prompt="$1"
if [ "$ASSUME_YES" = "1" ]; then
log "confirm SKIPPED (ASSUME_YES=1): $prompt"
return 0
fi
echo -n " >>> $prompt Continue? [y/N] "
read -r a
case "$a" in y|Y|yes|YES) return 0 ;; *) die "aborted by operator" ;; esac
}
# --------------------------------------------------------------------------- #
# Phase A — Add swap on cp-1 (idempotent)
# --------------------------------------------------------------------------- #
phase_add_swap() {
log "=== Phase A: add swap on $NODE ($SWAP_SIZE_MB MiB at $SWAP_PATH) ==="
[ "$SWAP_SIZE_MB" -ge 2048 ] || die "SWAP_SIZE_MB=$SWAP_SIZE_MB below 2048 MiB guardrail"
cat <<REMOTE | ssh_run_stdin
set -euo pipefail
SWAP_PATH="$SWAP_PATH"
SIZE_MB="$SWAP_SIZE_MB"
# Skip if a swapfile at this path is already active.
if swapon --show=NAME 2>/dev/null | grep -qx "\$SWAP_PATH"; then
echo "swap already on at \$SWAP_PATH — skipping"
free -h
exit 0
fi
# Root filesystem free space check — abort if less than 2*swap free.
avail_mb=\$(df -m --output=avail / | tail -1 | tr -d ' ')
need_mb=\$(( SIZE_MB * 2 ))
if [ "\$avail_mb" -lt "\$need_mb" ]; then
echo "ERROR: only \${avail_mb} MiB free on /, need \${need_mb} MiB (2x swap for safety)"
exit 1
fi
# Create swapfile. fallocate is fast; dd is the fallback.
if ! fallocate -l "\${SIZE_MB}M" "\$SWAP_PATH" 2>/dev/null; then
dd if=/dev/zero of="\$SWAP_PATH" bs=1M count="\$SIZE_MB" status=progress
fi
chmod 600 "\$SWAP_PATH"
mkswap "\$SWAP_PATH"
swapon "\$SWAP_PATH"
# Persist via fstab (dedup).
if ! grep -q "^\$SWAP_PATH " /etc/fstab; then
echo "\$SWAP_PATH none swap sw 0 0" >> /etc/fstab
fi
# Moderate swappiness — swap as safety net, not aggressive paging.
sysctl -w vm.swappiness=10
if [ ! -f /etc/sysctl.d/99-k3s-swap.conf ] || ! grep -q '^vm.swappiness' /etc/sysctl.d/99-k3s-swap.conf; then
echo 'vm.swappiness=10' > /etc/sysctl.d/99-k3s-swap.conf
fi
echo "--- swap after ---"
free -h
swapon --show
sysctl vm.swappiness
REMOTE
# Verify from operator view.
if [ "$DRY_RUN" != "1" ]; then
log "verifying kubelet still Ready after swap add"
local ready
ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
[ "$ready" = "True" ] || die "kubelet on $NODE not Ready after swap add — halt"
log "kubelet Ready=True — Phase A complete"
fi
}
# --------------------------------------------------------------------------- #
# Guardrail probes
# --------------------------------------------------------------------------- #
kubectl_latency_s() {
# returns integer seconds elapsed for `kubectl get nodes >/dev/null`.
local start end
start=$(date +%s)
kubectl get nodes >/dev/null 2>&1 || echo "kubectl-error" >&2
end=$(date +%s)
echo $(( end - start ))
}
cp1_mem_available_mib() {
# returns integer MiB `MemAvailable` on cp-1.
ssh $SSH_OPTS "root@$CP1_HOST" "awk '/^MemAvailable:/{printf \"%d\n\", \$2/1024}' /proc/meminfo" 2>/dev/null || echo 0
}
guard_kine_healthy() {
local s
s=$(kubectl_latency_s)
if [ "$s" -gt "$MAX_KUBECTL_SECONDS" ]; then
die "kubectl get nodes took ${s}s (>${MAX_KUBECTL_SECONDS}s) — kine cascade risk, HALT"
fi
log " kine ok: kubectl get nodes took ${s}s"
}
guard_cp1_memory() {
local m
m=$(cp1_mem_available_mib)
if [ "$m" -lt "$MIN_CP1_MEM_MIB" ]; then
die "cp-1 MemAvailable=${m} MiB below ${MIN_CP1_MEM_MIB} MiB floor — HALT"
fi
log " cp-1 mem ok: MemAvailable=${m} MiB"
}
# --------------------------------------------------------------------------- #
# Phase B — Preflight
# --------------------------------------------------------------------------- #
phase_preflight() {
log "=== Phase B: preflight ==="
log "-- cluster health"
if [ "$DRY_RUN" != "1" ]; then
if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then
die "cluster is not healthy — refuse to start cp-1 update"
fi
else
log "DRY-RUN would run: $HEALTH_SCRIPT"
fi
log "-- swap on cp-1"
if [ "$DRY_RUN" != "1" ]; then
local swap_total
swap_total=$(ssh $SSH_OPTS "root@$CP1_HOST" "awk '/^SwapTotal:/{print \$2}' /proc/meminfo")
[ "${swap_total:-0}" -ge $((2 * 1024 * 1024)) ] \
|| die "cp-1 SwapTotal=${swap_total} KiB below 2 GiB — run --add-swap first"
log " cp-1 SwapTotal=$(( swap_total / 1024 )) MiB"
fi
log "-- fsn1 workers Ready and have >=${MIN_WORKER_MEM_MIB} MiB MemAvailable"
# Worker private IPs (worker-4 nbg1 excluded — cannot host fsn1 RWO PVs).
local ips=("10.42.1.2" "10.42.1.3" "10.42.1.5" "10.42.1.7")
local names=("k3s-worker-1" "k3s-worker-2" "k3s-worker-3" "k3s-worker-5")
if [ "$DRY_RUN" != "1" ]; then
for i in "${!ips[@]}"; do
local nm="${names[$i]}"
local ip="${ips[$i]}"
local ready
ready=$(kubectl get node "$nm" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
[ "$ready" = "True" ] || die "worker $nm not Ready (Ready=$ready)"
local mem
mem=$(ssh $SSH_OPTS -J "root@$CP1_HOST" "root@$ip" \
"awk '/^MemAvailable:/{printf \"%d\n\", \$2/1024}' /proc/meminfo" 2>/dev/null || echo 0)
[ "$mem" -ge "$MIN_WORKER_MEM_MIB" ] \
|| die "worker $nm MemAvailable=${mem} MiB below ${MIN_WORKER_MEM_MIB} MiB — HALT"
log " $nm ok: MemAvailable=${mem} MiB"
done
fi
log "-- kine latency probe"
if [ "$DRY_RUN" != "1" ]; then guard_kine_healthy; fi
log "-- k3s SQLite datastore snapshot"
# k3s etcd-snapshot in embedded-SQLite mode copies the sqlite file to
# /var/lib/rancher/k3s/server/db/snapshots/.
ssh_run "k3s etcd-snapshot save --name pre-cp1-os-update-${TS}"
ssh_run "ls -la /var/lib/rancher/k3s/server/db/snapshots/ | tail -10"
log "-- persist eviction plan"
if [ "$DRY_RUN" != "1" ]; then
local plan="/tmp/cp1-stateful-plan-${TS}.txt"
ssh $SSH_OPTS "root@$CP1_HOST" "cat > $plan" <<EOF
# cp-1 stateful eviction plan generated at $TS
# Order: idle -> active, small -> heavy. Do NOT reorder without design review.
$(for e in "${STATEFUL_ORDER[@]}"; do echo "$e"; done)
EOF
log " wrote $plan on cp-1"
else
log "DRY-RUN would write /tmp/cp1-stateful-plan-${TS}.txt with STATEFUL_ORDER"
fi
log "=== Phase B: preflight OK ==="
}
# --------------------------------------------------------------------------- #
# Phase C — Cordon + move stateful pods off cp-1, one at a time
# --------------------------------------------------------------------------- #
phase_drain_stateful() {
log "=== Phase C: cordon + batched stateful move ==="
log "-- cordon $NODE (prevents rescheduled pods from landing back on cp-1)"
run kubectl cordon "$NODE"
local i=0
for entry in "${STATEFUL_ORDER[@]}"; do
i=$((i + 1))
local ns="${entry%%:*}"
local pod="${entry##*:}"
log "--- [${i}/${#STATEFUL_ORDER[@]}] moving $ns/$pod"
# Confirm the pod actually IS on cp-1 before touching it.
if [ "$DRY_RUN" != "1" ]; then
local on
on=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null || echo "")
if [ -z "$on" ]; then
log " $ns/$pod not found — sts may have changed; skipping"
continue
fi
if [ "$on" != "$NODE" ]; then
log " $ns/$pod already on $on (not cp-1) — skipping"
continue
fi
fi
# Delete the pod. StatefulSet controller will re-create it; scheduler will
# pick a fsn1 worker because cp-1 is cordoned and worker-4 is nbg1.
run kubectl -n "$ns" delete pod "$pod" --wait=false
if [ "$DRY_RUN" = "1" ]; then
log " DRY-RUN skipping wait-for-ready"
continue
fi
log " waiting for $ns/$pod to be Ready on a non-cp-1 node (max 300s)"
local deadline=$(( $(date +%s) + 300 ))
local new_node="" ready=""
while [ $(date +%s) -lt $deadline ]; do
new_node=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null || echo "")
ready=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "")
if [ -n "$new_node" ] && [ "$new_node" != "$NODE" ] && [ "$ready" = "True" ]; then
log " $ns/$pod -> $new_node OK"
break
fi
sleep 5
done
[ "$ready" = "True" ] && [ "$new_node" != "$NODE" ] \
|| die "$ns/$pod did not reach Ready on a non-cp-1 node in 300s — HALT (leave cp-1 cordoned)"
log " guardrails after $ns/$pod"
guard_kine_healthy
guard_cp1_memory
if [ "$i" -lt "${#STATEFUL_ORDER[@]}" ]; then
log " settle pause ${STATEFUL_SETTLE_SECONDS}s"
sleep "$STATEFUL_SETTLE_SECONDS"
fi
done
log "=== Phase C: all stateful pods evicted from cp-1 ==="
}
# --------------------------------------------------------------------------- #
# Phase D — Drain remaining pods + apt on cp-1
# --------------------------------------------------------------------------- #
phase_apt() {
log "=== Phase D: drain (remaining deployments) + apt on $NODE ==="
# Guardrail: refuse to run if any StatefulSet pod is still on cp-1.
if [ "$DRY_RUN" != "1" ]; then
local sts_on_cp1
sts_on_cp1=$(kubectl get pods -A -o json --field-selector spec.nodeName=$NODE \
| jq -r '.items[] | select(.metadata.ownerReferences[0].kind=="StatefulSet") | "\(.metadata.namespace)/\(.metadata.name)"' \
| wc -l)
[ "$sts_on_cp1" -eq 0 ] \
|| die "$sts_on_cp1 StatefulSet pod(s) still on cp-1 — Phase C incomplete; refuse Phase D"
fi
log "-- drain (timeout ${DRAIN_TIMEOUT_SECONDS}s)"
set +e
if [ "$DRY_RUN" = "1" ]; then
log "DRY-RUN would run: kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --timeout=${DRAIN_TIMEOUT_SECONDS}s"
local rc=0
else
kubectl drain "$NODE" \
--ignore-daemonsets \
--delete-emptydir-data \
--timeout="${DRAIN_TIMEOUT_SECONDS}s" 2>&1 | tee -a "$LOG_LOCAL"
local rc=${PIPESTATUS[0]}
fi
set -e
if [ "$rc" -ne 0 ]; then
log "drain FAILED (rc=$rc). Never force. Uncordoning."
run kubectl uncordon "$NODE"
die "drain failed on $NODE — investigate PDB / orphan pods; do NOT proceed"
fi
log "-- apt on cp-1"
cat <<'REMOTE' | ssh_run_stdin
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
# Rollback references (Phase-A didn't need this, but Phase D does).
uname -r > /root/pre-apt-kernel
dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv
echo "pre-apt kernel: $(cat /root/pre-apt-kernel)"
# Recover from any half-finished dpkg state before touching apt.
if dpkg --audit | grep -qE .; then
echo "dpkg audit reported issues, running dpkg --configure -a"
dpkg --configure -a || true
fi
apt-get update
if ! apt-get $APT_OPTS upgrade; then
echo "upgrade failed, attempting apt-get -f install"
apt-get $APT_OPTS -f install
apt-get $APT_OPTS upgrade
fi
apt-get $APT_OPTS dist-upgrade
apt-get $APT_OPTS autoremove --purge
apt-get clean
if [ -f /var/run/reboot-required ]; then
echo "REBOOT_REQUIRED=yes"
echo "REBOOT_REASON<<EOF"
cat /var/run/reboot-required.pkgs 2>/dev/null || echo "(no package list)"
echo "EOF"
else
echo "REBOOT_REQUIRED=no"
fi
REMOTE
log "=== Phase D: apt complete (check REBOOT_REQUIRED in the log) ==="
}
# --------------------------------------------------------------------------- #
# Phase E — Reboot cp-1 and wait for api-server /livez
# --------------------------------------------------------------------------- #
phase_reboot() {
log "=== Phase E: reboot $NODE ==="
# Tell cp-1 to reboot. The ssh command will hang up mid-command — that's fine.
if [ "$DRY_RUN" != "1" ]; then
log "issuing 'systemctl reboot' on $NODE (ssh will drop; expected)"
ssh $SSH_OPTS "root@$CP1_HOST" 'systemctl reboot' 2>&1 | tee -a "$LOG_LOCAL" || true
log "waiting 15s for ssh to fully drop before polling api-server"
sleep 15
else
log "DRY-RUN would ssh root@$CP1_HOST 'systemctl reboot'"
fi
log "-- poll api-server /livez (timeout ${REBOOT_MAX_WAIT_SECONDS}s)"
if [ "$DRY_RUN" != "1" ]; then
local deadline=$(( $(date +%s) + REBOOT_MAX_WAIT_SECONDS ))
local code=000
while [ $(date +%s) -lt $deadline ]; do
code=$(curl -sk -o /dev/null -w '%{http_code}' "https://$CP1_HOST:6443/livez" 2>/dev/null || echo 000)
if [ "$code" = "200" ]; then
log " api-server /livez=200 (elapsed $(( REBOOT_MAX_WAIT_SECONDS - (deadline - $(date +%s)) ))s)"
break
fi
sleep 5
done
[ "$code" = "200" ] || die "api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate; check 'hcloud server describe k3s-cp-1' and Hetzner console"
fi
log "-- wait for kubelet Ready on $NODE (max 300s)"
if [ "$DRY_RUN" != "1" ]; then
local deadline=$(( $(date +%s) + 300 ))
local ready=Unknown
while [ $(date +%s) -lt $deadline ]; do
ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
[ "$ready" = "True" ] && break
sleep 5
done
[ "$ready" = "True" ] || die "kubelet on $NODE never returned Ready — escalate (do NOT change k3s config)"
log " kubelet Ready=True"
fi
log "=== Phase E: cp-1 is back ==="
}
# --------------------------------------------------------------------------- #
# Phase F — Uncordon + verify + finalize
# --------------------------------------------------------------------------- #
phase_finalize() {
log "=== Phase F: uncordon + verify ==="
run kubectl uncordon "$NODE"
log "-- settle wait ${POST_UNCORDON_WAIT_SECONDS}s"
[ "$DRY_RUN" = "1" ] || sleep "$POST_UNCORDON_WAIT_SECONDS"
log "-- cluster health"
if [ "$DRY_RUN" != "1" ]; then
if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then
die "cluster health failed after cp-1 update — escalate, do NOT touch k3s"
fi
fi
log "-- apt history summary (audit)"
ssh_run 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
log "-- old snapshots (>30d) — listing only, review manually"
ssh_run 'find /var/lib/rancher/k3s/server/db/snapshots/ -type f -mtime +30 -name "pre-*" -print 2>/dev/null || true'
log "=== cp-1 OS update complete — attach $LOG_LOCAL to the execution ticket ==="
}
# --------------------------------------------------------------------------- #
# --run — orchestrate all phases with confirmations
# --------------------------------------------------------------------------- #
phase_run_all() {
log "=== full cp-1 update run (log: $LOG_LOCAL) ==="
confirm "Phase A (add swap) — proceed?"
phase_add_swap
confirm "Phase B (preflight) — proceed?"
phase_preflight
confirm "Phase C (cordon + batched stateful move) — proceed?"
phase_drain_stateful
confirm "Phase D (drain remaining + apt) — proceed?"
phase_apt
confirm "Phase E (reboot cp-1; api-server unavailable ~90-180s) — proceed?"
phase_reboot
confirm "Phase F (uncordon + verify) — proceed?"
phase_finalize
log "=== FULL RUN COMPLETE ==="
}
# --------------------------------------------------------------------------- #
# Arg parse
# --------------------------------------------------------------------------- #
usage() { grep -E '^# ' "$0" | sed 's/^# \{0,1\}//'; exit 2; }
[ $# -ge 1 ] || usage
# Init the local log file up front so tee always has a target.
: > "$LOG_LOCAL"
log "update-cp-1.sh started (DRY_RUN=$DRY_RUN)"
log "log file: $LOG_LOCAL"
case "$1" in
--dry-run)
DRY_RUN=1
export DRY_RUN
log "DRY_RUN=1 — walking Phases A..F without touching state"
phase_add_swap
phase_preflight
phase_drain_stateful
phase_apt
phase_reboot
phase_finalize
;;
--add-swap) phase_add_swap ;;
--preflight) phase_preflight ;;
--drain-stateful) phase_drain_stateful ;;
--apt) phase_apt ;;
--reboot) phase_reboot ;;
--finalize) phase_finalize ;;
--run) phase_run_all ;;
-h|--help) usage ;;
*) echo "unknown arg: $1" >&2; usage ;;
esac
exec "$SCRIPT_DIR/update-cp-node.sh" k3s-cp-1 "$@"

595
infrastructure/scripts/os-update/update-cp-node.sh Executable file
View file

@ -0,0 +1,595 @@
#!/bin/bash
# update-cp-node.sh — controlled OS update for any HA k3s control-plane node.
#
# See ../CP_UPDATE_PROCEDURE.md for the full design rationale.
#
# Usage:
# update-cp-node.sh <node> --dry-run # print what would be done, touch nothing
# update-cp-node.sh <node> --add-swap # Phase A only (idempotent, safe standalone)
# update-cp-node.sh <node> --preflight # Phase B only
# update-cp-node.sh <node> --drain # Phase C only (cordon + drain)
# update-cp-node.sh <node> --apt # Phase D only (requires <node> already fully drained)
# update-cp-node.sh <node> --reboot # Phase E only (requires --apt reported REBOOT_REQUIRED=yes)
# update-cp-node.sh <node> --finalize # Phase F only (uncordon + verify)
# update-cp-node.sh <node> --run # all phases with confirmation between each (or ASSUME_YES=1)
#
# <node> must be one of: k3s-cp-1, k3s-cp-2, k3s-cp-3.
#
# Environment overrides:
# SWAP_SIZE_MB default 4096 (>=2048 required)
# SWAP_PATH default /swapfile
# DRAIN_TIMEOUT_SECONDS default 600
# REBOOT_MAX_WAIT_SECONDS default 600
# POST_UNCORDON_WAIT_SECONDS default 180
# MIN_TARGET_MEM_MIB default 200 (target-CP MemAvailable floor mid-drain)
# MAX_KUBECTL_SECONDS default 5 (kine-latency guardrail; softened for HA etcd)
# SSH_OPTS default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new"
# ASSUME_YES=1 skip interactive confirmations in --run
# DRY_RUN=1 do not execute state-mutating commands; log-only
#
# Log directory contract (matches update-node.sh):
# Every command's stdout+stderr is tee'd to /tmp/os-update-<node>-<UTC-ts>.log on
# the operator machine. Attach it to the execution ticket at the end.
#
# NEVER touches k3s config, k3s services, containerd, or any manifest. Only
# fixes it will attempt: apt/dpkg recovery on the target CP (Phase D). Any other
# problem -> escalate and STOP.
set -euo pipefail
# --------------------------------------------------------------------------- #
# CP topology — public/routable IPs used to reach each CP via ssh + /livez.
# Kept out-of-cluster on purpose: during a full-cluster incident the operator
# machine must be able to reach each CP without going through k3s.
# --------------------------------------------------------------------------- #
declare -A CP_HOST=(
[k3s-cp-1]="178.105.17.239"
[k3s-cp-2]="188.245.85.199"
[k3s-cp-3]="49.13.92.162"
)
# --------------------------------------------------------------------------- #
# Arg parse — expect <node> as $1
# --------------------------------------------------------------------------- #
usage() { grep -E '^# ' "$0" | sed 's/^# \{0,1\}//'; exit 2; }
[ $# -ge 2 ] || usage
NODE="$1"; shift
TARGET_HOST="${CP_HOST[$NODE]:-}"
if [ -z "$TARGET_HOST" ]; then
echo "unknown CP node: $NODE (allowed: ${!CP_HOST[*]})" >&2
exit 2
fi
# --------------------------------------------------------------------------- #
# Config
# --------------------------------------------------------------------------- #
SWAP_SIZE_MB="${SWAP_SIZE_MB:-4096}"
SWAP_PATH="${SWAP_PATH:-/swapfile}"
DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}"
REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}"
POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}"
MIN_TARGET_MEM_MIB="${MIN_TARGET_MEM_MIB:-200}"
MAX_KUBECTL_SECONDS="${MAX_KUBECTL_SECONDS:-5}"
SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}"
DRY_RUN="${DRY_RUN:-0}"
ASSUME_YES="${ASSUME_YES:-0}"
TS="$(date -u +%Y%m%dT%H%M%SZ)"
LOG_LOCAL="/tmp/os-update-${NODE}-${TS}.log"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
HEALTH_SCRIPT="$SCRIPT_DIR/cluster-health.sh"
# Pick a peer CP for etcd status probes. Requirements:
# - not the target
# - etcdctl available (via non-login ssh PATH)
# We probe each candidate at parse time. Order preference: cp-1 first (it has
# etcdctl installed since the pre-HA era), then cp-2, then cp-3. The Phase A
# `--add-swap` step also installs `etcd-client` via apt so cp-2/cp-3 pick up
# etcdctl on their first pass.
peer_host_for() {
local target="$1"
local order=(k3s-cp-1 k3s-cp-2 k3s-cp-3)
for n in "${order[@]}"; do
[ "$n" = "$target" ] && continue
local h="${CP_HOST[$n]}"
if ssh $SSH_OPTS -o BatchMode=yes "root@$h" 'command -v etcdctl >/dev/null 2>&1' 2>/dev/null; then
echo "$h"; return 0
fi
done
# In DRY_RUN we don't need a real etcdctl-capable peer.
if [ "${DRY_RUN:-0}" = "1" ]; then
for n in "${order[@]}"; do
[ "$n" != "$target" ] && { echo "${CP_HOST[$n]}"; return 0; }
done
fi
# No non-target CP has etcdctl. Return empty so Phase B/E can fail
# explicitly with a targeted "install etcd-client on cp-X first" message.
echo ""
return 0
}
PEER_HOST="$(peer_host_for "$NODE")"
# --------------------------------------------------------------------------- #
# Logging + safe-run helpers
# --------------------------------------------------------------------------- #
log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOG_LOCAL"; }
die() { log "FATAL: $*"; exit 1; }
warn() { log "WARN: $*"; }
run() {
if [ "$DRY_RUN" = "1" ]; then
log "DRY-RUN would exec: $*"
return 0
fi
log "exec: $*"
"$@" 2>&1 | tee -a "$LOG_LOCAL"
}
ssh_run() {
local target="${1:-root@$TARGET_HOST}"; shift || true
if [ "$DRY_RUN" = "1" ]; then
log "DRY-RUN would ssh $target: $*"
return 0
fi
log "ssh $target: $*"
ssh $SSH_OPTS "$target" "$@" 2>&1 | tee -a "$LOG_LOCAL"
}
# ssh_run_stdin: pipe a heredoc through bash -s on the target; used for multi-line remote blocks.
ssh_run_stdin() {
local target="root@${TARGET_HOST}"
if [ "$DRY_RUN" = "1" ]; then
log "DRY-RUN would ssh $target with stdin script:"
sed 's/^/ | /' | tee -a "$LOG_LOCAL"
return 0
fi
log "ssh $target (heredoc)"
ssh $SSH_OPTS "$target" "bash -s" 2>&1 | tee -a "$LOG_LOCAL"
}
confirm() {
local prompt="$1"
if [ "$ASSUME_YES" = "1" ]; then
log "confirm SKIPPED (ASSUME_YES=1): $prompt"
return 0
fi
echo -n " >>> $prompt Continue? [y/N] "
read -r a
case "$a" in y|Y|yes|YES) return 0 ;; *) die "aborted by operator" ;; esac
}
# --------------------------------------------------------------------------- #
# etcd helpers — run etcdctl on the target OR on a peer CP.
# --------------------------------------------------------------------------- #
ETCDCTL_ENV='ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key'
etcd_status_via() {
# $1 = host (IP)
local host="$1"
ssh $SSH_OPTS "root@$host" "$ETCDCTL_ENV endpoint status --cluster -w table" 2>&1
}
etcd_leader_name() {
# returns the etcd member NAME (e.g. k3s-cp-3-9d305472) whose row shows IS LEADER=true.
# Uses simple-format output (no -w table) for reliable parsing.
local host="$1"
ssh $SSH_OPTS "root@$host" "$ETCDCTL_ENV endpoint status --cluster -w simple" 2>/dev/null \
| awk -F, '$5 ~ /true/ {print $1}'
}
target_is_leader() {
# returns 0 if the target IP is the current etcd leader, 1 otherwise.
local host="$1" # peer to query from
local target="$2" # target IP to compare
local leader_ep
leader_ep=$(ssh $SSH_OPTS "root@$host" "$ETCDCTL_ENV endpoint status --cluster -w simple" 2>/dev/null \
| awk -F, '$5 ~ /true/ {print $1}')
# leader_ep looks like https://49.13.92.162:2379
echo "$leader_ep" | grep -q "://${target}:" && return 0 || return 1
}
# --------------------------------------------------------------------------- #
# Phase A — Add swap on the target CP (idempotent)
# --------------------------------------------------------------------------- #
phase_add_swap() {
log "=== Phase A: add swap on $NODE ($SWAP_SIZE_MB MiB at $SWAP_PATH) ==="
[ "$SWAP_SIZE_MB" -ge 2048 ] || die "SWAP_SIZE_MB=$SWAP_SIZE_MB below 2048 MiB guardrail"
cat <<REMOTE | ssh_run_stdin
set -euo pipefail
SWAP_PATH="$SWAP_PATH"
SIZE_MB="$SWAP_SIZE_MB"
# Skip if a swapfile at this path is already active.
if swapon --show=NAME 2>/dev/null | grep -qx "\$SWAP_PATH"; then
echo "swap already on at \$SWAP_PATH -- skipping"
free -h
exit 0
fi
# Root filesystem free space check -- abort if less than 2*swap free.
avail_mb=\$(df -m --output=avail / | tail -1 | tr -d ' ')
need_mb=\$(( SIZE_MB * 2 ))
if [ "\$avail_mb" -lt "\$need_mb" ]; then
echo "ERROR: only \${avail_mb} MiB free on /, need \${need_mb} MiB (2x swap for safety)"
exit 1
fi
# Create swapfile. fallocate is fast; dd is the fallback.
if ! fallocate -l "\${SIZE_MB}M" "\$SWAP_PATH" 2>/dev/null; then
dd if=/dev/zero of="\$SWAP_PATH" bs=1M count="\$SIZE_MB" status=progress
fi
chmod 600 "\$SWAP_PATH"
mkswap "\$SWAP_PATH"
swapon "\$SWAP_PATH"
# Persist via fstab (dedup).
if ! grep -q "^\$SWAP_PATH " /etc/fstab; then
echo "\$SWAP_PATH none swap sw 0 0" >> /etc/fstab
fi
# Moderate swappiness -- swap as safety net, not aggressive paging.
sysctl -w vm.swappiness=10
if [ ! -f /etc/sysctl.d/99-k3s-swap.conf ] || ! grep -q '^vm.swappiness' /etc/sysctl.d/99-k3s-swap.conf; then
echo 'vm.swappiness=10' > /etc/sysctl.d/99-k3s-swap.conf
fi
echo "--- swap after ---"
free -h
swapon --show
sysctl vm.swappiness
# Ensure etcdctl is available for etcd-quorum probes (idempotent apt install).
# Needed because when this CP is the target of a later update, another CP
# must probe etcd cluster status; if cp-1 is the target, one of cp-2/cp-3
# is the probing peer and must have etcdctl.
if ! command -v etcdctl >/dev/null 2>&1; then
echo "--- installing etcd-client (provides etcdctl) ---"
export DEBIAN_FRONTEND=noninteractive
apt-get update -y >/dev/null
apt-get install -y etcd-client
command -v etcdctl && etcdctl version
fi
REMOTE
if [ "$DRY_RUN" != "1" ]; then
log "verifying kubelet still Ready after swap add"
local ready
ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
[ "$ready" = "True" ] || die "kubelet on $NODE not Ready after swap add -- halt"
log "kubelet Ready=True -- Phase A complete"
fi
}
# --------------------------------------------------------------------------- #
# Guardrail probes
# --------------------------------------------------------------------------- #
kubectl_latency_s() {
local start end
start=$(date +%s)
kubectl get nodes >/dev/null 2>&1 || echo "kubectl-error" >&2
end=$(date +%s)
echo $(( end - start ))
}
target_mem_available_mib() {
ssh $SSH_OPTS "root@$TARGET_HOST" "awk '/^MemAvailable:/{printf \"%d\n\", \$2/1024}' /proc/meminfo" 2>/dev/null || echo 0
}
guard_kine_healthy() {
local s
s=$(kubectl_latency_s)
if [ "$s" -gt "$MAX_KUBECTL_SECONDS" ]; then
die "kubectl get nodes took ${s}s (>${MAX_KUBECTL_SECONDS}s) -- etcd slow, HALT"
fi
log " kubectl-latency ok: ${s}s"
}
guard_target_memory() {
local m
m=$(target_mem_available_mib)
if [ "$m" -lt "$MIN_TARGET_MEM_MIB" ]; then
die "$NODE MemAvailable=${m} MiB below ${MIN_TARGET_MEM_MIB} MiB floor -- HALT"
fi
log " $NODE mem ok: MemAvailable=${m} MiB"
}
# --------------------------------------------------------------------------- #
# Phase B — Preflight
# --------------------------------------------------------------------------- #
phase_preflight() {
log "=== Phase B: preflight for $NODE ==="
log "-- cluster health"
if [ "$DRY_RUN" != "1" ]; then
if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then
die "cluster is not healthy -- refuse to start $NODE update"
fi
else
log "DRY-RUN would run: $HEALTH_SCRIPT"
fi
log "-- confirm $NODE is a control-plane node"
if [ "$DRY_RUN" != "1" ]; then
local is_cp
is_cp=$(kubectl get node "$NODE" -o jsonpath='{.metadata.labels.node-role\.kubernetes\.io/control-plane}' 2>/dev/null || echo "")
[ "$is_cp" = "true" ] || die "$NODE is not labelled control-plane -- refuse (use update-node.sh for workers)"
fi
log "-- verify no OTHER CP is currently cordoned"
if [ "$DRY_RUN" != "1" ]; then
local other_cordoned
other_cordoned=$(kubectl get nodes -l node-role.kubernetes.io/control-plane=true \
-o json | jq -r --arg n "$NODE" '.items[] | select(.metadata.name != $n) | select(.spec.unschedulable == true) | .metadata.name' \
| tr '\n' ' ')
if [ -n "${other_cordoned// /}" ]; then
die "another CP is already cordoned: $other_cordoned -- refuse (one CP at a time)"
fi
log " no other CP cordoned -- proceeding"
fi
log "-- etcd cluster status (all members must be started)"
if [ "$DRY_RUN" != "1" ]; then
if [ -z "$PEER_HOST" ]; then
die "no non-target CP has etcdctl installed -- run \`update-cp-node.sh <other-cp> --add-swap\` first on one of the OTHER CPs (that step installs etcd-client), then retry"
fi
etcd_status_via "$PEER_HOST" | tee -a "$LOG_LOCAL"
fi
log "-- swap on $NODE"
if [ "$DRY_RUN" != "1" ]; then
local swap_total
swap_total=$(ssh $SSH_OPTS "root@$TARGET_HOST" "awk '/^SwapTotal:/{print \$2}' /proc/meminfo")
[ "${swap_total:-0}" -ge $((2 * 1024 * 1024)) ] \
|| die "$NODE SwapTotal=${swap_total} KiB below 2 GiB -- run --add-swap first"
log " $NODE SwapTotal=$(( swap_total / 1024 )) MiB"
fi
log "-- kubectl-latency probe"
if [ "$DRY_RUN" != "1" ]; then guard_kine_healthy; fi
log "-- record current etcd leader"
if [ "$DRY_RUN" != "1" ] && [ -n "$PEER_HOST" ]; then
if target_is_leader "$PEER_HOST" "$TARGET_HOST"; then
warn "$NODE IS the current etcd leader. Per CP ordering rule, prefer updating a follower first."
warn " Not aborting -- operator/agent must confirm this is intentional."
else
log " $NODE is a FOLLOWER -- safe to proceed."
fi
fi
log "-- k3s etcd snapshot"
ssh_run "root@$TARGET_HOST" "k3s etcd-snapshot save --name pre-cp-os-update-${NODE}-${TS}"
ssh_run "root@$TARGET_HOST" "ls -la /var/lib/rancher/k3s/server/db/snapshots/ | tail -10"
log "=== Phase B: preflight OK ==="
}
# --------------------------------------------------------------------------- #
# Phase C — Cordon + drain
# --------------------------------------------------------------------------- #
phase_drain() {
log "=== Phase C: cordon + drain $NODE ==="
log "-- cordon $NODE"
run kubectl cordon "$NODE"
log "-- drain $NODE (timeout ${DRAIN_TIMEOUT_SECONDS}s)"
set +e
if [ "$DRY_RUN" = "1" ]; then
log "DRY-RUN would run: kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --timeout=${DRAIN_TIMEOUT_SECONDS}s"
local rc=0
else
kubectl drain "$NODE" \
--ignore-daemonsets \
--delete-emptydir-data \
--timeout="${DRAIN_TIMEOUT_SECONDS}s" 2>&1 | tee -a "$LOG_LOCAL"
local rc=${PIPESTATUS[0]}
fi
set -e
if [ "$rc" -ne 0 ]; then
log "drain FAILED (rc=$rc). Never force. Uncordoning."
run kubectl uncordon "$NODE"
die "drain failed on $NODE -- investigate PDB / orphan pods; do NOT proceed"
fi
log "-- post-drain guardrails"
if [ "$DRY_RUN" != "1" ]; then
guard_kine_healthy
guard_target_memory
fi
log "=== Phase C: $NODE drained ==="
}
# --------------------------------------------------------------------------- #
# Phase D — apt on the target
# --------------------------------------------------------------------------- #
phase_apt() {
log "=== Phase D: apt on $NODE ==="
cat <<'REMOTE' | ssh_run_stdin
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
uname -r > /root/pre-apt-kernel
dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv
echo "pre-apt kernel: $(cat /root/pre-apt-kernel)"
if dpkg --audit | grep -qE .; then
echo "dpkg audit reported issues, running dpkg --configure -a"
dpkg --configure -a || true
fi
apt-get update
if ! apt-get $APT_OPTS upgrade; then
echo "upgrade failed, attempting apt-get -f install"
apt-get $APT_OPTS -f install
apt-get $APT_OPTS upgrade
fi
apt-get $APT_OPTS dist-upgrade
apt-get $APT_OPTS autoremove --purge
apt-get clean
if [ -f /var/run/reboot-required ]; then
echo "REBOOT_REQUIRED=yes"
echo "REBOOT_REASON<<EOF"
cat /var/run/reboot-required.pkgs 2>/dev/null || echo "(no package list)"
echo "EOF"
else
echo "REBOOT_REQUIRED=no"
fi
REMOTE
log "=== Phase D: apt complete (check REBOOT_REQUIRED in the log) ==="
}
# --------------------------------------------------------------------------- #
# Phase E — Reboot and wait for target /livez + kubelet Ready + etcd rejoin
# --------------------------------------------------------------------------- #
phase_reboot() {
log "=== Phase E: reboot $NODE ==="
if [ "$DRY_RUN" != "1" ]; then
log "issuing 'systemctl reboot' on $NODE (ssh will drop; expected)"
ssh $SSH_OPTS "root@$TARGET_HOST" 'systemctl reboot' 2>&1 | tee -a "$LOG_LOCAL" || true
log "waiting 15s for ssh to fully drop before polling"
sleep 15
else
log "DRY-RUN would ssh root@$TARGET_HOST 'systemctl reboot'"
fi
log "-- poll $NODE api-server /livez (timeout ${REBOOT_MAX_WAIT_SECONDS}s)"
if [ "$DRY_RUN" != "1" ]; then
local deadline=$(( $(date +%s) + REBOOT_MAX_WAIT_SECONDS ))
local code=000
while [ $(date +%s) -lt $deadline ]; do
code=$(curl -sk -o /dev/null -w '%{http_code}' "https://$TARGET_HOST:6443/livez" 2>/dev/null || echo 000)
if [ "$code" = "200" ]; then
log " $NODE api-server /livez=200"
break
fi
sleep 5
done
[ "$code" = "200" ] || die "$NODE api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s -- escalate; check 'hcloud server describe $NODE' and Hetzner console"
fi
log "-- verify etcd cluster status from peer ($PEER_HOST) -- $NODE should be 'started'"
if [ "$DRY_RUN" != "1" ] && [ -n "$PEER_HOST" ]; then
etcd_status_via "$PEER_HOST" | tee -a "$LOG_LOCAL"
fi
log "-- wait for kubelet Ready on $NODE (max 300s)"
if [ "$DRY_RUN" != "1" ]; then
local deadline=$(( $(date +%s) + 300 ))
local ready=Unknown
while [ $(date +%s) -lt $deadline ]; do
ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
[ "$ready" = "True" ] && break
sleep 5
done
[ "$ready" = "True" ] || die "kubelet on $NODE never returned Ready -- escalate (do NOT change k3s config)"
log " kubelet Ready=True"
fi
log "=== Phase E: $NODE is back ==="
}
# --------------------------------------------------------------------------- #
# Phase F — Uncordon + verify + finalize
# --------------------------------------------------------------------------- #
phase_finalize() {
log "=== Phase F: uncordon + verify $NODE ==="
run kubectl uncordon "$NODE"
log "-- settle wait ${POST_UNCORDON_WAIT_SECONDS}s"
[ "$DRY_RUN" = "1" ] || sleep "$POST_UNCORDON_WAIT_SECONDS"
log "-- cluster health"
if [ "$DRY_RUN" != "1" ]; then
if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then
die "cluster health failed after $NODE update -- escalate, do NOT touch k3s"
fi
fi
log "-- etcd cluster status (all three should be started)"
if [ "$DRY_RUN" != "1" ] && [ -n "$PEER_HOST" ]; then
etcd_status_via "$PEER_HOST" | tee -a "$LOG_LOCAL"
fi
log "-- apt history summary (audit)"
ssh_run "root@$TARGET_HOST" 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
log "-- old snapshots (>30d) -- listing only, review manually"
ssh_run "root@$TARGET_HOST" 'find /var/lib/rancher/k3s/server/db/snapshots/ -type f -mtime +30 -name "pre-*" -print 2>/dev/null || true'
log "=== $NODE OS update complete -- attach $LOG_LOCAL to the execution ticket ==="
}
# --------------------------------------------------------------------------- #
# --run — orchestrate all phases with confirmations
# --------------------------------------------------------------------------- #
phase_run_all() {
log "=== full $NODE update run (log: $LOG_LOCAL) ==="
confirm "Phase A (add swap) -- proceed?"
phase_add_swap
confirm "Phase B (preflight) -- proceed?"
phase_preflight
confirm "Phase C (cordon + drain) -- proceed?"
phase_drain
confirm "Phase D (apt) -- proceed?"
phase_apt
confirm "Phase E (reboot $NODE; api-server on THIS node unavailable ~90-180s, other 2 CPs keep serving) -- proceed?"
phase_reboot
confirm "Phase F (uncordon + verify) -- proceed?"
phase_finalize
log "=== FULL RUN COMPLETE for $NODE ==="
}
# --------------------------------------------------------------------------- #
# Phase dispatch
# --------------------------------------------------------------------------- #
: > "$LOG_LOCAL"
log "update-cp-node.sh started (NODE=$NODE, TARGET_HOST=$TARGET_HOST, PEER_HOST=$PEER_HOST, DRY_RUN=$DRY_RUN)"
log "log file: $LOG_LOCAL"
case "$1" in
--dry-run)
DRY_RUN=1
export DRY_RUN
log "DRY_RUN=1 -- walking Phases A..F without touching state"
phase_add_swap
phase_preflight
phase_drain
phase_apt
phase_reboot
phase_finalize
;;
--add-swap) phase_add_swap ;;
--preflight) phase_preflight ;;
--drain) phase_drain ;;
--apt) phase_apt ;;
--reboot) phase_reboot ;;
--finalize) phase_finalize ;;
--run) phase_run_all ;;
-h|--help) usage ;;
*) echo "unknown arg: $1" >&2; usage ;;
esac