docs(os-update): point kine guardrails and order rules to cp-1 procedure

Follow-up to the cp-1 design commit: update the two remaining places in
OS_UPDATE_PROCEDURE.md that still said "cp-1 last" / "planned as a distinct
issue" so they now name CP1_UPDATE_PROCEDURE.md + update-cp-1.sh.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

infrastructure/CP1_UPDATE_PROCEDURE.md

@@ -0,0 +1,382 @@
+# k3s-cp-1 OS Update Procedure
+
+**Purpose:** apply the weekly Ubuntu OS updates to `k3s-cp-1`, the **sole** control-plane node of the k3s cluster, without triggering the kine/SQLite thundering-herd cascade documented in [DEV-495](/DEV/issues/DEV-495) and without losing api-server access longer than a normal reboot.
+
+**Audience:** the CTO agent, or an operator with root SSH to the cluster. Execution is board-approval-gated — see `## Approval gates` below.
+
+**Why cp-1 is not covered by the standard [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md):**
+
+- cp-1 has 3.7 GiB RAM and **zero swap** — during the 2026-08-16 worker-3 drain, `MemAvailable` dropped to 43 MiB while kine backed up on writes. cp-1's own drain evicts **more** state at once than any worker drain.
+- cp-1 currently hosts 5 StatefulSets (`harbor-database`, `harbor-redis`, `nextcloud-redis-replicas`, `stalwart`, `stalwart-postgres`) plus 8 single-replica Deployments — all backed by Hetzner-CSI RWO volumes pinned to the fsn1 datacenter. A one-shot `kubectl drain` reschedules >13 pods simultaneously → guaranteed kine cascade.
+- Rebooting cp-1 removes the entire kube-apiserver for the reboot window (~90–180 s expected). There is no fallback control-plane.
+
+The four kine thundering-herd guardrails from [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md) still apply here. This procedure implements them for cp-1.
+
+---
+
+## Scope
+
+**In scope**
+
+- Add a durable ≥ 2 GiB swapfile on cp-1 (idempotent one-off — `--add-swap`).
+- Pre-drain redistribution of stateful pods off cp-1, one at a time.
+- k3s SQLite datastore snapshot as the restore point.
+- `apt-get update/upgrade/dist-upgrade/autoremove` on cp-1.
+- Controlled reboot with external liveness monitoring.
+- Uncordon + cluster health verify.
+
+**Out of scope — do NOT do here**
+
+- Any change to `/etc/rancher/k3s/*`, `/etc/systemd/system/k3s*.service*`, or the k3s binary version. k3s upgrades go through system-upgrade-controller (see `K3S_OPERATIONS.md`).
+- Any change to manifests under `apps/`, `infrastructure/`, or applied via ArgoCD.
+- Deleting PVs / PVCs. StatefulSets that get rescheduled off cp-1 stay on their new node — do NOT try to move them back.
+- Fixing application-level problems ("harbor-core is CrashLoopBackOff after apt") — that's an app problem, escalate.
+- Rebuilding the node. If cp-1 does not return after reboot, escalate; the rebuild path is `ADD_WORKER_NODE.md` plus board approval, not this document.
+
+---
+
+## Approval gates
+
+This document has **two** independent gates. Neither happens without explicit board approval on the corresponding Paperclip issue:
+
+1. **Add swap (Phase A).** Non-invasive, non-state-mutating, kubelet already runs with `failSwapOn=false`. Requires board approval because it modifies the CP node.
+2. **Full OS update (Phases B–F).** Requires board approval because it drains cp-1 and reboots the sole api-server. Do NOT execute without an explicit `request_board_approval` acceptance on the execution ticket.
+
+Both gates are independent — swap can (and should) be added first, in a quiet window, before the full update is scheduled.
+
+---
+
+## Current cp-1 workload (snapshot 2026-08-16)
+
+Re-derive live before any execution — this table is a design reference only.
+
+| Kind | Namespace/Name | Notes |
+|------|-----------------|-------|
+| StatefulSet | `harbor/harbor-database-0` | Harbor Postgres, RWO 10 GiB fsn1 |
+| StatefulSet | `harbor/harbor-redis-0` | Harbor cache, RWO 10 GiB fsn1 |
+| StatefulSet | `nextcloud/nextcloud-redis-replicas-0` | Nextcloud Redis replica, RWO 10 GiB fsn1 |
+| StatefulSet | `stalwart/stalwart-0` | Mail server, RWO 20 GiB fsn1 — **critical, move last** |
+| StatefulSet | `stalwart/stalwart-postgres-0` | Mail metadata DB, RWO 10 GiB fsn1 |
+| Deployment | `harbor/harbor-core` | Registry frontend, single replica |
+| Deployment | `harbor/harbor-jobservice` | Registry job worker, single replica |
+| Deployment | `monitoring/alertmanager` | Alertmanager, single replica |
+| Deployment | `monitoring/loki` | Loki (single-binary), single replica |
+| Deployment | `monitoring/prometheus` | Prometheus, single replica |
+| Deployment | `observability/blackbox-exporter-*` | Blackbox exporter |
+| Deployment | `opencloud/tika` | Apache Tika, single replica |
+| Deployment | `passbolt/passbolt` | Passbolt web, single replica |
+| DaemonSet | `kube-system/hcloud-csi-node-*` | Ignored by drain |
+| DaemonSet | `kube-system/svclb-*` | Ignored by drain |
+| DaemonSet | `observability/*-node-exporter-*` | Ignored by drain |
+| DaemonSet | `observability/loki-stack-promtail-*` | Ignored by drain |
+
+**Absorption capacity (fsn1 workers only — nbg1 worker-4 cannot receive fsn1 RWO volumes):**
+
+- `k3s-worker-5`: ~3.0 GiB free, 6 pods scheduled — primary target for the two heaviest stateful pods.
+- `k3s-worker-2`: ~2.0 GiB free, 14 pods.
+- `k3s-worker-3`: ~2.3 GiB free, 15 pods, 2 sts.
+- `k3s-worker-1`: ~1.9 GiB free, 23 pods, 6 sts — **avoid piling more onto this one**.
+
+---
+
+## Phase A — Add swap (one-off, idempotent)
+
+**Goal:** eliminate the "3.7 GiB, no swap" underlying constraint before we ever try to drain cp-1.
+
+**Preconditions:**
+
+- kubelet in this k3s already runs with `failSwapOn=false` (confirmed via `/api/v1/nodes/k3s-cp-1/proxy/configz`) — enabling swap does NOT break the kubelet.
+- cp-1 `/` has ≥ 10 GiB free (currently 21 GiB free of 75 GiB).
+- Board approval on the swap-add ticket.
+
+**Sizing:** default **4 GiB** swap. Rationale — cp-1 baseline (k3s + hosted apps) already sits at ~3.1 GiB used; 4 GiB swap gives us headroom for the drain-eviction transient without inflating disk usage past ~10 % of `/`. Minimum acceptable per guardrail: 2 GiB.
+
+**Location:** `/swapfile` (root filesystem). Not a separate partition — reversible, no LVM changes, no ext4/xfs migration.
+
+**Steps (encoded in `update-cp-1.sh --add-swap`):**
+
+```bash
+# On cp-1 as root:
+SWAPFILE=/swapfile
+SIZE_MB=4096
+
+# Idempotency: skip if a swapfile of the target size already exists and is on.
+if swapon --show=NAME | grep -qx "$SWAPFILE"; then
+  echo "swap already on at $SWAPFILE"
+  exit 0
+fi
+
+# Create the file with fallocate; fall back to dd for filesystems without fallocate support.
+fallocate -l "${SIZE_MB}M" "$SWAPFILE" || dd if=/dev/zero of="$SWAPFILE" bs=1M count="$SIZE_MB" status=progress
+chmod 600 "$SWAPFILE"
+mkswap "$SWAPFILE"
+swapon "$SWAPFILE"
+
+# Persist across reboot. Guard against duplicate fstab entries.
+grep -q "^$SWAPFILE " /etc/fstab || echo "$SWAPFILE none swap sw 0 0" >> /etc/fstab
+
+# Moderate swappiness — we want swap as a safety net, not aggressive paging.
+sysctl -w vm.swappiness=10
+grep -q '^vm.swappiness' /etc/sysctl.d/99-k3s-swap.conf 2>/dev/null || {
+  echo 'vm.swappiness=10' > /etc/sysctl.d/99-k3s-swap.conf
+}
+
+# Verify.
+free -h
+swapon --show
+```
+
+**Rollback for Phase A:** `swapoff /swapfile && rm /swapfile` and remove the fstab line. This is safe at any time — swap is a soft resource.
+
+**Verification after Phase A:**
+
+- `free -h` shows `Swap: 4.0Gi` used ≈ 0.
+- `swapon --show` shows `/swapfile 4G`.
+- `sysctl vm.swappiness` returns `10`.
+- kubelet still Ready (`kubectl get node k3s-cp-1`).
+- No new `MemoryPressure` condition.
+
+---
+
+## Phase B — Preflight for the full OS update
+
+Everything from here on runs from the CTO's operator machine (not from cp-1 itself, because we lose kubectl during the reboot). SSH to cp-1 is fine — kubectl calls issued from cp-1 during the pre-drain phase are fine and get logged into `/tmp/os-update-cp-1-<ts>.log`.
+
+**Log directory contract** (same as `update-node.sh`): every command's stdout+stderr goes to `/tmp/os-update-cp-1-<UTC-timestamp>.log` on cp-1. Attach that log to the Paperclip execution ticket at the end.
+
+Run `update-cp-1.sh --preflight`:
+
+1. Cluster is currently healthy: `cluster-health.sh` returns 0.
+2. All 5 fsn1 workers are `Ready` (worker-1, worker-2, worker-3, worker-5, cp-1). worker-4 is nbg1 and irrelevant here.
+3. Each fsn1 worker has ≥ 500 MiB `MemAvailable`.
+4. `time kubectl get nodes` returns in ≤ 2 s (kine is not already stressed).
+5. cp-1 has swap on — abort if `free -h` shows `Swap: 0B`.
+6. **k3s datastore snapshot.** cp-1 runs k3s in embedded-SQLite mode (`--datastore-endpoint` is unset). `k3s etcd-snapshot save --name pre-cp1-os-update-<UTC-date>` produces a copy of the SQLite file under `/var/lib/rancher/k3s/server/db/snapshots/`. This is our restore point. Snapshots older than 30 days are pruned in Phase F.
+
+If any preflight check fails: STOP. Do not proceed. Do not add pods to cp-1 to "rebalance later" — that's a separate task.
+
+---
+
+## Phase C — Move stateful pods off cp-1 (batched)
+
+**Rule** ([[k3s-drain-kine-thundering-herd]] guardrail #1): no drain that reschedules > 3 StatefulSets at once. So we DO NOT `kubectl drain k3s-cp-1` while stateful pods still live on it. We move them one at a time first.
+
+**Cordon cp-1 immediately.** Cordon only prevents *new* scheduling — existing pods stay put. Cordoning first ensures that when we delete a stateful pod, the StatefulSet controller cannot re-create it back on cp-1.
+
+```bash
+kubectl cordon k3s-cp-1
+```
+
+**Order of eviction (idle → active, tiny → heavy):**
+
+| # | Pod | Why in this position |
+|---|-----|----------------------|
+| 1 | `harbor/harbor-redis-0` | Idle cache, cold restart is instant, low write pressure |
+| 2 | `nextcloud/nextcloud-redis-replicas-0` | Replica-1 of a Redis replicaset — non-primary, safe to bounce |
+| 3 | `harbor/harbor-database-0` | Registry Postgres, mostly idle (registry pulls, not writes) |
+| 4 | `stalwart/stalwart-postgres-0` | Mail metadata DB — active but recoverable; move before the mail server itself |
+| 5 | `stalwart/stalwart-0` | Mail server, most important. Save for last so mail keeps flowing until the very end |
+
+For each pod:
+
+```bash
+# 1. Delete the pod — StatefulSet controller will re-create it on a schedulable fsn1 worker.
+kubectl -n "$NS" delete pod "$POD" --wait=false
+
+# 2. Wait for the *new* pod (same name — StatefulSets keep identities) to be scheduled
+#    somewhere OTHER than cp-1 and reach Ready=True.
+deadline=$(( $(date +%s) + 300 ))
+while [ $(date +%s) -lt $deadline ]; do
+  new_node=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.spec.nodeName}' 2>/dev/null || true)
+  ready=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)
+  if [ -n "$new_node" ] && [ "$new_node" != "k3s-cp-1" ] && [ "$ready" = "True" ]; then
+    echo "  $POD -> $new_node OK"
+    break
+  fi
+  sleep 5
+done
+[ "$ready" = "True" ] || die "pod $NS/$POD did not reach Ready on a non-cp-1 node in 300s — HALT"
+
+# 3. Kine health probe — abort the batch if the api server is getting slow.
+t=$( { time kubectl get nodes >/dev/null; } 2>&1 | awk '/real/{print $2}' )
+# t is like "0m1.234s" — if the m field is >0 or the s field is >5, halt.
+# See update-cp-1.sh for the exact parse; > 5 s -> halt the whole cycle.
+
+# 4. Free-memory probe on cp-1.
+avail=$(ssh root@cp-1 "free -m | awk '/Mem:/{print \$7}'")
+[ "$avail" -lt 200 ] && die "cp-1 MemAvailable dropped below 200 MiB — HALT"
+
+# 5. Settle pause before the next eviction — kine needs to catch up on the write
+#    burst from the just-attached PV and the just-scheduled pod.
+sleep 60
+```
+
+**Halt conditions during Phase C** (any one → STOP, do NOT proceed to Phase D):
+
+- `kubectl get nodes` takes > 5 s.
+- cp-1 `MemAvailable` drops below 200 MiB.
+- Any target worker enters `MemoryPressure=True` or `DiskPressure=True`.
+- Any stateful pod does not reach Ready on a new node within 5 minutes (may indicate PV-attach or PDB issue).
+
+If we halt: leave cp-1 **cordoned** but do not reboot. The pods that have already moved stay where they are; the pods that haven't will still be on cp-1. Escalate on the execution ticket with `blocked` and name the halt condition.
+
+---
+
+## Phase D — Drain remaining pods + apt
+
+Once all 5 stateful pods are off cp-1 and cluster health is green:
+
+```bash
+# Drain everything else — only Deployment pods left (single-replica each,
+# no PV attach, so reschedule is fast).
+kubectl drain k3s-cp-1 \
+  --ignore-daemonsets \
+  --delete-emptydir-data \
+  --timeout="${DRAIN_TIMEOUT_SECONDS:-600}s"
+```
+
+If drain reports a PDB block: do NOT `--force`. Uncordon cp-1, mark the run as `blocked` on the PDB, and escalate. This is a design bug in the workload's PDB — fix it separately.
+
+**apt on cp-1** (mirrors `update-node.sh` step 3; `update-cp-1.sh --apt` runs this via `ssh root@cp-1 bash -s`):
+
+```bash
+export DEBIAN_FRONTEND=noninteractive
+APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
+uname -r > /root/pre-apt-kernel   # for rollback reference
+dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv
+
+if dpkg --audit | grep -qE .; then dpkg --configure -a || true; fi
+apt-get update
+apt-get $APT_OPTS upgrade || { apt-get $APT_OPTS -f install; apt-get $APT_OPTS upgrade; }
+apt-get $APT_OPTS dist-upgrade
+apt-get $APT_OPTS autoremove --purge
+apt-get clean
+[ -f /var/run/reboot-required ] && echo REBOOT_REQUIRED=yes || echo REBOOT_REQUIRED=no
+```
+
+`pre-apt-kernel` and `pre-apt-packages.tsv` are the local rollback references — see Rollback below.
+
+---
+
+## Phase E — Reboot handling (api-server unavailability)
+
+**Duration expectation:** 90–180 s of api-server unavailability. The operator machine will get `Unable to connect to the server` from kubectl during this window — that is expected, not an alarm.
+
+**Escalation trigger:** api-server not back on `/livez` after **10 minutes** → escalate. First check Hetzner console via `hcloud server describe k3s-cp-1` for boot state; if kernel-panic / initramfs, use grub previous-kernel path (see Rollback). Do NOT rebuild the node.
+
+**Steps (executed by `update-cp-1.sh --reboot`):**
+
+```bash
+# 1. Tell cp-1 to reboot. This SSH will hang up mid-command — that's fine.
+ssh $SSH_OPTS root@cp-1 'systemctl reboot' || true
+
+# 2. Deliberate 15 s pause — SSH needs to actually drop, don't race the poll.
+sleep 15
+
+# 3. Poll the api-server livez from the OPERATOR machine (not from cp-1).
+#    Using --insecure (`-k`) because the server cert is self-signed by k3s.
+deadline=$(( $(date +%s) + ${REBOOT_MAX_WAIT_SECONDS:-600} ))
+while [ $(date +%s) -lt $deadline ]; do
+  code=$(curl -sk -o /dev/null -w '%{http_code}' https://178.105.17.239:6443/livez 2>/dev/null || echo 000)
+  if [ "$code" = "200" ]; then
+    echo "  api-server /livez=200"
+    break
+  fi
+  sleep 5
+done
+[ "$code" = "200" ] || die "api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate; check hcloud console"
+
+# 4. Wait for kubelet Ready on cp-1 from the api-server view.
+deadline=$(( $(date +%s) + 300 ))
+while [ $(date +%s) -lt $deadline ]; do
+  ready=$(kubectl get node k3s-cp-1 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
+  [ "$ready" = "True" ] && break
+  sleep 5
+done
+[ "$ready" = "True" ] || die "kubelet on cp-1 never returned Ready — escalate (do NOT change k3s config)"
+```
+
+**What to do if the api-server takes 3–10 minutes:** it may be normal on this box (large SQLite → replay). Wait it out. Do NOT restart k3s to "help" — kine SQLite replay is stateful and interrupting it can corrupt the DB. See [[k3s-drain-kine-thundering-herd]] recovery: doing nothing is a valid path.
+
+**What to do if it takes > 10 minutes:** escalate; run the Hetzner-console diagnosis; if the console shows a bootable Ubuntu but the k3s service is failing, that's the boundary — this procedure stops here. Follow `K3S_OPERATIONS.md` for the k3s recovery path.
+
+---
+
+## Phase F — Uncordon + verify + finalize
+
+```bash
+# Uncordon cp-1. StatefulSet pods will NOT be moved back — that's correct
+# behaviour, they were moved for a reason. Rebalancing is a separate task.
+kubectl uncordon k3s-cp-1
+
+# Settle wait — kine needs to process the flood of "node schedulable again" events.
+sleep "${POST_UNCORDON_WAIT_SECONDS:-180}"
+
+# Full cluster health.
+RETRY_ON_TRANSIENT=1 infrastructure/scripts/os-update/cluster-health.sh
+
+# Print the apt history for audit.
+ssh root@cp-1 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
+
+# Prune snapshots older than 30 days.
+ssh root@cp-1 'find /var/lib/rancher/k3s/server/db/snapshots/ -type f -mtime +30 -name "pre-*" -print'
+# (list only — actual deletion is a manual review after the run)
+```
+
+Attach the `/tmp/os-update-cp-1-<ts>.log` and the ordered ledger of moved pods to the execution ticket. Mark the ticket `done` on green health, or `blocked` naming the specific residual issue on red.
+
+---
+
+## Rollback / recovery
+
+### If apt broke a package
+
+- On cp-1, `dpkg --audit` to find half-configured packages.
+- `dpkg --configure -a`, then `apt-get -f install`.
+- If a specific package broke and you know the previous version from `/root/pre-apt-packages.tsv`, `apt-get install <pkg>=<old-version>`.
+
+### If the new kernel does not boot
+
+- Hetzner cloud console: send `hcloud server request-console k3s-cp-1` → get VNC URL, watch boot.
+- If grub is up, select the previous-kernel entry. Ubuntu keeps ≥ 1 old kernel installed by default (we verified `6.8.0-137` current; the previous `6.8.0-124` was in use during the DEV-478 cycle).
+- Once booted on the old kernel, `apt-get remove` the broken kernel and pin the working one:
+  ```bash
+  apt-mark hold linux-image-<broken-version> linux-headers-<broken-version>
+  ```
+- Escalate on the ticket regardless — a kernel rollback is a follow-up investigation, not a "done" outcome.
+
+### If the node does not return at all
+
+- Do NOT `hcloud server delete`. Do NOT re-provision.
+- Escalate to the board with the Hetzner console output and the `/tmp/os-update-cp-1-*.log`.
+- The `--cluster-reset --cluster-reset-restore-path=<snapshot>` recovery path exists (see `K3S_OPERATIONS.md`) but requires board approval per stateful-service safety rules. The pre-flight snapshot from Phase B is the restore point.
+
+### If Phase C halted mid-eviction
+
+- cp-1 is cordoned, some stateful pods have moved, some haven't. Cluster is functional.
+- Uncordon cp-1 (`kubectl uncordon k3s-cp-1`) — StatefulSets that stayed on cp-1 keep running, moved ones stay where they went.
+- Do NOT proceed to Phase D. Open a follow-up ticket with the halt condition. Retry in the next maintenance window after fixing the halt condition.
+
+---
+
+## Automation entry points
+
+- `infrastructure/scripts/os-update/update-cp-1.sh` — this whole flow, with subcommands:
+  - `--add-swap` — Phase A only, idempotent, safe standalone.
+  - `--dry-run` — walk Phases B/C/D/E/F printing exactly what would be done without touching anything. Safe to run any time; used for design review.
+  - `--preflight` — Phase B only.
+  - `--drain-stateful` — Phase C only (cordon + batched stateful moves).
+  - `--apt` — Phase D apt commands only (requires cp-1 already fully drained).
+  - `--reboot` — Phase E only (requires apt already done).
+  - `--finalize` — Phase F only (uncordon + verify).
+  - `--run` — do all phases in order, with a confirmation prompt between each unless `ASSUME_YES=1`.
+
+The script follows the same log-dir contract as `update-node.sh` (`/tmp/os-update-cp-1-<UTC-timestamp>.log`).
+
+---
+
+## Change history
+
+| Date | Change | By |
+|------|--------|-----|
+| 2026-08-16 | Initial cp-1 update design (DEV-496) | CTO agent |

infrastructure/OS_UPDATE_PROCEDURE.md

@@ -40,7 +40,7 @@ ssh root@178.105.17.239 'kubectl get nodes -o wide'
 1. runner + workers that host no PVs (safest — lowest disruption)
 2. remaining workers
 3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
-4. **k3s-cp-1 last** — single control plane; the API server goes away during its reboot.
+4. **k3s-cp-1 excluded** — single control plane; has its own dedicated procedure and script. See `CP1_UPDATE_PROCEDURE.md` and `scripts/os-update/update-cp-1.sh`.
 
 **Concurrency:** exactly one node at a time. Never in parallel.
 
@@ -55,7 +55,7 @@ All four rules below MUST be observed on every DEV-478 fire. The reference imple
 1. **Pre-plan drain order for StatefulSets.** Before draining any worker, `kubectl get pods -n <ns> -o wide` against every namespace with StatefulSets and count how many will be evicted from the target node. If a single drain would evict **more than 3 StatefulSets at once**, redistribute first: cordon+delete individual StatefulSet pods one namespace at a time and wait for each reschedule to settle before draining the whole node.
 2. **cp-1 must have swap before finishing the cycle.** cp-1 has 0 swap. Add at least 2 GiB of swap on cp-1 before the cp-1 update step (and ideally before draining the last stateful-heavy worker). This is a one-off setup task; once done it is a durable capability.
 3. **Halt on kine slowness.** During any drain, keep a `time kubectl get nodes` running from cp-1. If it exceeds **5 s** in real time, halt the cycle immediately (uncordon the current node, do not proceed), verify cluster health, and escalate. The 5 s threshold is the leading indicator that kine has fallen behind and the taint-eviction cascade is about to start.
-4. **cp-1 update is a separate design task.** Draining cp-1's own pods (Stalwart, coredns, harbor-database if there, etc.) plus rebooting the single apiserver is the highest-risk step of the whole cycle. It MUST be planned and approved as a distinct issue before it runs; it is NOT covered by the standard `os-update.sh` cycle in its current form.
+4. **cp-1 update is a separate design task.** Draining cp-1's own pods (Stalwart, coredns, harbor-database if there, etc.) plus rebooting the single apiserver is the highest-risk step of the whole cycle. It MUST be planned and approved as a distinct issue before it runs; it is NOT covered by the standard `os-update.sh` cycle. See `CP1_UPDATE_PROCEDURE.md` and `scripts/os-update/update-cp-1.sh` (DEV-496).
 
 ---
 
@@ -237,10 +237,13 @@ Only proceed to the next node when ALL of the above are green. If not:
 
 ### 8. Control plane special handling
 
-`k3s-cp-1` is the single control plane node. During its reboot:
+`k3s-cp-1` is the single control plane node and is **NOT** updated by the weekly `os-update.sh` cycle — it has its own dedicated procedure and script. See `CP1_UPDATE_PROCEDURE.md` and `scripts/os-update/update-cp-1.sh`. Reasons:
-- The kube-apiserver is unavailable — kubectl commands from the operator machine will error out.
+
-- The `kubectl` waits in step 5/7 must run from a machine that is NOT the control plane, or must be scheduled after the control plane's SSH is back and `curl -k https://localhost:6443/healthz` returns `ok`.
+- cp-1 hosts many stateful pods (currently 5 StatefulSets + 8 single-replica Deployments) — a one-shot `kubectl drain` would trigger the kine/SQLite cascade documented in the "Kine thundering-herd guardrails" section above (DEV-495).
-- Skip the drain for DaemonSet pods on the control plane (`--ignore-daemonsets` covers that), but hosted apps that scheduled onto CP (rare — verify with `kubectl get pods -A --field-selector spec.nodeName=k3s-cp-1`) will be evicted.
+- cp-1 has 3.7 GiB RAM and (until Phase A of the cp-1 procedure is done) zero swap. The cp-1 procedure adds a durable 4 GiB swapfile before draining.
+- Rebooting cp-1 removes the entire kube-apiserver — the cp-1 procedure polls `/livez` from an external operator machine, not from cp-1 itself.
+
+`os-update.sh` will `SKIP` cp-1 and print a pointer to `CP1_UPDATE_PROCEDURE.md`.
 
 ---
 
@@ -301,8 +304,9 @@ If a node was skipped or errored → mark `blocked` with the unblock action, or
 
 ## Automation entry points
 
-- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything.
+- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything. Explicitly skips `k3s-cp-1`.
-- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry.
+- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry. Not for `k3s-cp-1`.
+- `infrastructure/scripts/os-update/update-cp-1.sh` — dedicated cp-1 update flow (add swap, batched stateful eviction, apt, reboot with external liveness monitor). See `CP1_UPDATE_PROCEDURE.md`.
 - `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure.
 
 Read the script sources for the exact behavior before running them. They mirror this procedure step for step.

infrastructure/scripts/os-update/os-update.sh

@@ -68,9 +68,11 @@ fi
 
 # --- 3. node ordering ---------------------------------------------------------
 # Ordering rule:
-#   - workers before control plane
+#   - workers only
 #   - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last
-#   - k3s-cp-1 always last
+#   - k3s-cp-1 is EXCLUDED and never updated by this script — see CP1_UPDATE_PROCEDURE.md
+#     and scripts/os-update/update-cp-1.sh. The rationale is the kine thundering-herd
+#     guardrails documented in OS_UPDATE_PROCEDURE.md (added after DEV-495).
 STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true)
 # If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers.
 CP_NAME="k3s-cp-1"
@@ -94,7 +96,11 @@ for n in $ALL_NODES; do
   fi
 done
 
-ORDER=("${workers[@]}" "${stalwart_workers[@]}" "$CP_NAME")
+ORDER=("${workers[@]}" "${stalwart_workers[@]}")
+# NOTE: cp-1 intentionally excluded. To update cp-1, run `scripts/os-update/update-cp-1.sh`.
+if printf '%s\n' "$ALL_NODES" | grep -qx "$CP_NAME"; then
+  log "[plan] EXCLUDING $CP_NAME — use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)"
+fi
 
 if [ -n "$ONLY" ]; then
   ORDER=("$ONLY")

infrastructure/scripts/os-update/update-cp-1.sh

@@ -0,0 +1,577 @@
+#!/bin/bash
+# update-cp-1.sh — controlled OS update for the k3s control-plane node.
+#
+# See ../CP1_UPDATE_PROCEDURE.md for the full design rationale.
+#
+# Usage:
+#   update-cp-1.sh --dry-run           # print what would be done, touch nothing
+#   update-cp-1.sh --add-swap          # Phase A only (idempotent, safe standalone)
+#   update-cp-1.sh --preflight         # Phase B only
+#   update-cp-1.sh --drain-stateful    # Phase C only  (cordon + batched sts moves)
+#   update-cp-1.sh --apt               # Phase D only  (requires cp-1 fully drained)
+#   update-cp-1.sh --reboot            # Phase E only  (requires --apt reported REBOOT_REQUIRED=yes)
+#   update-cp-1.sh --finalize          # Phase F only  (uncordon + verify)
+#   update-cp-1.sh --run               # all phases with a confirmation between each (or ASSUME_YES=1)
+#
+# Environment overrides:
+#   CP1_HOST                     default 178.105.17.239
+#   SWAP_SIZE_MB                 default 4096       (>=2048 required)
+#   SWAP_PATH                    default /swapfile
+#   DRAIN_TIMEOUT_SECONDS        default 600
+#   REBOOT_MAX_WAIT_SECONDS      default 600
+#   POST_UNCORDON_WAIT_SECONDS   default 180
+#   STATEFUL_SETTLE_SECONDS      default 60         (pause between batched sts moves)
+#   MIN_WORKER_MEM_MIB           default 500        (per-worker MemAvailable floor at preflight)
+#   MIN_CP1_MEM_MIB              default 200        (cp-1 MemAvailable floor mid-drain)
+#   MAX_KUBECTL_SECONDS          default 5          (kine-health guardrail)
+#   SSH_OPTS                     default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new"
+#   ASSUME_YES=1                 skip interactive confirmations in --run
+#   DRY_RUN=1                    do not execute state-mutating commands; log-only
+#
+# Log directory contract (matches update-node.sh):
+#   Every command's stdout+stderr is tee'd to /tmp/os-update-cp-1-<UTC-ts>.log on
+#   the operator machine AND to /tmp/os-update-cp-1-<UTC-ts>.log on cp-1 (via the
+#   ssh command wrappers). Attach both to the execution ticket at the end.
+#
+# NEVER touches k3s config, k3s services, containerd, or any manifest. Only
+# fixes it will attempt: apt/dpkg recovery on cp-1 (see Phase D). Any other
+# problem → escalate and STOP.
+
+set -euo pipefail
+
+# --------------------------------------------------------------------------- #
+# Config
+# --------------------------------------------------------------------------- #
+CP1_HOST="${CP1_HOST:-178.105.17.239}"
+SWAP_SIZE_MB="${SWAP_SIZE_MB:-4096}"
+SWAP_PATH="${SWAP_PATH:-/swapfile}"
+DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}"
+REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}"
+POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}"
+STATEFUL_SETTLE_SECONDS="${STATEFUL_SETTLE_SECONDS:-60}"
+MIN_WORKER_MEM_MIB="${MIN_WORKER_MEM_MIB:-500}"
+MIN_CP1_MEM_MIB="${MIN_CP1_MEM_MIB:-200}"
+MAX_KUBECTL_SECONDS="${MAX_KUBECTL_SECONDS:-5}"
+SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}"
+DRY_RUN="${DRY_RUN:-0}"
+ASSUME_YES="${ASSUME_YES:-0}"
+
+NODE="k3s-cp-1"
+TS="$(date -u +%Y%m%dT%H%M%SZ)"
+LOG_LOCAL="/tmp/os-update-cp-1-${TS}.log"
+LOG_REMOTE="/tmp/os-update-cp-1-${TS}.log"
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+HEALTH_SCRIPT="$SCRIPT_DIR/cluster-health.sh"
+
+# --------------------------------------------------------------------------- #
+# Stateful eviction ledger — order matters (idle -> active, small -> heavy).
+# --------------------------------------------------------------------------- #
+# The list is derived live from the cluster in preflight and stored to
+# /tmp/cp1-stateful-plan-<ts>.txt. This ORDER is the authoritative fallback.
+STATEFUL_ORDER=(
+  "harbor:harbor-redis-0"
+  "nextcloud:nextcloud-redis-replicas-0"
+  "harbor:harbor-database-0"
+  "stalwart:stalwart-postgres-0"
+  "stalwart:stalwart-0"
+)
+
+# --------------------------------------------------------------------------- #
+# Logging + safe-run helpers
+# --------------------------------------------------------------------------- #
+log()  { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOG_LOCAL"; }
+die()  { log "FATAL: $*"; exit 1; }
+warn() { log "WARN:  $*"; }
+
+# run/ssh_run/kubectl_run: honour DRY_RUN — print, do not execute state changes.
+run() {
+  if [ "$DRY_RUN" = "1" ]; then
+    log "DRY-RUN would exec: $*"
+    return 0
+  fi
+  log "exec: $*"
+  "$@" 2>&1 | tee -a "$LOG_LOCAL"
+}
+
+ssh_run() {
+  local target="root@${CP1_HOST}"
+  if [ "$DRY_RUN" = "1" ]; then
+    log "DRY-RUN would ssh $target: $*"
+    return 0
+  fi
+  log "ssh $target: $*"
+  ssh $SSH_OPTS "$target" "$@" 2>&1 | tee -a "$LOG_LOCAL"
+}
+
+# ssh_run_stdin: pipe a heredoc through bash -s on cp-1; used for multi-line remote
+# blocks (swap add, apt).
+ssh_run_stdin() {
+  local target="root@${CP1_HOST}"
+  if [ "$DRY_RUN" = "1" ]; then
+    log "DRY-RUN would ssh $target with stdin script:"
+    sed 's/^/    | /' | tee -a "$LOG_LOCAL"
+    return 0
+  fi
+  log "ssh $target (heredoc)"
+  ssh $SSH_OPTS "$target" "bash -s" 2>&1 | tee -a "$LOG_LOCAL"
+}
+
+confirm() {
+  local prompt="$1"
+  if [ "$ASSUME_YES" = "1" ]; then
+    log "confirm SKIPPED (ASSUME_YES=1): $prompt"
+    return 0
+  fi
+  echo -n "  >>> $prompt  Continue? [y/N] "
+  read -r a
+  case "$a" in y|Y|yes|YES) return 0 ;; *) die "aborted by operator" ;; esac
+}
+
+# --------------------------------------------------------------------------- #
+# Phase A — Add swap on cp-1 (idempotent)
+# --------------------------------------------------------------------------- #
+phase_add_swap() {
+  log "=== Phase A: add swap on $NODE ($SWAP_SIZE_MB MiB at $SWAP_PATH) ==="
+  [ "$SWAP_SIZE_MB" -ge 2048 ] || die "SWAP_SIZE_MB=$SWAP_SIZE_MB below 2048 MiB guardrail"
+
+  cat <<REMOTE | ssh_run_stdin
+set -euo pipefail
+SWAP_PATH="$SWAP_PATH"
+SIZE_MB="$SWAP_SIZE_MB"
+
+# Skip if a swapfile at this path is already active.
+if swapon --show=NAME 2>/dev/null | grep -qx "\$SWAP_PATH"; then
+  echo "swap already on at \$SWAP_PATH — skipping"
+  free -h
+  exit 0
+fi
+
+# Root filesystem free space check — abort if less than 2*swap free.
+avail_mb=\$(df -m --output=avail / | tail -1 | tr -d ' ')
+need_mb=\$(( SIZE_MB * 2 ))
+if [ "\$avail_mb" -lt "\$need_mb" ]; then
+  echo "ERROR: only \${avail_mb} MiB free on /, need \${need_mb} MiB (2x swap for safety)"
+  exit 1
+fi
+
+# Create swapfile. fallocate is fast; dd is the fallback.
+if ! fallocate -l "\${SIZE_MB}M" "\$SWAP_PATH" 2>/dev/null; then
+  dd if=/dev/zero of="\$SWAP_PATH" bs=1M count="\$SIZE_MB" status=progress
+fi
+chmod 600 "\$SWAP_PATH"
+mkswap "\$SWAP_PATH"
+swapon "\$SWAP_PATH"
+
+# Persist via fstab (dedup).
+if ! grep -q "^\$SWAP_PATH " /etc/fstab; then
+  echo "\$SWAP_PATH none swap sw 0 0" >> /etc/fstab
+fi
+
+# Moderate swappiness — swap as safety net, not aggressive paging.
+sysctl -w vm.swappiness=10
+if [ ! -f /etc/sysctl.d/99-k3s-swap.conf ] || ! grep -q '^vm.swappiness' /etc/sysctl.d/99-k3s-swap.conf; then
+  echo 'vm.swappiness=10' > /etc/sysctl.d/99-k3s-swap.conf
+fi
+
+echo "--- swap after ---"
+free -h
+swapon --show
+sysctl vm.swappiness
+REMOTE
+
+  # Verify from operator view.
+  if [ "$DRY_RUN" != "1" ]; then
+    log "verifying kubelet still Ready after swap add"
+    local ready
+    ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
+    [ "$ready" = "True" ] || die "kubelet on $NODE not Ready after swap add — halt"
+    log "kubelet Ready=True — Phase A complete"
+  fi
+}
+
+# --------------------------------------------------------------------------- #
+# Guardrail probes
+# --------------------------------------------------------------------------- #
+kubectl_latency_s() {
+  # returns integer seconds elapsed for `kubectl get nodes >/dev/null`.
+  local start end
+  start=$(date +%s)
+  kubectl get nodes >/dev/null 2>&1 || echo "kubectl-error" >&2
+  end=$(date +%s)
+  echo $(( end - start ))
+}
+
+cp1_mem_available_mib() {
+  # returns integer MiB `MemAvailable` on cp-1.
+  ssh $SSH_OPTS "root@$CP1_HOST" "awk '/^MemAvailable:/{printf \"%d\n\", \$2/1024}' /proc/meminfo" 2>/dev/null || echo 0
+}
+
+guard_kine_healthy() {
+  local s
+  s=$(kubectl_latency_s)
+  if [ "$s" -gt "$MAX_KUBECTL_SECONDS" ]; then
+    die "kubectl get nodes took ${s}s (>${MAX_KUBECTL_SECONDS}s) — kine cascade risk, HALT"
+  fi
+  log "  kine ok: kubectl get nodes took ${s}s"
+}
+
+guard_cp1_memory() {
+  local m
+  m=$(cp1_mem_available_mib)
+  if [ "$m" -lt "$MIN_CP1_MEM_MIB" ]; then
+    die "cp-1 MemAvailable=${m} MiB below ${MIN_CP1_MEM_MIB} MiB floor — HALT"
+  fi
+  log "  cp-1 mem ok: MemAvailable=${m} MiB"
+}
+
+# --------------------------------------------------------------------------- #
+# Phase B — Preflight
+# --------------------------------------------------------------------------- #
+phase_preflight() {
+  log "=== Phase B: preflight ==="
+
+  log "-- cluster health"
+  if [ "$DRY_RUN" != "1" ]; then
+    if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then
+      die "cluster is not healthy — refuse to start cp-1 update"
+    fi
+  else
+    log "DRY-RUN would run: $HEALTH_SCRIPT"
+  fi
+
+  log "-- swap on cp-1"
+  if [ "$DRY_RUN" != "1" ]; then
+    local swap_total
+    swap_total=$(ssh $SSH_OPTS "root@$CP1_HOST" "awk '/^SwapTotal:/{print \$2}' /proc/meminfo")
+    [ "${swap_total:-0}" -ge $((2 * 1024 * 1024)) ] \
+      || die "cp-1 SwapTotal=${swap_total} KiB below 2 GiB — run --add-swap first"
+    log "  cp-1 SwapTotal=$(( swap_total / 1024 )) MiB"
+  fi
+
+  log "-- fsn1 workers Ready and have >=${MIN_WORKER_MEM_MIB} MiB MemAvailable"
+  # Worker private IPs (worker-4 nbg1 excluded — cannot host fsn1 RWO PVs).
+  local ips=("10.42.1.2" "10.42.1.3" "10.42.1.5" "10.42.1.7")
+  local names=("k3s-worker-1" "k3s-worker-2" "k3s-worker-3" "k3s-worker-5")
+  if [ "$DRY_RUN" != "1" ]; then
+    for i in "${!ips[@]}"; do
+      local nm="${names[$i]}"
+      local ip="${ips[$i]}"
+      local ready
+      ready=$(kubectl get node "$nm" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
+      [ "$ready" = "True" ] || die "worker $nm not Ready (Ready=$ready)"
+      local mem
+      mem=$(ssh $SSH_OPTS -J "root@$CP1_HOST" "root@$ip" \
+        "awk '/^MemAvailable:/{printf \"%d\n\", \$2/1024}' /proc/meminfo" 2>/dev/null || echo 0)
+      [ "$mem" -ge "$MIN_WORKER_MEM_MIB" ] \
+        || die "worker $nm MemAvailable=${mem} MiB below ${MIN_WORKER_MEM_MIB} MiB — HALT"
+      log "  $nm ok: MemAvailable=${mem} MiB"
+    done
+  fi
+
+  log "-- kine latency probe"
+  if [ "$DRY_RUN" != "1" ]; then guard_kine_healthy; fi
+
+  log "-- k3s SQLite datastore snapshot"
+  # k3s etcd-snapshot in embedded-SQLite mode copies the sqlite file to
+  # /var/lib/rancher/k3s/server/db/snapshots/.
+  ssh_run "k3s etcd-snapshot save --name pre-cp1-os-update-${TS}"
+  ssh_run "ls -la /var/lib/rancher/k3s/server/db/snapshots/ | tail -10"
+
+  log "-- persist eviction plan"
+  if [ "$DRY_RUN" != "1" ]; then
+    local plan="/tmp/cp1-stateful-plan-${TS}.txt"
+    ssh $SSH_OPTS "root@$CP1_HOST" "cat > $plan" <<EOF
+# cp-1 stateful eviction plan generated at $TS
+# Order: idle -> active, small -> heavy. Do NOT reorder without design review.
+$(for e in "${STATEFUL_ORDER[@]}"; do echo "$e"; done)
+EOF
+    log "  wrote $plan on cp-1"
+  else
+    log "DRY-RUN would write /tmp/cp1-stateful-plan-${TS}.txt with STATEFUL_ORDER"
+  fi
+
+  log "=== Phase B: preflight OK ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Phase C — Cordon + move stateful pods off cp-1, one at a time
+# --------------------------------------------------------------------------- #
+phase_drain_stateful() {
+  log "=== Phase C: cordon + batched stateful move ==="
+
+  log "-- cordon $NODE (prevents rescheduled pods from landing back on cp-1)"
+  run kubectl cordon "$NODE"
+
+  local i=0
+  for entry in "${STATEFUL_ORDER[@]}"; do
+    i=$((i + 1))
+    local ns="${entry%%:*}"
+    local pod="${entry##*:}"
+
+    log "--- [${i}/${#STATEFUL_ORDER[@]}] moving $ns/$pod"
+
+    # Confirm the pod actually IS on cp-1 before touching it.
+    if [ "$DRY_RUN" != "1" ]; then
+      local on
+      on=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null || echo "")
+      if [ -z "$on" ]; then
+        log "  $ns/$pod not found — sts may have changed; skipping"
+        continue
+      fi
+      if [ "$on" != "$NODE" ]; then
+        log "  $ns/$pod already on $on (not cp-1) — skipping"
+        continue
+      fi
+    fi
+
+    # Delete the pod. StatefulSet controller will re-create it; scheduler will
+    # pick a fsn1 worker because cp-1 is cordoned and worker-4 is nbg1.
+    run kubectl -n "$ns" delete pod "$pod" --wait=false
+
+    if [ "$DRY_RUN" = "1" ]; then
+      log "  DRY-RUN skipping wait-for-ready"
+      continue
+    fi
+
+    log "  waiting for $ns/$pod to be Ready on a non-cp-1 node (max 300s)"
+    local deadline=$(( $(date +%s) + 300 ))
+    local new_node="" ready=""
+    while [ $(date +%s) -lt $deadline ]; do
+      new_node=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null || echo "")
+      ready=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "")
+      if [ -n "$new_node" ] && [ "$new_node" != "$NODE" ] && [ "$ready" = "True" ]; then
+        log "  $ns/$pod -> $new_node OK"
+        break
+      fi
+      sleep 5
+    done
+    [ "$ready" = "True" ] && [ "$new_node" != "$NODE" ] \
+      || die "$ns/$pod did not reach Ready on a non-cp-1 node in 300s — HALT (leave cp-1 cordoned)"
+
+    log "  guardrails after $ns/$pod"
+    guard_kine_healthy
+    guard_cp1_memory
+
+    if [ "$i" -lt "${#STATEFUL_ORDER[@]}" ]; then
+      log "  settle pause ${STATEFUL_SETTLE_SECONDS}s"
+      sleep "$STATEFUL_SETTLE_SECONDS"
+    fi
+  done
+
+  log "=== Phase C: all stateful pods evicted from cp-1 ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Phase D — Drain remaining pods + apt on cp-1
+# --------------------------------------------------------------------------- #
+phase_apt() {
+  log "=== Phase D: drain (remaining deployments) + apt on $NODE ==="
+
+  # Guardrail: refuse to run if any StatefulSet pod is still on cp-1.
+  if [ "$DRY_RUN" != "1" ]; then
+    local sts_on_cp1
+    sts_on_cp1=$(kubectl get pods -A -o json --field-selector spec.nodeName=$NODE \
+      | jq -r '.items[] | select(.metadata.ownerReferences[0].kind=="StatefulSet") | "\(.metadata.namespace)/\(.metadata.name)"' \
+      | wc -l)
+    [ "$sts_on_cp1" -eq 0 ] \
+      || die "$sts_on_cp1 StatefulSet pod(s) still on cp-1 — Phase C incomplete; refuse Phase D"
+  fi
+
+  log "-- drain (timeout ${DRAIN_TIMEOUT_SECONDS}s)"
+  set +e
+  if [ "$DRY_RUN" = "1" ]; then
+    log "DRY-RUN would run: kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --timeout=${DRAIN_TIMEOUT_SECONDS}s"
+    local rc=0
+  else
+    kubectl drain "$NODE" \
+      --ignore-daemonsets \
+      --delete-emptydir-data \
+      --timeout="${DRAIN_TIMEOUT_SECONDS}s" 2>&1 | tee -a "$LOG_LOCAL"
+    local rc=${PIPESTATUS[0]}
+  fi
+  set -e
+
+  if [ "$rc" -ne 0 ]; then
+    log "drain FAILED (rc=$rc). Never force. Uncordoning."
+    run kubectl uncordon "$NODE"
+    die "drain failed on $NODE — investigate PDB / orphan pods; do NOT proceed"
+  fi
+
+  log "-- apt on cp-1"
+  cat <<'REMOTE' | ssh_run_stdin
+set -euo pipefail
+export DEBIAN_FRONTEND=noninteractive
+APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
+
+# Rollback references (Phase-A didn't need this, but Phase D does).
+uname -r > /root/pre-apt-kernel
+dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv
+echo "pre-apt kernel: $(cat /root/pre-apt-kernel)"
+
+# Recover from any half-finished dpkg state before touching apt.
+if dpkg --audit | grep -qE .; then
+  echo "dpkg audit reported issues, running dpkg --configure -a"
+  dpkg --configure -a || true
+fi
+
+apt-get update
+
+if ! apt-get $APT_OPTS upgrade; then
+  echo "upgrade failed, attempting apt-get -f install"
+  apt-get $APT_OPTS -f install
+  apt-get $APT_OPTS upgrade
+fi
+
+apt-get $APT_OPTS dist-upgrade
+apt-get $APT_OPTS autoremove --purge
+apt-get clean
+
+if [ -f /var/run/reboot-required ]; then
+  echo "REBOOT_REQUIRED=yes"
+  echo "REBOOT_REASON<<EOF"
+  cat /var/run/reboot-required.pkgs 2>/dev/null || echo "(no package list)"
+  echo "EOF"
+else
+  echo "REBOOT_REQUIRED=no"
+fi
+REMOTE
+
+  log "=== Phase D: apt complete (check REBOOT_REQUIRED in the log) ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Phase E — Reboot cp-1 and wait for api-server /livez
+# --------------------------------------------------------------------------- #
+phase_reboot() {
+  log "=== Phase E: reboot $NODE ==="
+
+  # Tell cp-1 to reboot. The ssh command will hang up mid-command — that's fine.
+  if [ "$DRY_RUN" != "1" ]; then
+    log "issuing 'systemctl reboot' on $NODE (ssh will drop; expected)"
+    ssh $SSH_OPTS "root@$CP1_HOST" 'systemctl reboot' 2>&1 | tee -a "$LOG_LOCAL" || true
+    log "waiting 15s for ssh to fully drop before polling api-server"
+    sleep 15
+  else
+    log "DRY-RUN would ssh root@$CP1_HOST 'systemctl reboot'"
+  fi
+
+  log "-- poll api-server /livez (timeout ${REBOOT_MAX_WAIT_SECONDS}s)"
+  if [ "$DRY_RUN" != "1" ]; then
+    local deadline=$(( $(date +%s) + REBOOT_MAX_WAIT_SECONDS ))
+    local code=000
+    while [ $(date +%s) -lt $deadline ]; do
+      code=$(curl -sk -o /dev/null -w '%{http_code}' "https://$CP1_HOST:6443/livez" 2>/dev/null || echo 000)
+      if [ "$code" = "200" ]; then
+        log "  api-server /livez=200 (elapsed $(( REBOOT_MAX_WAIT_SECONDS - (deadline - $(date +%s)) ))s)"
+        break
+      fi
+      sleep 5
+    done
+    [ "$code" = "200" ] || die "api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate; check 'hcloud server describe k3s-cp-1' and Hetzner console"
+  fi
+
+  log "-- wait for kubelet Ready on $NODE (max 300s)"
+  if [ "$DRY_RUN" != "1" ]; then
+    local deadline=$(( $(date +%s) + 300 ))
+    local ready=Unknown
+    while [ $(date +%s) -lt $deadline ]; do
+      ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
+      [ "$ready" = "True" ] && break
+      sleep 5
+    done
+    [ "$ready" = "True" ] || die "kubelet on $NODE never returned Ready — escalate (do NOT change k3s config)"
+    log "  kubelet Ready=True"
+  fi
+
+  log "=== Phase E: cp-1 is back ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Phase F — Uncordon + verify + finalize
+# --------------------------------------------------------------------------- #
+phase_finalize() {
+  log "=== Phase F: uncordon + verify ==="
+
+  run kubectl uncordon "$NODE"
+
+  log "-- settle wait ${POST_UNCORDON_WAIT_SECONDS}s"
+  [ "$DRY_RUN" = "1" ] || sleep "$POST_UNCORDON_WAIT_SECONDS"
+
+  log "-- cluster health"
+  if [ "$DRY_RUN" != "1" ]; then
+    if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then
+      die "cluster health failed after cp-1 update — escalate, do NOT touch k3s"
+    fi
+  fi
+
+  log "-- apt history summary (audit)"
+  ssh_run 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
+
+  log "-- old snapshots (>30d) — listing only, review manually"
+  ssh_run 'find /var/lib/rancher/k3s/server/db/snapshots/ -type f -mtime +30 -name "pre-*" -print 2>/dev/null || true'
+
+  log "=== cp-1 OS update complete — attach $LOG_LOCAL to the execution ticket ==="
+}
+
+# --------------------------------------------------------------------------- #
+# --run — orchestrate all phases with confirmations
+# --------------------------------------------------------------------------- #
+phase_run_all() {
+  log "=== full cp-1 update run (log: $LOG_LOCAL) ==="
+
+  confirm "Phase A (add swap) — proceed?"
+  phase_add_swap
+
+  confirm "Phase B (preflight) — proceed?"
+  phase_preflight
+
+  confirm "Phase C (cordon + batched stateful move) — proceed?"
+  phase_drain_stateful
+
+  confirm "Phase D (drain remaining + apt) — proceed?"
+  phase_apt
+
+  confirm "Phase E (reboot cp-1; api-server unavailable ~90-180s) — proceed?"
+  phase_reboot
+
+  confirm "Phase F (uncordon + verify) — proceed?"
+  phase_finalize
+
+  log "=== FULL RUN COMPLETE ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Arg parse
+# --------------------------------------------------------------------------- #
+usage() { grep -E '^# ' "$0" | sed 's/^# \{0,1\}//'; exit 2; }
+
+[ $# -ge 1 ] || usage
+
+# Init the local log file up front so tee always has a target.
+: > "$LOG_LOCAL"
+log "update-cp-1.sh started (DRY_RUN=$DRY_RUN)"
+log "log file: $LOG_LOCAL"
+
+case "$1" in
+  --dry-run)
+    DRY_RUN=1
+    export DRY_RUN
+    log "DRY_RUN=1 — walking Phases A..F without touching state"
+    phase_add_swap
+    phase_preflight
+    phase_drain_stateful
+    phase_apt
+    phase_reboot
+    phase_finalize
+    ;;
+  --add-swap)       phase_add_swap ;;
+  --preflight)      phase_preflight ;;
+  --drain-stateful) phase_drain_stateful ;;
+  --apt)            phase_apt ;;
+  --reboot)         phase_reboot ;;
+  --finalize)       phase_finalize ;;
+  --run)            phase_run_all ;;
+  -h|--help)        usage ;;
+  *)                echo "unknown arg: $1" >&2; usage ;;
+esac