stack.basicstack.de/infrastructure/CP1_UPDATE_PROCEDURE.md

# k3s-cp-1 OS Update Procedure

**Purpose:** apply the weekly Ubuntu OS updates to `k3s-cp-1`, the **sole** control-plane node of the k3s cluster, without triggering the kine/SQLite thundering-herd cascade documented in [DEV-495](/DEV/issues/DEV-495) and without losing api-server access longer than a normal reboot.

**Audience:** the CTO agent, or an operator with root SSH to the cluster. Execution is board-approval-gated — see `## Approval gates` below.

**Why cp-1 is not covered by the standard [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md):**

- cp-1 has 3.7 GiB RAM and **zero swap** — during the 2026-08-16 worker-3 drain, `MemAvailable` dropped to 43 MiB while kine backed up on writes. cp-1's own drain evicts **more** state at once than any worker drain.
- cp-1 currently hosts 5 StatefulSets (`harbor-database`, `harbor-redis`, `nextcloud-redis-replicas`, `stalwart`, `stalwart-postgres`) plus 8 single-replica Deployments — all backed by Hetzner-CSI RWO volumes pinned to the fsn1 datacenter. A one-shot `kubectl drain` reschedules >13 pods simultaneously → guaranteed kine cascade.
- Rebooting cp-1 removes the entire kube-apiserver for the reboot window (~90–180 s expected). There is no fallback control-plane.

The four kine thundering-herd guardrails from [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md) still apply here. This procedure implements them for cp-1.

---

## Scope

**In scope**

- Add a durable ≥ 2 GiB swapfile on cp-1 (idempotent one-off — `--add-swap`).
- Pre-drain redistribution of stateful pods off cp-1, one at a time.
- k3s SQLite datastore snapshot as the restore point.
- `apt-get update/upgrade/dist-upgrade/autoremove` on cp-1.
- Controlled reboot with external liveness monitoring.
- Uncordon + cluster health verify.

**Out of scope — do NOT do here**

- Any change to `/etc/rancher/k3s/*`, `/etc/systemd/system/k3s*.service*`, or the k3s binary version. k3s upgrades go through system-upgrade-controller (see `K3S_OPERATIONS.md`).
- Any change to manifests under `apps/`, `infrastructure/`, or applied via ArgoCD.
- Deleting PVs / PVCs. StatefulSets that get rescheduled off cp-1 stay on their new node — do NOT try to move them back.
- Fixing application-level problems ("harbor-core is CrashLoopBackOff after apt") — that's an app problem, escalate.
- Rebuilding the node. If cp-1 does not return after reboot, escalate; the rebuild path is `ADD_WORKER_NODE.md` plus board approval, not this document.

---

## Approval gates

This document has **two** independent gates. Neither happens without explicit board approval on the corresponding Paperclip issue:

1. **Add swap (Phase A).** Non-invasive, non-state-mutating, kubelet already runs with `failSwapOn=false`. Requires board approval because it modifies the CP node.
2. **Full OS update (Phases B–F).** Requires board approval because it drains cp-1 and reboots the sole api-server. Do NOT execute without an explicit `request_board_approval` acceptance on the execution ticket.

Both gates are independent — swap can (and should) be added first, in a quiet window, before the full update is scheduled.

---

## Current cp-1 workload (snapshot 2026-08-16)

Re-derive live before any execution — this table is a design reference only.

| Kind | Namespace/Name | Notes |
|------|-----------------|-------|
| StatefulSet | `harbor/harbor-database-0` | Harbor Postgres, RWO 10 GiB fsn1 |
| StatefulSet | `harbor/harbor-redis-0` | Harbor cache, RWO 10 GiB fsn1 |
| StatefulSet | `nextcloud/nextcloud-redis-replicas-0` | Nextcloud Redis replica, RWO 10 GiB fsn1 |
| StatefulSet | `stalwart/stalwart-0` | Mail server, RWO 20 GiB fsn1 — **critical, move last** |
| StatefulSet | `stalwart/stalwart-postgres-0` | Mail metadata DB, RWO 10 GiB fsn1 |
| Deployment | `harbor/harbor-core` | Registry frontend, single replica |
| Deployment | `harbor/harbor-jobservice` | Registry job worker, single replica |
| Deployment | `monitoring/alertmanager` | Alertmanager, single replica |
| Deployment | `monitoring/loki` | Loki (single-binary), single replica |
| Deployment | `monitoring/prometheus` | Prometheus, single replica |
| Deployment | `observability/blackbox-exporter-*` | Blackbox exporter |
| Deployment | `opencloud/tika` | Apache Tika, single replica |
| Deployment | `passbolt/passbolt` | Passbolt web, single replica |
| DaemonSet | `kube-system/hcloud-csi-node-*` | Ignored by drain |
| DaemonSet | `kube-system/svclb-*` | Ignored by drain |
| DaemonSet | `observability/*-node-exporter-*` | Ignored by drain |
| DaemonSet | `observability/loki-stack-promtail-*` | Ignored by drain |

**Absorption capacity (fsn1 workers only — nbg1 worker-4 cannot receive fsn1 RWO volumes):**

- `k3s-worker-5`: ~3.0 GiB free, 6 pods scheduled — primary target for the two heaviest stateful pods.
- `k3s-worker-2`: ~2.0 GiB free, 14 pods.
- `k3s-worker-3`: ~2.3 GiB free, 15 pods, 2 sts.
- `k3s-worker-1`: ~1.9 GiB free, 23 pods, 6 sts — **avoid piling more onto this one**.

---

## Phase A — Add swap (one-off, idempotent)

**Goal:** eliminate the "3.7 GiB, no swap" underlying constraint before we ever try to drain cp-1.

**Preconditions:**

- kubelet in this k3s already runs with `failSwapOn=false` (confirmed via `/api/v1/nodes/k3s-cp-1/proxy/configz`) — enabling swap does NOT break the kubelet.
- cp-1 `/` has ≥ 10 GiB free (currently 21 GiB free of 75 GiB).
- Board approval on the swap-add ticket.

**Sizing:** default **4 GiB** swap. Rationale — cp-1 baseline (k3s + hosted apps) already sits at ~3.1 GiB used; 4 GiB swap gives us headroom for the drain-eviction transient without inflating disk usage past ~10 % of `/`. Minimum acceptable per guardrail: 2 GiB.

**Location:** `/swapfile` (root filesystem). Not a separate partition — reversible, no LVM changes, no ext4/xfs migration.

**Steps (encoded in `update-cp-1.sh --add-swap`):**

```bash
# On cp-1 as root:
SWAPFILE=/swapfile
SIZE_MB=4096

# Idempotency: skip if a swapfile of the target size already exists and is on.
if swapon --show=NAME | grep -qx "$SWAPFILE"; then
  echo "swap already on at $SWAPFILE"
  exit 0
fi

# Create the file with fallocate; fall back to dd for filesystems without fallocate support.
fallocate -l "${SIZE_MB}M" "$SWAPFILE" || dd if=/dev/zero of="$SWAPFILE" bs=1M count="$SIZE_MB" status=progress
chmod 600 "$SWAPFILE"
mkswap "$SWAPFILE"
swapon "$SWAPFILE"

# Persist across reboot. Guard against duplicate fstab entries.
grep -q "^$SWAPFILE " /etc/fstab || echo "$SWAPFILE none swap sw 0 0" >> /etc/fstab

# Moderate swappiness — we want swap as a safety net, not aggressive paging.
sysctl -w vm.swappiness=10
grep -q '^vm.swappiness' /etc/sysctl.d/99-k3s-swap.conf 2>/dev/null || {
  echo 'vm.swappiness=10' > /etc/sysctl.d/99-k3s-swap.conf
}

# Verify.
free -h
swapon --show
```

**Rollback for Phase A:** `swapoff /swapfile && rm /swapfile` and remove the fstab line. This is safe at any time — swap is a soft resource.

**Verification after Phase A:**

- `free -h` shows `Swap: 4.0Gi` used ≈ 0.
- `swapon --show` shows `/swapfile 4G`.
- `sysctl vm.swappiness` returns `10`.
- kubelet still Ready (`kubectl get node k3s-cp-1`).
- No new `MemoryPressure` condition.

---

## Phase B — Preflight for the full OS update

Everything from here on runs from the CTO's operator machine (not from cp-1 itself, because we lose kubectl during the reboot). SSH to cp-1 is fine — kubectl calls issued from cp-1 during the pre-drain phase are fine and get logged into `/tmp/os-update-cp-1-<ts>.log`.

**Log directory contract** (same as `update-node.sh`): every command's stdout+stderr goes to `/tmp/os-update-cp-1-<UTC-timestamp>.log` on cp-1. Attach that log to the Paperclip execution ticket at the end.

Run `update-cp-1.sh --preflight`:

1. Cluster is currently healthy: `cluster-health.sh` returns 0.
2. All 5 fsn1 workers are `Ready` (worker-1, worker-2, worker-3, worker-5, cp-1). worker-4 is nbg1 and irrelevant here.
3. Each fsn1 worker has ≥ 500 MiB `MemAvailable`.
4. `time kubectl get nodes` returns in ≤ 2 s (kine is not already stressed).
5. cp-1 has swap on — abort if `free -h` shows `Swap: 0B`.
6. **k3s datastore snapshot.** cp-1 runs k3s in embedded-SQLite mode (`--datastore-endpoint` is unset). `k3s etcd-snapshot save --name pre-cp1-os-update-<UTC-date>` produces a copy of the SQLite file under `/var/lib/rancher/k3s/server/db/snapshots/`. This is our restore point. Snapshots older than 30 days are pruned in Phase F.

If any preflight check fails: STOP. Do not proceed. Do not add pods to cp-1 to "rebalance later" — that's a separate task.

---

## Phase C — Move stateful pods off cp-1 (batched)

**Rule** ([[k3s-drain-kine-thundering-herd]] guardrail #1): no drain that reschedules > 3 StatefulSets at once. So we DO NOT `kubectl drain k3s-cp-1` while stateful pods still live on it. We move them one at a time first.

**Cordon cp-1 immediately.** Cordon only prevents *new* scheduling — existing pods stay put. Cordoning first ensures that when we delete a stateful pod, the StatefulSet controller cannot re-create it back on cp-1.

```bash
kubectl cordon k3s-cp-1
```

**Order of eviction (idle → active, tiny → heavy):**

| # | Pod | Why in this position |
|---|-----|----------------------|
| 1 | `harbor/harbor-redis-0` | Idle cache, cold restart is instant, low write pressure |
| 2 | `nextcloud/nextcloud-redis-replicas-0` | Replica-1 of a Redis replicaset — non-primary, safe to bounce |
| 3 | `harbor/harbor-database-0` | Registry Postgres, mostly idle (registry pulls, not writes) |
| 4 | `stalwart/stalwart-postgres-0` | Mail metadata DB — active but recoverable; move before the mail server itself |
| 5 | `stalwart/stalwart-0` | Mail server, most important. Save for last so mail keeps flowing until the very end |

For each pod:

```bash
# 1. Delete the pod — StatefulSet controller will re-create it on a schedulable fsn1 worker.
kubectl -n "$NS" delete pod "$POD" --wait=false

# 2. Wait for the *new* pod (same name — StatefulSets keep identities) to be scheduled
#    somewhere OTHER than cp-1 and reach Ready=True.
deadline=$(( $(date +%s) + 300 ))
while [ $(date +%s) -lt $deadline ]; do
  new_node=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.spec.nodeName}' 2>/dev/null || true)
  ready=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)
  if [ -n "$new_node" ] && [ "$new_node" != "k3s-cp-1" ] && [ "$ready" = "True" ]; then
    echo "  $POD -> $new_node OK"
    break
  fi
  sleep 5
done
[ "$ready" = "True" ] || die "pod $NS/$POD did not reach Ready on a non-cp-1 node in 300s — HALT"

# 3. Kine health probe — abort the batch if the api server is getting slow.
t=$( { time kubectl get nodes >/dev/null; } 2>&1 | awk '/real/{print $2}' )
# t is like "0m1.234s" — if the m field is >0 or the s field is >5, halt.
# See update-cp-1.sh for the exact parse; > 5 s -> halt the whole cycle.

# 4. Free-memory probe on cp-1.
avail=$(ssh root@cp-1 "free -m | awk '/Mem:/{print \$7}'")
[ "$avail" -lt 200 ] && die "cp-1 MemAvailable dropped below 200 MiB — HALT"

# 5. Settle pause before the next eviction — kine needs to catch up on the write
#    burst from the just-attached PV and the just-scheduled pod.
sleep 60
```

**Halt conditions during Phase C** (any one → STOP, do NOT proceed to Phase D):

- `kubectl get nodes` takes > 5 s.
- cp-1 `MemAvailable` drops below 200 MiB.
- Any target worker enters `MemoryPressure=True` or `DiskPressure=True`.
- Any stateful pod does not reach Ready on a new node within 5 minutes (may indicate PV-attach or PDB issue).

If we halt: leave cp-1 **cordoned** but do not reboot. The pods that have already moved stay where they are; the pods that haven't will still be on cp-1. Escalate on the execution ticket with `blocked` and name the halt condition.

---

## Phase D — Drain remaining pods + apt

Once all 5 stateful pods are off cp-1 and cluster health is green:

```bash
# Drain everything else — only Deployment pods left (single-replica each,
# no PV attach, so reschedule is fast).
kubectl drain k3s-cp-1 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout="${DRAIN_TIMEOUT_SECONDS:-600}s"
```

If drain reports a PDB block: do NOT `--force`. Uncordon cp-1, mark the run as `blocked` on the PDB, and escalate. This is a design bug in the workload's PDB — fix it separately.

**apt on cp-1** (mirrors `update-node.sh` step 3; `update-cp-1.sh --apt` runs this via `ssh root@cp-1 bash -s`):

```bash
export DEBIAN_FRONTEND=noninteractive
APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
uname -r > /root/pre-apt-kernel   # for rollback reference
dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv

if dpkg --audit | grep -qE .; then dpkg --configure -a || true; fi
apt-get update
apt-get $APT_OPTS upgrade || { apt-get $APT_OPTS -f install; apt-get $APT_OPTS upgrade; }
apt-get $APT_OPTS dist-upgrade
apt-get $APT_OPTS autoremove --purge
apt-get clean
[ -f /var/run/reboot-required ] && echo REBOOT_REQUIRED=yes || echo REBOOT_REQUIRED=no
```

`pre-apt-kernel` and `pre-apt-packages.tsv` are the local rollback references — see Rollback below.

---

## Phase E — Reboot handling (api-server unavailability)

**Duration expectation:** 90–180 s of api-server unavailability. The operator machine will get `Unable to connect to the server` from kubectl during this window — that is expected, not an alarm.

**Escalation trigger:** api-server not back on `/livez` after **10 minutes** → escalate. First check Hetzner console via `hcloud server describe k3s-cp-1` for boot state; if kernel-panic / initramfs, use grub previous-kernel path (see Rollback). Do NOT rebuild the node.

**Steps (executed by `update-cp-1.sh --reboot`):**

```bash
# 1. Tell cp-1 to reboot. This SSH will hang up mid-command — that's fine.
ssh $SSH_OPTS root@cp-1 'systemctl reboot' || true

# 2. Deliberate 15 s pause — SSH needs to actually drop, don't race the poll.
sleep 15

# 3. Poll the api-server livez from the OPERATOR machine (not from cp-1).
#    Using --insecure (`-k`) because the server cert is self-signed by k3s.
deadline=$(( $(date +%s) + ${REBOOT_MAX_WAIT_SECONDS:-600} ))
while [ $(date +%s) -lt $deadline ]; do
  code=$(curl -sk -o /dev/null -w '%{http_code}' https://178.105.17.239:6443/livez 2>/dev/null || echo 000)
  if [ "$code" = "200" ]; then
    echo "  api-server /livez=200"
    break
  fi
  sleep 5
done
[ "$code" = "200" ] || die "api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate; check hcloud console"

# 4. Wait for kubelet Ready on cp-1 from the api-server view.
deadline=$(( $(date +%s) + 300 ))
while [ $(date +%s) -lt $deadline ]; do
  ready=$(kubectl get node k3s-cp-1 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
  [ "$ready" = "True" ] && break
  sleep 5
done
[ "$ready" = "True" ] || die "kubelet on cp-1 never returned Ready — escalate (do NOT change k3s config)"
```

**What to do if the api-server takes 3–10 minutes:** it may be normal on this box (large SQLite → replay). Wait it out. Do NOT restart k3s to "help" — kine SQLite replay is stateful and interrupting it can corrupt the DB. See [[k3s-drain-kine-thundering-herd]] recovery: doing nothing is a valid path.

**What to do if it takes > 10 minutes:** escalate; run the Hetzner-console diagnosis; if the console shows a bootable Ubuntu but the k3s service is failing, that's the boundary — this procedure stops here. Follow `K3S_OPERATIONS.md` for the k3s recovery path.

---

## Phase F — Uncordon + verify + finalize

```bash
# Uncordon cp-1. StatefulSet pods will NOT be moved back — that's correct
# behaviour, they were moved for a reason. Rebalancing is a separate task.
kubectl uncordon k3s-cp-1

# Settle wait — kine needs to process the flood of "node schedulable again" events.
sleep "${POST_UNCORDON_WAIT_SECONDS:-180}"

# Full cluster health.
RETRY_ON_TRANSIENT=1 infrastructure/scripts/os-update/cluster-health.sh

# Print the apt history for audit.
ssh root@cp-1 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'

# Prune snapshots older than 30 days.
ssh root@cp-1 'find /var/lib/rancher/k3s/server/db/snapshots/ -type f -mtime +30 -name "pre-*" -print'
# (list only — actual deletion is a manual review after the run)
```

Attach the `/tmp/os-update-cp-1-<ts>.log` and the ordered ledger of moved pods to the execution ticket. Mark the ticket `done` on green health, or `blocked` naming the specific residual issue on red.

---

## Rollback / recovery

### If apt broke a package

- On cp-1, `dpkg --audit` to find half-configured packages.
- `dpkg --configure -a`, then `apt-get -f install`.
- If a specific package broke and you know the previous version from `/root/pre-apt-packages.tsv`, `apt-get install <pkg>=<old-version>`.

### If the new kernel does not boot

- Hetzner cloud console: send `hcloud server request-console k3s-cp-1` → get VNC URL, watch boot.
- If grub is up, select the previous-kernel entry. Ubuntu keeps ≥ 1 old kernel installed by default (we verified `6.8.0-137` current; the previous `6.8.0-124` was in use during the DEV-478 cycle).
- Once booted on the old kernel, `apt-get remove` the broken kernel and pin the working one:
  ```bash
  apt-mark hold linux-image-<broken-version> linux-headers-<broken-version>
  ```
- Escalate on the ticket regardless — a kernel rollback is a follow-up investigation, not a "done" outcome.

### If the node does not return at all

- Do NOT `hcloud server delete`. Do NOT re-provision.
- Escalate to the board with the Hetzner console output and the `/tmp/os-update-cp-1-*.log`.
- The `--cluster-reset --cluster-reset-restore-path=<snapshot>` recovery path exists (see `K3S_OPERATIONS.md`) but requires board approval per stateful-service safety rules. The pre-flight snapshot from Phase B is the restore point.

### If Phase C halted mid-eviction

- cp-1 is cordoned, some stateful pods have moved, some haven't. Cluster is functional.
- Uncordon cp-1 (`kubectl uncordon k3s-cp-1`) — StatefulSets that stayed on cp-1 keep running, moved ones stay where they went.
- Do NOT proceed to Phase D. Open a follow-up ticket with the halt condition. Retry in the next maintenance window after fixing the halt condition.

---

## Automation entry points

- `infrastructure/scripts/os-update/update-cp-1.sh` — this whole flow, with subcommands:
  - `--add-swap` — Phase A only, idempotent, safe standalone.
  - `--dry-run` — walk Phases B/C/D/E/F printing exactly what would be done without touching anything. Safe to run any time; used for design review.
  - `--preflight` — Phase B only.
  - `--drain-stateful` — Phase C only (cordon + batched stateful moves).
  - `--apt` — Phase D apt commands only (requires cp-1 already fully drained).
  - `--reboot` — Phase E only (requires apt already done).
  - `--finalize` — Phase F only (uncordon + verify).
  - `--run` — do all phases in order, with a confirmation prompt between each unless `ASSUME_YES=1`.

The script follows the same log-dir contract as `update-node.sh` (`/tmp/os-update-cp-1-<UTC-timestamp>.log`).

---

## Change history

| Date | Change | By |
|------|--------|-----|
| 2026-08-16 | Initial cp-1 update design (DEV-496) | CTO agent |