The 180s post-uncordon health check in `update-node.sh` step 7 was single-shot.
Any transient yellow signal (Init:*, Running-but-not-Ready, deployment below
replicas during reschedule) at the exact moment of the check would FATAL the
whole cycle and require operator wake + manual `--start-from <next-node>`.
During DEV-593 this fired three times in one cycle (opencloud init, harbor
jobservice reinit) for conditions that cleared within seconds.
Wrap the health check in a bounded retry loop:
- First check at `POST_UNCORDON_WAIT_SECONDS` (default 180s, unchanged).
- Up to `HEALTH_MAX_RETRIES` (default 10) more attempts at
`HEALTH_RETRY_INTERVAL_SECONDS` (default 30s) = +5 min max.
- Short-circuit halt immediately on persistent pod errors
(CrashLoopBackOff, ImagePullBackOff, ErrImagePull,
CreateContainerConfigError, InvalidImageName) — these don't recover on
their own and shouldn't burn the retry budget.
- Each retry calls `cluster-health.sh` with `RETRY_ON_TRANSIENT=0` so the
outer loop controls cadence (no double 30s sleeps per attempt).
No k3s or manifest changes. Preflight path is untouched.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
DEV-586 (2026-09-27) aborted mid-drain because cp-1's pubkey had drifted out
of four nodes' /root/.ssh/authorized_keys — probably Hetzner cloud-init
re-syncing from the HC SSH-keys API on reboot (the operator key is not in
that keyset). Both the apt-phase SSH and the CP peer-probe SSH died once
they tried to reach the drifted targets, and recovery cost real time.
os-update.sh now runs a preflight against the fully-computed plan:
- Reads cp-1's pubkey once (fails loudly if operator can't reach cp-1).
- For every planned worker + CP: proves operator SSH works, then grep-checks
cp-1's pubkey in /root/.ssh/authorized_keys. Missing pubkeys are appended
via the operator's own SSH (auto-heal). Verify via grep after the write.
- Nodes still unreachable at the end block the whole cycle with the exact
fix command in the log — nothing gets drained until this passes.
- Preflight report: /tmp/os-update-<UTC-ts>/ssh-preflight.txt (ok/repaired/
problem lists) as audit trail.
- Escape hatch: --skip-ssh-preflight for bootstrap scenarios.
Also factors the node->ssh-endpoint case into a shared node_ssh_endpoint()
helper so the preflight, finalize/apt-history loop, and any future call site
can't drift apart.
Verified end-to-end against the live cluster: all planned nodes reachable
and cp-1's pubkey currently present on cp-2/cp-3, worker-1..6.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The weekly rolling OS-update cycle has been silently stripping docker.io from
worker nodes (DEV-498), breaking the Forgejo runner whose hostPath mount for
/var/run/docker.sock requires the socket to exist. Fix in two layers:
- Defensive pin: update-node.sh now runs `apt-mark manual docker.io` in the
apt phase (whenever it is installed) so `apt-get autoremove --purge` cannot
silently drop it during subsequent upgrades.
- Post-reboot reconciliation: new `ensure-node-docker.sh` installs docker.io
if missing, enables + starts the systemd unit, waits for /var/run/docker.sock,
and re-applies the `basicstack.de/docker=true` label. Wired into
update-node.sh between kubelet-Ready and uncordon. No-op on nodes without
the label (safe for cp-1 and the update runner).
Verified idempotent against all 5 labeled workers; `apt-mark manual docker.io`
now set on every worker (survived across reboots by design).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update
procedure (drain -> apt -> reboot -> verify -> uncordon -> health ->
next). Explicit MUST NOT list around k3s config, PVs, and manifests.
- infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip
routine (Sun 03:00 Europe/Berlin) that fires this procedure.
- infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh,
os-update.sh, README. Enforces the same guardrails in code:
workers-first-then-CP, one node at a time, no --force drains, halts on
reboot/kubelet/health failure, never touches k3s config.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
