The 180s post-uncordon health check in `update-node.sh` step 7 was single-shot.
Any transient yellow signal (Init:*, Running-but-not-Ready, deployment below
replicas during reschedule) at the exact moment of the check would FATAL the
whole cycle and require operator wake + manual `--start-from <next-node>`.
During DEV-593 this fired three times in one cycle (opencloud init, harbor
jobservice reinit) for conditions that cleared within seconds.
Wrap the health check in a bounded retry loop:
- First check at `POST_UNCORDON_WAIT_SECONDS` (default 180s, unchanged).
- Up to `HEALTH_MAX_RETRIES` (default 10) more attempts at
`HEALTH_RETRY_INTERVAL_SECONDS` (default 30s) = +5 min max.
- Short-circuit halt immediately on persistent pod errors
(CrashLoopBackOff, ImagePullBackOff, ErrImagePull,
CreateContainerConfigError, InvalidImageName) — these don't recover on
their own and shouldn't burn the retry budget.
- Each retry calls `cluster-health.sh` with `RETRY_ON_TRANSIENT=0` so the
outer loop controls cadence (no double 30s sleeps per attempt).
No k3s or manifest changes. Preflight path is untouched.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
DEV-586 (2026-09-27) aborted mid-drain because cp-1's pubkey had drifted out
of four nodes' /root/.ssh/authorized_keys — probably Hetzner cloud-init
re-syncing from the HC SSH-keys API on reboot (the operator key is not in
that keyset). Both the apt-phase SSH and the CP peer-probe SSH died once
they tried to reach the drifted targets, and recovery cost real time.
os-update.sh now runs a preflight against the fully-computed plan:
- Reads cp-1's pubkey once (fails loudly if operator can't reach cp-1).
- For every planned worker + CP: proves operator SSH works, then grep-checks
cp-1's pubkey in /root/.ssh/authorized_keys. Missing pubkeys are appended
via the operator's own SSH (auto-heal). Verify via grep after the write.
- Nodes still unreachable at the end block the whole cycle with the exact
fix command in the log — nothing gets drained until this passes.
- Preflight report: /tmp/os-update-<UTC-ts>/ssh-preflight.txt (ok/repaired/
problem lists) as audit trail.
- Escape hatch: --skip-ssh-preflight for bootstrap scenarios.
Also factors the node->ssh-endpoint case into a shared node_ssh_endpoint()
helper so the preflight, finalize/apt-history loop, and any future call site
can't drift apart.
Verified end-to-end against the live cluster: all planned nodes reachable
and cp-1's pubkey currently present on cp-2/cp-3, worker-1..6.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
k3s api-server /livez on this cluster is auth-gated and returns HTTP 401
without a bearer token. The Phase E reboot poll hardcoded HTTP 200, so
every CP reboot would time out at 10 min ("api-server did not return
within 600s") even though cp-2 was actually Ready and etcd-rejoined
within ~90 s.
Any HTTP response from the api-server (200/401/403) means it is up and
serving; only network-level failures (000, connection refused) count as
"not yet back". Added --max-time 5 to keep polling snappy immediately
after the reboot when TCP may accept-then-drop.
Verified during DEV-566 cycle:
curl -sk -o/dev/null -w '%{http_code}' https://<cp>:6443/livez -> 401
on cp-1, cp-2, cp-3 (all Ready).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
k3s-worker-6 was joined on 2026-08-30 but never added to the
node_ssh_target case in update-node.sh, so the weekly cycle would die
with 'unknown node k3s-worker-6' as soon as it reached that node. Map
it to its public IP (49.12.223.133) — same pattern as worker-3/4/5.
Also add the corresponding case in os-update.sh's apt-history digest.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- provision-k3s-worker.sh now sets --node-ip=<private-ip> on the agent
install so k3s does not default to the public IP (DEV-224 root cause)
- K3S_OPERATIONS.md node table reflects the current cluster (cp-2/cp-3,
worker-4/5/6), documents the Hetzner Cloud Volume datacenter constraint,
and calls out the --node-ip requirement in the manual join steps
Co-Authored-By: Paperclip <noreply@paperclip.ing>
os-update.sh now walks all three CPs after workers, one at a time, etcd
followers first and leader last. Each CP is delegated to update-cp-node.sh
--run with ASSUME_YES=1; a health gate (cluster-health.sh +
etcdctl endpoint status) runs between CPs. Adds --skip-cps and --only-cps
flags; --only <node> and --start-from <node> now route to the correct list
automatically.
OS_UPDATE_PROCEDURE.md updated: CPs are no longer excluded, §8 renamed to
"Control plane handling" and describes the leader-last cycle + delegation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an
HA-aware procedure that covers all three CPs. Rationale: after DEV-510
the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs
are NoSchedule-tainted, so:
- Kine cascade is no longer the driving risk; embedded 3-node etcd
tolerates one member down. The kubectl-latency guardrail stays as a
soft signal.
- CPs host no StatefulSets or single-replica Deployments. The old
Phase C batched stateful eviction dance is obsolete.
- CP reboots no longer lose the api-server; two peers keep serving.
/livez is polled from the operator machine and cross-checked with
etcdctl endpoint status from a peer CP.
New CP ordering rule: one CP at a time, leader last, health gate
between CPs. Refuses to start if another CP is already cordoned.
Automation:
- New scripts/os-update/update-cp-node.sh <node> is the entry point.
Supports --add-swap / --preflight / --drain / --apt / --reboot /
--finalize / --run / --dry-run. Phase A also installs etcd-client
on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the
target. Peer picker prefers a CP that has etcdctl; refuses cp-1
target with a targeted error if no other CP has etcdctl yet.
- scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards
to update-cp-node.sh k3s-cp-1 "$@" for backwards compat.
- CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc.
- OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and
automation entry point list all point at the new HA-aware artifacts.
Verified: bash -n clean on both scripts; --dry-run walks all six
phases for each of cp-1/cp-2/cp-3; --preflight against live cluster
correctly aborts cp-1 with "install etcdctl on another CP first" and
cp-2/cp-3 with "run --add-swap first" (their current state).
Execution is board-approval-gated; DEV-497 (cp-1-only executor) is
superseded by a successor ticket.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Since DEV-510 landed HA control plane (2026-08-22), the cluster runs cp-1
plus cp-2/cp-3. The previous exclusion in os-update.sh matched `k3s-cp-1`
by hard-coded name only, which would have caused cp-2 and cp-3 to be
treated as regular fsn1 workers and drained/rebooted without the
CP-specific procedure.
Fix: select the CP list from `kubectl get nodes -l
node-role.kubernetes.io/control-plane` and skip any of those nodes. This
covers all present and future CPs automatically.
Also updated OS_UPDATE_PROCEDURE.md topology table and order rule to
document that all three CPs exist and are excluded from the weekly
cycle. The HA-aware CP OS-update procedure is a separate follow-up.
Verified on the current cluster:
[plan] EXCLUDING control-plane nodes: k3s-cp-1 k3s-cp-2 k3s-cp-3
[plan] ordered nodes (6): k3s-worker-4 k3s-update-runner k3s-worker-1
k3s-worker-2 k3s-worker-3 k3s-worker-5
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The weekly rolling OS-update cycle has been silently stripping docker.io from
worker nodes (DEV-498), breaking the Forgejo runner whose hostPath mount for
/var/run/docker.sock requires the socket to exist. Fix in two layers:
- Defensive pin: update-node.sh now runs `apt-mark manual docker.io` in the
apt phase (whenever it is installed) so `apt-get autoremove --purge` cannot
silently drop it during subsequent upgrades.
- Post-reboot reconciliation: new `ensure-node-docker.sh` installs docker.io
if missing, enables + starts the systemd unit, waits for /var/run/docker.sock,
and re-applies the `basicstack.de/docker=true` label. Wired into
update-node.sh between kubelet-Ready and uncordon. No-op on nodes without
the label (safe for cp-1 and the update runner).
Verified idempotent against all 5 labeled workers; `apt-mark manual docker.io`
now set on every worker (survived across reboots by design).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
cp-1 is excluded from os-update.sh because a one-shot drain would trigger
the kine cascade documented in DEV-495. This adds:
- CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful
eviction (Phase C), drain+apt (Phase D), reboot with external livez
monitor (Phase E), uncordon+verify (Phase F), and rollback paths.
- scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the
same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports
--dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot,
--finalize, --run.
- os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1
script; kine thundering-herd guardrails preserved.
- OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure.
Execution requires separate board approval; this change is design +
dry-run artifact only.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update
procedure (drain -> apt -> reboot -> verify -> uncordon -> health ->
next). Explicit MUST NOT list around k3s config, PVs, and manifests.
- infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip
routine (Sun 03:00 Europe/Berlin) that fires this procedure.
- infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh,
os-update.sh, README. Enforces the same guardrails in code:
workers-first-then-CP, one node at a time, no --force drains, halts on
reboot/kubelet/health failure, never touches k3s config.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
