basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/infrastructure/scripts/os-update/README.md
CTO (Paperclip Agent) 5b627f9e73 feat(os-update): bounded retry on post-uncordon health gate (DEV-595) 
The 180s post-uncordon health check in `update-node.sh` step 7 was single-shot.
Any transient yellow signal (Init:*, Running-but-not-Ready, deployment below
replicas during reschedule) at the exact moment of the check would FATAL the
whole cycle and require operator wake + manual `--start-from <next-node>`.
During DEV-593 this fired three times in one cycle (opencloud init, harbor
jobservice reinit) for conditions that cleared within seconds.

Wrap the health check in a bounded retry loop:
- First check at `POST_UNCORDON_WAIT_SECONDS` (default 180s, unchanged).
- Up to `HEALTH_MAX_RETRIES` (default 10) more attempts at
  `HEALTH_RETRY_INTERVAL_SECONDS` (default 30s) = +5 min max.
- Short-circuit halt immediately on persistent pod errors
  (CrashLoopBackOff, ImagePullBackOff, ErrImagePull,
  CreateContainerConfigError, InvalidImageName) — these don't recover on
  their own and shouldn't burn the retry budget.
- Each retry calls `cluster-health.sh` with `RETRY_ON_TRANSIENT=0` so the
  outer loop controls cadence (no double 30s sleeps per attempt).

No k3s or manifest changes. Preflight path is untouched.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-11 02:57:31 +00:00

3.4 KiB
Raw Blame History

OS Update Automation

Scripts that implement the weekly rolling Ubuntu OS-update procedure.

Authoritative doc: ../../OS_UPDATE_PROCEDURE.md — read it first. The scripts here mirror that procedure step-for-step.

Files

Script Purpose
cluster-health.sh Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node.
update-node.sh <node> Drain, apt-update, reboot-if-required, wait for Ready, ensure Docker on labeled nodes, uncordon, post-node health check. Retriable per-node.
ensure-node-docker.sh <node> Idempotent: on nodes labeled basicstack.de/docker=true, install docker.io if missing, apt-mark manual, enable+start the docker service, wait for /var/run/docker.sock, re-apply the label. No-op on nodes without the label. Called from update-node.sh between "kubelet Ready" and "uncordon"; also runnable ad-hoc for recovery.
os-update.sh Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest.

Order of operations (encoded in os-update.sh)

  1. Workers with no stateful affinity concerns first.
  2. fsn1 workers (potential Stalwart hosts) last among workers — see stalwart-datacenter-affinity note.
  3. k3s-cp-1 last (single control plane).
  4. One node at a time. Never in parallel.

Guardrails the scripts enforce

  • Preflight cluster health failure → refuse to start.
  • SSH-drift preflight (DEV-587): before draining anything, verify the operator can SSH to every planned node and that cp-1's pubkey is present in each target's /root/.ssh/authorized_keys. Missing pubkeys are auto-appended via the operator's own SSH; nodes that reject the operator entirely fail the whole cycle with the exact repair command. Preflight report is written to /tmp/os-update-<UTC-ts>/ssh-preflight.txt. Escape hatch: --skip-ssh-preflight for bootstrap.
  • Drain with a PDB conflict → uncordon and mark the node SKIPPED_DRAIN_FAILED, never --force.
  • Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config.
  • kubelet doesn't return Ready → hard stop, escalate. Do NOT touch k3s config.
  • Post-node cluster health check red → first retry up to HEALTH_MAX_RETRIES (default 10) more times at HEALTH_RETRY_INTERVAL_SECONDS (default 30s) to let transient reinit windows settle (DEV-595). Short-circuit halt immediately on any CrashLoopBackOff/ImagePullBackOff/ErrImagePull/CreateContainerConfigError/InvalidImageName. Still red after all attempts → hard stop, do not proceed to the next node.
  • Absolute rule: no k3s config, no manifests, no PVs, no service files touched — apt/dpkg only.

Typical invocations

# Dry-run: print the ordered plan, touch nothing.
./os-update.sh --dry-run

# Full cycle (weekly, triggered by the Paperclip routine).
./os-update.sh

# Retry a single node (after fixing a manual issue).
./update-node.sh k3s-worker-3

# Restart a partial cycle from a specific node onward.
./os-update.sh --start-from k3s-worker-4

# Ad-hoc: reconcile Docker on a single node (e.g. after emergency ops).
./ensure-node-docker.sh k3s-worker-3

Logs land in /tmp/os-update-<UTC-timestamp>/ on the machine that ran the cycle.