basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/infrastructure/scripts/os-update
History
CTO Agent 05000b3e06 docs(infra): design cp-1 OS update procedure (DEV-496) 
cp-1 is excluded from os-update.sh because a one-shot drain would trigger
the kine cascade documented in DEV-495. This adds:

- CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful
  eviction (Phase C), drain+apt (Phase D), reboot with external livez
  monitor (Phase E), uncordon+verify (Phase F), and rollback paths.
- scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the
  same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports
  --dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot,
  --finalize, --run.
- os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1
  script; kine thundering-herd guardrails preserved.
- OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure.

Execution requires separate board approval; this change is design +
dry-run artifact only.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 19:28:52 +00:00
..
cluster-health.sh Add weekly rolling OS-update procedure for k3s nodes (DEV-462) 2026-08-09 15:55:39 +00:00
os-update.sh docs(infra): design cp-1 OS update procedure (DEV-496) 2026-08-16 19:28:52 +00:00
README.md Add weekly rolling OS-update procedure for k3s nodes (DEV-462) 2026-08-09 15:55:39 +00:00
update-cp-1.sh docs(infra): design cp-1 OS update procedure (DEV-496) 2026-08-16 19:28:52 +00:00
update-node.sh Add weekly rolling OS-update procedure for k3s nodes (DEV-462) 2026-08-09 15:55:39 +00:00

README.md

OS Update Automation

Scripts that implement the weekly rolling Ubuntu OS-update procedure.

Authoritative doc: ../../OS_UPDATE_PROCEDURE.md — read it first. The scripts here mirror that procedure step-for-step.

Files

Script Purpose
cluster-health.sh Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node.
update-node.sh <node> Drain, apt-update, reboot-if-required, wait for Ready, uncordon, post-node health check. Retriable per-node.
os-update.sh Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest.

Order of operations (encoded in os-update.sh)

  1. Workers with no stateful affinity concerns first.
  2. fsn1 workers (potential Stalwart hosts) last among workers — see stalwart-datacenter-affinity note.
  3. k3s-cp-1 last (single control plane).
  4. One node at a time. Never in parallel.

Guardrails the scripts enforce

  • Preflight cluster health failure → refuse to start.
  • Drain with a PDB conflict → uncordon and mark the node SKIPPED_DRAIN_FAILED, never --force.
  • Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config.
  • kubelet doesn't return Ready → hard stop, escalate. Do NOT touch k3s config.
  • Post-node cluster health check red → hard stop, do not proceed to the next node.
  • Absolute rule: no k3s config, no manifests, no PVs, no service files touched — apt/dpkg only.

Typical invocations

# Dry-run: print the ordered plan, touch nothing.
./os-update.sh --dry-run

# Full cycle (weekly, triggered by the Paperclip routine).
./os-update.sh

# Retry a single node (after fixing a manual issue).
./update-node.sh k3s-worker-3

# Restart a partial cycle from a specific node onward.
./os-update.sh --start-from k3s-worker-4

Logs land in /tmp/os-update-<UTC-timestamp>/ on the machine that ran the cycle.