basicstack/stack.basicstack.de

History

CTO Agent a30274d1eb feat(os-update): integrate control planes into weekly rolling cycle (DEV-553) os-update.sh now walks all three CPs after workers, one at a time, etcd followers first and leader last. Each CP is delegated to update-cp-node.sh --run with ASSUME_YES=1; a health gate (cluster-health.sh + etcdctl endpoint status) runs between CPs. Adds --skip-cps and --only-cps flags; --only <node> and --start-from <node> now route to the correct list automatically. OS_UPDATE_PROCEDURE.md updated: CPs are no longer excluded, §8 renamed to "Control plane handling" and describes the leader-last cycle + delegation. Co-Authored-By: Paperclip <noreply@paperclip.ing>		2026-08-30 09:41:22 +00:00
..
cluster-health.sh	Add weekly rolling OS-update procedure for k3s nodes (DEV-462)	2026-08-09 15:55:39 +00:00
ensure-node-docker.sh	feat(os-update): reconcile Docker + label after node updates (DEV-499)	2026-08-16 20:22:43 +00:00
os-update.sh	feat(os-update): integrate control planes into weekly rolling cycle (DEV-553)	2026-08-30 09:41:22 +00:00
README.md	feat(os-update): reconcile Docker + label after node updates (DEV-499)	2026-08-16 20:22:43 +00:00
update-cp-1.sh	os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515)	2026-08-23 01:53:14 +00:00
update-cp-node.sh	os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515)	2026-08-23 01:53:14 +00:00
update-node.sh	feat(os-update): reconcile Docker + label after node updates (DEV-499)	2026-08-16 20:22:43 +00:00

README.md

OS Update Automation

Scripts that implement the weekly rolling Ubuntu OS-update procedure.

Authoritative doc: ../../OS_UPDATE_PROCEDURE.md — read it first. The scripts here mirror that procedure step-for-step.

Files

Script	Purpose
`cluster-health.sh`	Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node.
`update-node.sh <node>`	Drain, apt-update, reboot-if-required, wait for Ready, ensure Docker on labeled nodes, uncordon, post-node health check. Retriable per-node.
`ensure-node-docker.sh <node>`	Idempotent: on nodes labeled `basicstack.de/docker=true`, install docker.io if missing, `apt-mark manual`, enable+start the docker service, wait for `/var/run/docker.sock`, re-apply the label. No-op on nodes without the label. Called from `update-node.sh` between "kubelet Ready" and "uncordon"; also runnable ad-hoc for recovery.
`os-update.sh`	Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest.

Order of operations (encoded in `os-update.sh`)

Workers with no stateful affinity concerns first.
fsn1 workers (potential Stalwart hosts) last among workers — see stalwart-datacenter-affinity note.
k3s-cp-1 last (single control plane).
One node at a time. Never in parallel.

Guardrails the scripts enforce

Preflight cluster health failure → refuse to start.
Drain with a PDB conflict → uncordon and mark the node SKIPPED_DRAIN_FAILED, never --force.
Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config.
kubelet doesn't return Ready → hard stop, escalate. Do NOT touch k3s config.
Post-node cluster health check red → hard stop, do not proceed to the next node.
Absolute rule: no k3s config, no manifests, no PVs, no service files touched — apt/dpkg only.

Typical invocations

# Dry-run: print the ordered plan, touch nothing.
./os-update.sh --dry-run

# Full cycle (weekly, triggered by the Paperclip routine).
./os-update.sh

# Retry a single node (after fixing a manual issue).
./update-node.sh k3s-worker-3

# Restart a partial cycle from a specific node onward.
./os-update.sh --start-from k3s-worker-4

# Ad-hoc: reconcile Docker on a single node (e.g. after emergency ops).
./ensure-node-docker.sh k3s-worker-3

Logs land in /tmp/os-update-<UTC-timestamp>/ on the machine that ran the cycle.