19af6918f0
The weekly rolling OS-update cycle has been silently stripping docker.io from worker nodes (DEV-498), breaking the Forgejo runner whose hostPath mount for /var/run/docker.sock requires the socket to exist. Fix in two layers: - Defensive pin: update-node.sh now runs `apt-mark manual docker.io` in the apt phase (whenever it is installed) so `apt-get autoremove --purge` cannot silently drop it during subsequent upgrades. - Post-reboot reconciliation: new `ensure-node-docker.sh` installs docker.io if missing, enables + starts the systemd unit, waits for /var/run/docker.sock, and re-applies the `basicstack.de/docker=true` label. Wired into update-node.sh between kubelet-Ready and uncordon. No-op on nodes without the label (safe for cp-1 and the update runner). Verified idempotent against all 5 labeled workers; `apt-mark manual docker.io` now set on every worker (survived across reboots by design). Co-Authored-By: Paperclip <noreply@paperclip.ing>
51 lines
2.6 KiB
Markdown
51 lines
2.6 KiB
Markdown
# OS Update Automation
|
|
|
|
Scripts that implement the weekly rolling Ubuntu OS-update procedure.
|
|
|
|
**Authoritative doc:** [`../../OS_UPDATE_PROCEDURE.md`](../../OS_UPDATE_PROCEDURE.md) — read it first. The scripts here mirror that procedure step-for-step.
|
|
|
|
## Files
|
|
|
|
| Script | Purpose |
|
|
|--------|---------|
|
|
| `cluster-health.sh` | Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node. |
|
|
| `update-node.sh <node>` | Drain, apt-update, reboot-if-required, wait for Ready, ensure Docker on labeled nodes, uncordon, post-node health check. Retriable per-node. |
|
|
| `ensure-node-docker.sh <node>` | Idempotent: on nodes labeled `basicstack.de/docker=true`, install docker.io if missing, `apt-mark manual`, enable+start the docker service, wait for `/var/run/docker.sock`, re-apply the label. No-op on nodes without the label. Called from `update-node.sh` between "kubelet Ready" and "uncordon"; also runnable ad-hoc for recovery. |
|
|
| `os-update.sh` | Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest. |
|
|
|
|
## Order of operations (encoded in `os-update.sh`)
|
|
|
|
1. Workers with no stateful affinity concerns first.
|
|
2. `fsn1` workers (potential Stalwart hosts) last among workers — see [`stalwart-datacenter-affinity`](../../K3S_OPERATIONS.md) note.
|
|
3. `k3s-cp-1` last (single control plane).
|
|
4. One node at a time. Never in parallel.
|
|
|
|
## Guardrails the scripts enforce
|
|
|
|
- Preflight cluster health failure → refuse to start.
|
|
- Drain with a PDB conflict → uncordon and mark the node `SKIPPED_DRAIN_FAILED`, never `--force`.
|
|
- Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config.
|
|
- kubelet doesn't return `Ready` → hard stop, escalate. Do NOT touch k3s config.
|
|
- Post-node cluster health check red → hard stop, do not proceed to the next node.
|
|
- Absolute rule: **no k3s config, no manifests, no PVs, no service files touched** — apt/dpkg only.
|
|
|
|
## Typical invocations
|
|
|
|
```bash
|
|
# Dry-run: print the ordered plan, touch nothing.
|
|
./os-update.sh --dry-run
|
|
|
|
# Full cycle (weekly, triggered by the Paperclip routine).
|
|
./os-update.sh
|
|
|
|
# Retry a single node (after fixing a manual issue).
|
|
./update-node.sh k3s-worker-3
|
|
|
|
# Restart a partial cycle from a specific node onward.
|
|
./os-update.sh --start-from k3s-worker-4
|
|
|
|
# Ad-hoc: reconcile Docker on a single node (e.g. after emergency ops).
|
|
./ensure-node-docker.sh k3s-worker-3
|
|
```
|
|
|
|
Logs land in `/tmp/os-update-<UTC-timestamp>/` on the machine that ran the cycle.
|