e848d225cd
Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an HA-aware procedure that covers all three CPs. Rationale: after DEV-510 the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs are NoSchedule-tainted, so: - Kine cascade is no longer the driving risk; embedded 3-node etcd tolerates one member down. The kubectl-latency guardrail stays as a soft signal. - CPs host no StatefulSets or single-replica Deployments. The old Phase C batched stateful eviction dance is obsolete. - CP reboots no longer lose the api-server; two peers keep serving. /livez is polled from the operator machine and cross-checked with etcdctl endpoint status from a peer CP. New CP ordering rule: one CP at a time, leader last, health gate between CPs. Refuses to start if another CP is already cordoned. Automation: - New scripts/os-update/update-cp-node.sh <node> is the entry point. Supports --add-swap / --preflight / --drain / --apt / --reboot / --finalize / --run / --dry-run. Phase A also installs etcd-client on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the target. Peer picker prefers a CP that has etcdctl; refuses cp-1 target with a targeted error if no other CP has etcdctl yet. - scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards to update-cp-node.sh k3s-cp-1 "$@" for backwards compat. - CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc. - OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and automation entry point list all point at the new HA-aware artifacts. Verified: bash -n clean on both scripts; --dry-run walks all six phases for each of cp-1/cp-2/cp-3; --preflight against live cluster correctly aborts cp-1 with "install etcdctl on another CP first" and cp-2/cp-3 with "run --add-swap first" (their current state). Execution is board-approval-gated; DEV-497 (cp-1-only executor) is superseded by a successor ticket. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|---|---|---|
| .. | ||
| cluster-health.sh | ||
| ensure-node-docker.sh | ||
| os-update.sh | ||
| README.md | ||
| update-cp-1.sh | ||
| update-cp-node.sh | ||
| update-node.sh | ||
OS Update Automation
Scripts that implement the weekly rolling Ubuntu OS-update procedure.
Authoritative doc: ../../OS_UPDATE_PROCEDURE.md — read it first. The scripts here mirror that procedure step-for-step.
Files
| Script | Purpose |
|---|---|
cluster-health.sh |
Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node. |
update-node.sh <node> |
Drain, apt-update, reboot-if-required, wait for Ready, ensure Docker on labeled nodes, uncordon, post-node health check. Retriable per-node. |
ensure-node-docker.sh <node> |
Idempotent: on nodes labeled basicstack.de/docker=true, install docker.io if missing, apt-mark manual, enable+start the docker service, wait for /var/run/docker.sock, re-apply the label. No-op on nodes without the label. Called from update-node.sh between "kubelet Ready" and "uncordon"; also runnable ad-hoc for recovery. |
os-update.sh |
Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest. |
Order of operations (encoded in os-update.sh)
- Workers with no stateful affinity concerns first.
fsn1workers (potential Stalwart hosts) last among workers — seestalwart-datacenter-affinitynote.k3s-cp-1last (single control plane).- One node at a time. Never in parallel.
Guardrails the scripts enforce
- Preflight cluster health failure → refuse to start.
- Drain with a PDB conflict → uncordon and mark the node
SKIPPED_DRAIN_FAILED, never--force. - Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config.
- kubelet doesn't return
Ready→ hard stop, escalate. Do NOT touch k3s config. - Post-node cluster health check red → hard stop, do not proceed to the next node.
- Absolute rule: no k3s config, no manifests, no PVs, no service files touched — apt/dpkg only.
Typical invocations
# Dry-run: print the ordered plan, touch nothing.
./os-update.sh --dry-run
# Full cycle (weekly, triggered by the Paperclip routine).
./os-update.sh
# Retry a single node (after fixing a manual issue).
./update-node.sh k3s-worker-3
# Restart a partial cycle from a specific node onward.
./os-update.sh --start-from k3s-worker-4
# Ad-hoc: reconcile Docker on a single node (e.g. after emergency ops).
./ensure-node-docker.sh k3s-worker-3
Logs land in /tmp/os-update-<UTC-timestamp>/ on the machine that ran the cycle.