basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/infrastructure/CP1_UPDATE_PROCEDURE.md

15 lines
1.3 KiB
Markdown
Raw Normal View History

os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515) Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an HA-aware procedure that covers all three CPs. Rationale: after DEV-510 the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs are NoSchedule-tainted, so: - Kine cascade is no longer the driving risk; embedded 3-node etcd tolerates one member down. The kubectl-latency guardrail stays as a soft signal. - CPs host no StatefulSets or single-replica Deployments. The old Phase C batched stateful eviction dance is obsolete. - CP reboots no longer lose the api-server; two peers keep serving. /livez is polled from the operator machine and cross-checked with etcdctl endpoint status from a peer CP. New CP ordering rule: one CP at a time, leader last, health gate between CPs. Refuses to start if another CP is already cordoned. Automation: - New scripts/os-update/update-cp-node.sh <node> is the entry point. Supports --add-swap / --preflight / --drain / --apt / --reboot / --finalize / --run / --dry-run. Phase A also installs etcd-client on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the target. Peer picker prefers a CP that has etcdctl; refuses cp-1 target with a targeted error if no other CP has etcdctl yet. - scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards to update-cp-node.sh k3s-cp-1 "$@" for backwards compat. - CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc. - OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and automation entry point list all point at the new HA-aware artifacts. Verified: bash -n clean on both scripts; --dry-run walks all six phases for each of cp-1/cp-2/cp-3; --preflight against live cluster correctly aborts cp-1 with "install etcdctl on another CP first" and cp-2/cp-3 with "run --add-swap first" (their current state). Execution is board-approval-gated; DEV-497 (cp-1-only executor) is superseded by a successor ticket. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-23 01:53:14 +00:00
# k3s-cp-1 OS Update Procedure — SUPERSEDED
docs(infra): design cp-1 OS update procedure (DEV-496) cp-1 is excluded from os-update.sh because a one-shot drain would trigger the kine cascade documented in DEV-495. This adds: - CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful eviction (Phase C), drain+apt (Phase D), reboot with external livez monitor (Phase E), uncordon+verify (Phase F), and rollback paths. - scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports --dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot, --finalize, --run. - os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1 script; kine thundering-herd guardrails preserved. - OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure. Execution requires separate board approval; this change is design + dry-run artifact only. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 19:28:46 +00:00
os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515) Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an HA-aware procedure that covers all three CPs. Rationale: after DEV-510 the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs are NoSchedule-tainted, so: - Kine cascade is no longer the driving risk; embedded 3-node etcd tolerates one member down. The kubectl-latency guardrail stays as a soft signal. - CPs host no StatefulSets or single-replica Deployments. The old Phase C batched stateful eviction dance is obsolete. - CP reboots no longer lose the api-server; two peers keep serving. /livez is polled from the operator machine and cross-checked with etcdctl endpoint status from a peer CP. New CP ordering rule: one CP at a time, leader last, health gate between CPs. Refuses to start if another CP is already cordoned. Automation: - New scripts/os-update/update-cp-node.sh <node> is the entry point. Supports --add-swap / --preflight / --drain / --apt / --reboot / --finalize / --run / --dry-run. Phase A also installs etcd-client on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the target. Peer picker prefers a CP that has etcdctl; refuses cp-1 target with a targeted error if no other CP has etcdctl yet. - scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards to update-cp-node.sh k3s-cp-1 "$@" for backwards compat. - CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc. - OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and automation entry point list all point at the new HA-aware artifacts. Verified: bash -n clean on both scripts; --dry-run walks all six phases for each of cp-1/cp-2/cp-3; --preflight against live cluster correctly aborts cp-1 with "install etcdctl on another CP first" and cp-2/cp-3 with "run --add-swap first" (their current state). Execution is board-approval-gated; DEV-497 (cp-1-only executor) is superseded by a successor ticket. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-23 01:53:14 +00:00
**Superseded by:** [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) (DEV-515, 2026-08-23).
docs(infra): design cp-1 OS update procedure (DEV-496) cp-1 is excluded from os-update.sh because a one-shot drain would trigger the kine cascade documented in DEV-495. This adds: - CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful eviction (Phase C), drain+apt (Phase D), reboot with external livez monitor (Phase E), uncordon+verify (Phase F), and rollback paths. - scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports --dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot, --finalize, --run. - os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1 script; kine thundering-herd guardrails preserved. - OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure. Execution requires separate board approval; this change is design + dry-run artifact only. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 19:28:46 +00:00
os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515) Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an HA-aware procedure that covers all three CPs. Rationale: after DEV-510 the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs are NoSchedule-tainted, so: - Kine cascade is no longer the driving risk; embedded 3-node etcd tolerates one member down. The kubectl-latency guardrail stays as a soft signal. - CPs host no StatefulSets or single-replica Deployments. The old Phase C batched stateful eviction dance is obsolete. - CP reboots no longer lose the api-server; two peers keep serving. /livez is polled from the operator machine and cross-checked with etcdctl endpoint status from a peer CP. New CP ordering rule: one CP at a time, leader last, health gate between CPs. Refuses to start if another CP is already cordoned. Automation: - New scripts/os-update/update-cp-node.sh <node> is the entry point. Supports --add-swap / --preflight / --drain / --apt / --reboot / --finalize / --run / --dry-run. Phase A also installs etcd-client on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the target. Peer picker prefers a CP that has etcdctl; refuses cp-1 target with a targeted error if no other CP has etcdctl yet. - scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards to update-cp-node.sh k3s-cp-1 "$@" for backwards compat. - CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc. - OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and automation entry point list all point at the new HA-aware artifacts. Verified: bash -n clean on both scripts; --dry-run walks all six phases for each of cp-1/cp-2/cp-3; --preflight against live cluster correctly aborts cp-1 with "install etcdctl on another CP first" and cp-2/cp-3 with "run --add-swap first" (their current state). Execution is board-approval-gated; DEV-497 (cp-1-only executor) is superseded by a successor ticket. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-23 01:53:14 +00:00
**Why:** this document was written when cp-1 was the **sole** control-plane node and the datastore was embedded SQLite via kine. After [DEV-510](/DEV/issues/DEV-510) the cluster runs HA embedded etcd across `k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`, and all three CPs are `NoSchedule`-tainted (they host no StatefulSets or single-replica Deployments anymore). The single-CP kine cascade risk, the batched stateful eviction dance, and the external-`/livez` polling pattern no longer describe the current cluster.
docs(infra): design cp-1 OS update procedure (DEV-496) cp-1 is excluded from os-update.sh because a one-shot drain would trigger the kine cascade documented in DEV-495. This adds: - CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful eviction (Phase C), drain+apt (Phase D), reboot with external livez monitor (Phase E), uncordon+verify (Phase F), and rollback paths. - scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports --dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot, --finalize, --run. - os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1 script; kine thundering-herd guardrails preserved. - OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure. Execution requires separate board approval; this change is design + dry-run artifact only. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 19:28:46 +00:00
os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515) Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an HA-aware procedure that covers all three CPs. Rationale: after DEV-510 the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs are NoSchedule-tainted, so: - Kine cascade is no longer the driving risk; embedded 3-node etcd tolerates one member down. The kubectl-latency guardrail stays as a soft signal. - CPs host no StatefulSets or single-replica Deployments. The old Phase C batched stateful eviction dance is obsolete. - CP reboots no longer lose the api-server; two peers keep serving. /livez is polled from the operator machine and cross-checked with etcdctl endpoint status from a peer CP. New CP ordering rule: one CP at a time, leader last, health gate between CPs. Refuses to start if another CP is already cordoned. Automation: - New scripts/os-update/update-cp-node.sh <node> is the entry point. Supports --add-swap / --preflight / --drain / --apt / --reboot / --finalize / --run / --dry-run. Phase A also installs etcd-client on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the target. Peer picker prefers a CP that has etcdctl; refuses cp-1 target with a targeted error if no other CP has etcdctl yet. - scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards to update-cp-node.sh k3s-cp-1 "$@" for backwards compat. - CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc. - OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and automation entry point list all point at the new HA-aware artifacts. Verified: bash -n clean on both scripts; --dry-run walks all six phases for each of cp-1/cp-2/cp-3; --preflight against live cluster correctly aborts cp-1 with "install etcdctl on another CP first" and cp-2/cp-3 with "run --add-swap first" (their current state). Execution is board-approval-gated; DEV-497 (cp-1-only executor) is superseded by a successor ticket. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-23 01:53:14 +00:00
The generalized procedure covers cp-1 and cp-2 and cp-3. Use it.
docs(infra): design cp-1 OS update procedure (DEV-496) cp-1 is excluded from os-update.sh because a one-shot drain would trigger the kine cascade documented in DEV-495. This adds: - CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful eviction (Phase C), drain+apt (Phase D), reboot with external livez monitor (Phase E), uncordon+verify (Phase F), and rollback paths. - scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports --dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot, --finalize, --run. - os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1 script; kine thundering-herd guardrails preserved. - OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure. Execution requires separate board approval; this change is design + dry-run artifact only. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 19:28:46 +00:00
os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515) Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an HA-aware procedure that covers all three CPs. Rationale: after DEV-510 the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs are NoSchedule-tainted, so: - Kine cascade is no longer the driving risk; embedded 3-node etcd tolerates one member down. The kubectl-latency guardrail stays as a soft signal. - CPs host no StatefulSets or single-replica Deployments. The old Phase C batched stateful eviction dance is obsolete. - CP reboots no longer lose the api-server; two peers keep serving. /livez is polled from the operator machine and cross-checked with etcdctl endpoint status from a peer CP. New CP ordering rule: one CP at a time, leader last, health gate between CPs. Refuses to start if another CP is already cordoned. Automation: - New scripts/os-update/update-cp-node.sh <node> is the entry point. Supports --add-swap / --preflight / --drain / --apt / --reboot / --finalize / --run / --dry-run. Phase A also installs etcd-client on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the target. Peer picker prefers a CP that has etcdctl; refuses cp-1 target with a targeted error if no other CP has etcdctl yet. - scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards to update-cp-node.sh k3s-cp-1 "$@" for backwards compat. - CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc. - OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and automation entry point list all point at the new HA-aware artifacts. Verified: bash -n clean on both scripts; --dry-run walks all six phases for each of cp-1/cp-2/cp-3; --preflight against live cluster correctly aborts cp-1 with "install etcdctl on another CP first" and cp-2/cp-3 with "run --add-swap first" (their current state). Execution is board-approval-gated; DEV-497 (cp-1-only executor) is superseded by a successor ticket. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-23 01:53:14 +00:00
**Scripts:**
docs(infra): design cp-1 OS update procedure (DEV-496) cp-1 is excluded from os-update.sh because a one-shot drain would trigger the kine cascade documented in DEV-495. This adds: - CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful eviction (Phase C), drain+apt (Phase D), reboot with external livez monitor (Phase E), uncordon+verify (Phase F), and rollback paths. - scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports --dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot, --finalize, --run. - os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1 script; kine thundering-herd guardrails preserved. - OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure. Execution requires separate board approval; this change is design + dry-run artifact only. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 19:28:46 +00:00
os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515) Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an HA-aware procedure that covers all three CPs. Rationale: after DEV-510 the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs are NoSchedule-tainted, so: - Kine cascade is no longer the driving risk; embedded 3-node etcd tolerates one member down. The kubectl-latency guardrail stays as a soft signal. - CPs host no StatefulSets or single-replica Deployments. The old Phase C batched stateful eviction dance is obsolete. - CP reboots no longer lose the api-server; two peers keep serving. /livez is polled from the operator machine and cross-checked with etcdctl endpoint status from a peer CP. New CP ordering rule: one CP at a time, leader last, health gate between CPs. Refuses to start if another CP is already cordoned. Automation: - New scripts/os-update/update-cp-node.sh <node> is the entry point. Supports --add-swap / --preflight / --drain / --apt / --reboot / --finalize / --run / --dry-run. Phase A also installs etcd-client on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the target. Peer picker prefers a CP that has etcdctl; refuses cp-1 target with a targeted error if no other CP has etcdctl yet. - scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards to update-cp-node.sh k3s-cp-1 "$@" for backwards compat. - CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc. - OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and automation entry point list all point at the new HA-aware artifacts. Verified: bash -n clean on both scripts; --dry-run walks all six phases for each of cp-1/cp-2/cp-3; --preflight against live cluster correctly aborts cp-1 with "install etcdctl on another CP first" and cp-2/cp-3 with "run --add-swap first" (their current state). Execution is board-approval-gated; DEV-497 (cp-1-only executor) is superseded by a successor ticket. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-23 01:53:14 +00:00
- `infrastructure/scripts/os-update/update-cp-node.sh <node>` — the HA-aware entry point.
- `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compatibility with any existing runbook that names it explicitly.
docs(infra): design cp-1 OS update procedure (DEV-496) cp-1 is excluded from os-update.sh because a one-shot drain would trigger the kine cascade documented in DEV-495. This adds: - CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful eviction (Phase C), drain+apt (Phase D), reboot with external livez monitor (Phase E), uncordon+verify (Phase F), and rollback paths. - scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports --dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot, --finalize, --run. - os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1 script; kine thundering-herd guardrails preserved. - OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure. Execution requires separate board approval; this change is design + dry-run artifact only. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 19:28:46 +00:00
os-update: generalize CP procedure to HA cp-1/cp-2/cp-3 (DEV-515) Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an HA-aware procedure that covers all three CPs. Rationale: after DEV-510 the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs are NoSchedule-tainted, so: - Kine cascade is no longer the driving risk; embedded 3-node etcd tolerates one member down. The kubectl-latency guardrail stays as a soft signal. - CPs host no StatefulSets or single-replica Deployments. The old Phase C batched stateful eviction dance is obsolete. - CP reboots no longer lose the api-server; two peers keep serving. /livez is polled from the operator machine and cross-checked with etcdctl endpoint status from a peer CP. New CP ordering rule: one CP at a time, leader last, health gate between CPs. Refuses to start if another CP is already cordoned. Automation: - New scripts/os-update/update-cp-node.sh <node> is the entry point. Supports --add-swap / --preflight / --drain / --apt / --reboot / --finalize / --run / --dry-run. Phase A also installs etcd-client on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the target. Peer picker prefers a CP that has etcdctl; refuses cp-1 target with a targeted error if no other CP has etcdctl yet. - scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards to update-cp-node.sh k3s-cp-1 "$@" for backwards compat. - CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc. - OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and automation entry point list all point at the new HA-aware artifacts. Verified: bash -n clean on both scripts; --dry-run walks all six phases for each of cp-1/cp-2/cp-3; --preflight against live cluster correctly aborts cp-1 with "install etcdctl on another CP first" and cp-2/cp-3 with "run --add-swap first" (their current state). Execution is board-approval-gated; DEV-497 (cp-1-only executor) is superseded by a successor ticket. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-23 01:53:14 +00:00
**History note:** the original phase design (add swap → preflight → batched stateful eviction → drain → apt → reboot → finalize) is preserved in git history at commit `19af691` and `989033d` for future reference. The new procedure retains Phases A/B/D/E/F essentially unchanged and drops Phase C (batched stateful eviction) as obsolete.
Reference in a new issue Copy permalink