diff --git a/infrastructure/OS_UPDATE_PROCEDURE.md b/infrastructure/OS_UPDATE_PROCEDURE.md index 76ceebf..b158109 100644 --- a/infrastructure/OS_UPDATE_PROCEDURE.md +++ b/infrastructure/OS_UPDATE_PROCEDURE.md @@ -46,6 +46,19 @@ ssh root@178.105.17.239 'kubectl get nodes -o wide' --- +## Kine thundering-herd guardrails (added after 2026-08-16 incident, see DEV-495) + +The k3s control plane uses embedded SQLite (kine) as its datastore. On a single-CP cluster with limited RAM and no swap (cp-1 currently: 3.7 GiB RAM, 0 swap), draining a worker with many StatefulSets triggers a self-amplifying eviction-and-slow-SQL cascade: kine backs up on writes → apiserver hangs → node-lease renewals fail → taint-eviction kicks in on more nodes → more writes → kine falls further behind → OOM risk on cp-1. + +All four rules below MUST be observed on every DEV-478 fire. The reference implementation (`os-update.sh`) does not yet enforce (1)/(3) automatically; the operator must actively watch. + +1. **Pre-plan drain order for StatefulSets.** Before draining any worker, `kubectl get pods -n -o wide` against every namespace with StatefulSets and count how many will be evicted from the target node. If a single drain would evict **more than 3 StatefulSets at once**, redistribute first: cordon+delete individual StatefulSet pods one namespace at a time and wait for each reschedule to settle before draining the whole node. +2. **cp-1 must have swap before finishing the cycle.** cp-1 has 0 swap. Add at least 2 GiB of swap on cp-1 before the cp-1 update step (and ideally before draining the last stateful-heavy worker). This is a one-off setup task; once done it is a durable capability. +3. **Halt on kine slowness.** During any drain, keep a `time kubectl get nodes` running from cp-1. If it exceeds **5 s** in real time, halt the cycle immediately (uncordon the current node, do not proceed), verify cluster health, and escalate. The 5 s threshold is the leading indicator that kine has fallen behind and the taint-eviction cascade is about to start. +4. **cp-1 update is a separate design task.** Draining cp-1's own pods (Stalwart, coredns, harbor-database if there, etc.) plus rebooting the single apiserver is the highest-risk step of the whole cycle. It MUST be planned and approved as a distinct issue before it runs; it is NOT covered by the standard `os-update.sh` cycle in its current form. + +--- + ## Access Prereqs are the same as `CLUSTER_ACCESS.md`: @@ -307,3 +320,4 @@ A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this proce | Date | Change | By | |------|--------|-----| | 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) | +| 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) |