From 91983e8c8e6eb7b6cc700d0f1aa8e26a407ee1c4 Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 16 Aug 2026 18:35:40 +0000 Subject: [PATCH] docs(os-update): kine thundering-herd guardrails after DEV-495 incident MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add four hard rules to the OS_UPDATE_PROCEDURE: 1. Pre-plan drain order — no drain that evicts >3 StatefulSets at once 2. cp-1 must have swap before the cycle finishes (currently 0 swap, 3.7 GiB RAM) 3. Halt cycle if `kubectl get nodes` from cp-1 exceeds 5 s (kine slowness leading indicator) 4. cp-1 OS update is a separate design task, not part of standard os-update.sh cycle Root cause reference: DEV-495 (worker-3 drain 2026-08-16 caused kine SQLite cascade + taint-eviction storm + near-OOM on cp-1; cluster self-recovered without operator action after ~80 min). Co-Authored-By: Paperclip --- infrastructure/OS_UPDATE_PROCEDURE.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/infrastructure/OS_UPDATE_PROCEDURE.md b/infrastructure/OS_UPDATE_PROCEDURE.md index 76ceebf..b158109 100644 --- a/infrastructure/OS_UPDATE_PROCEDURE.md +++ b/infrastructure/OS_UPDATE_PROCEDURE.md @@ -46,6 +46,19 @@ ssh root@178.105.17.239 'kubectl get nodes -o wide' --- +## Kine thundering-herd guardrails (added after 2026-08-16 incident, see DEV-495) + +The k3s control plane uses embedded SQLite (kine) as its datastore. On a single-CP cluster with limited RAM and no swap (cp-1 currently: 3.7 GiB RAM, 0 swap), draining a worker with many StatefulSets triggers a self-amplifying eviction-and-slow-SQL cascade: kine backs up on writes → apiserver hangs → node-lease renewals fail → taint-eviction kicks in on more nodes → more writes → kine falls further behind → OOM risk on cp-1. + +All four rules below MUST be observed on every DEV-478 fire. The reference implementation (`os-update.sh`) does not yet enforce (1)/(3) automatically; the operator must actively watch. + +1. **Pre-plan drain order for StatefulSets.** Before draining any worker, `kubectl get pods -n -o wide` against every namespace with StatefulSets and count how many will be evicted from the target node. If a single drain would evict **more than 3 StatefulSets at once**, redistribute first: cordon+delete individual StatefulSet pods one namespace at a time and wait for each reschedule to settle before draining the whole node. +2. **cp-1 must have swap before finishing the cycle.** cp-1 has 0 swap. Add at least 2 GiB of swap on cp-1 before the cp-1 update step (and ideally before draining the last stateful-heavy worker). This is a one-off setup task; once done it is a durable capability. +3. **Halt on kine slowness.** During any drain, keep a `time kubectl get nodes` running from cp-1. If it exceeds **5 s** in real time, halt the cycle immediately (uncordon the current node, do not proceed), verify cluster health, and escalate. The 5 s threshold is the leading indicator that kine has fallen behind and the taint-eviction cascade is about to start. +4. **cp-1 update is a separate design task.** Draining cp-1's own pods (Stalwart, coredns, harbor-database if there, etc.) plus rebooting the single apiserver is the highest-risk step of the whole cycle. It MUST be planned and approved as a distinct issue before it runs; it is NOT covered by the standard `os-update.sh` cycle in its current form. + +--- + ## Access Prereqs are the same as `CLUSTER_ACCESS.md`: @@ -307,3 +320,4 @@ A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this proce | Date | Change | By | |------|--------|-----| | 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) | +| 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) |