basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

os-update: exclude all control-plane nodes by role, not just cp-1 (DEV-513)

Browse source 
Since DEV-510 landed HA control plane (2026-08-22), the cluster runs cp-1
plus cp-2/cp-3. The previous exclusion in os-update.sh matched `k3s-cp-1`
by hard-coded name only, which would have caused cp-2 and cp-3 to be
treated as regular fsn1 workers and drained/rebooted without the
CP-specific procedure.

Fix: select the CP list from `kubectl get nodes -l
node-role.kubernetes.io/control-plane` and skip any of those nodes. This
covers all present and future CPs automatically.

Also updated OS_UPDATE_PROCEDURE.md topology table and order rule to
document that all three CPs exist and are excluded from the weekly
cycle. The HA-aware CP OS-update procedure is a separate follow-up.

Verified on the current cluster:
  [plan] EXCLUDING control-plane nodes: k3s-cp-1 k3s-cp-2 k3s-cp-3
  [plan] ordered nodes (6): k3s-worker-4 k3s-update-runner k3s-worker-1
                            k3s-worker-2 k3s-worker-3 k3s-worker-5

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-08-23 01:39:25 +00:00
parent 89cf3094d0
commit 34350a03bf
2 changed files with 25 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
infrastructure/OS_UPDATE_PROCEDURE.md
View file

@ -22,7 +22,9 @@
| Role | Node | Private IP | Public IP | Datacenter | Notes | | Role | Node | Private IP | Public IP | Datacenter | Notes |
|------|------|-----------|-----------|------------|-------| |------|------|-----------|-----------|------------|-------|
| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | update LAST | | control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | EXCLUDED from `os-update.sh` — see CP1_UPDATE_PROCEDURE.md |
| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | EXCLUDED — joined 2026-08-22 (DEV-510); needs HA-aware CP OS-update procedure |
| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | EXCLUDED — joined 2026-08-22 (DEV-510); needs HA-aware CP OS-update procedure |
| worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | | | worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | |
| worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | | | worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | |
| worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | | | worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | |
@ -36,11 +38,11 @@ Always re-derive the live list before running — nodes may have been added/remo
ssh root@178.105.17.239 'kubectl get nodes -o wide' ssh root@178.105.17.239 'kubectl get nodes -o wide'
``` ```
**Order rule:** update ALL workers first, control plane LAST. Within workers, update in this order to protect stateful workloads: **Order rule:** update workers only. Within workers, update in this order to protect stateful workloads:
1. runner + workers that host no PVs (safest — lowest disruption) 1. runner + workers that host no PVs (safest — lowest disruption)
2. remaining workers 2. remaining workers
3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time. 3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
4. **k3s-cp-1 excluded** — single control plane; has its own dedicated procedure and script. See `CP1_UPDATE_PROCEDURE.md` and `scripts/os-update/update-cp-1.sh`. 4. **ALL control-plane nodes excluded** — `os-update.sh` selects nodes without the `node-role.kubernetes.io/control-plane` label, so `k3s-cp-1`, `k3s-cp-2`, and `k3s-cp-3` are all skipped automatically. CPs need their own procedure (`CP1_UPDATE_PROCEDURE.md` covers cp-1 today; an HA-aware successor is a follow-up).
**Concurrency:** exactly one node at a time. Never in parallel. **Concurrency:** exactly one node at a time. Never in parallel.

28
infrastructure/scripts/os-update/os-update.sh
View file

@ -70,19 +70,31 @@ fi
# Ordering rule: # Ordering rule:
# - workers only # - workers only
# - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last # - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last
# - k3s-cp-1 is EXCLUDED and never updated by this script — see CP1_UPDATE_PROCEDURE.md # - ALL control-plane nodes are EXCLUDED and never updated by this script. Rationale:
# and scripts/os-update/update-cp-1.sh. The rationale is the kine thundering-herd # * kine/etcd write-path is sensitive to concurrent drains (see kine thundering-herd
# guardrails documented in OS_UPDATE_PROCEDURE.md (added after DEV-495). # guardrails in OS_UPDATE_PROCEDURE.md, added after DEV-495).
# * rebooting a CP removes one apiserver — needs external liveness monitoring.
# * CP nodes host their own StatefulSet workloads that need batched eviction.
# Since DEV-510 (2026-08-22) the cluster runs HA control plane (cp-1/cp-2/cp-3). The
# exclusion here is role-based so ALL current and future CPs are covered automatically.
# To update a CP, use `scripts/os-update/update-cp-1.sh` (currently cp-1-only; will be
# generalized to any CP as part of the HA-aware CP OS-update procedure follow-up).
STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true) STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true)
# If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers. # If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers.
CP_NAME="k3s-cp-1"
CP_NODES=$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[*].metadata.name}' 2>/dev/null | tr ' ' '\n' || true)
ALL_NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n') ALL_NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n')
is_cp() {
local n="$1"
for c in $CP_NODES; do [ "$c" = "$n" ] && return 0; done
return 1
}
workers=() workers=()
stalwart_workers=() stalwart_workers=()
for n in $ALL_NODES; do for n in $ALL_NODES; do
[ "$n" = "$CP_NAME" ] && continue is_cp "$n" && continue
if [ -n "$STALWART_NODE" ] && [ "$n" = "$STALWART_NODE" ]; then if [ -n "$STALWART_NODE" ] && [ "$n" = "$STALWART_NODE" ]; then
stalwart_workers+=("$n") stalwart_workers+=("$n")
continue continue
@ -97,9 +109,9 @@ for n in $ALL_NODES; do
done done
ORDER=("${workers[@]}" "${stalwart_workers[@]}") ORDER=("${workers[@]}" "${stalwart_workers[@]}")
# NOTE: cp-1 intentionally excluded. To update cp-1, run `scripts/os-update/update-cp-1.sh`. # NOTE: CP nodes intentionally excluded. To update a CP, see CP1_UPDATE_PROCEDURE.md.
if printf '%s\n' "$ALL_NODES" | grep -qx "$CP_NAME"; then if [ -n "$CP_NODES" ]; then
log "[plan] EXCLUDING $CP_NAME — use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)" log "[plan] EXCLUDING control-plane nodes: $(echo $CP_NODES | tr '\n' ' ')— use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)"
fi fi
if [ -n "$ONLY" ]; then if [ -n "$ONLY" ]; then