feat(os-update): bounded retry on post-uncordon health gate (DEV-595)

The 180s post-uncordon health check in `update-node.sh` step 7 was single-shot.
Any transient yellow signal (Init:*, Running-but-not-Ready, deployment below
replicas during reschedule) at the exact moment of the check would FATAL the
whole cycle and require operator wake + manual `--start-from <next-node>`.
During DEV-593 this fired three times in one cycle (opencloud init, harbor
jobservice reinit) for conditions that cleared within seconds.

Wrap the health check in a bounded retry loop:
- First check at `POST_UNCORDON_WAIT_SECONDS` (default 180s, unchanged).
- Up to `HEALTH_MAX_RETRIES` (default 10) more attempts at
  `HEALTH_RETRY_INTERVAL_SECONDS` (default 30s) = +5 min max.
- Short-circuit halt immediately on persistent pod errors
  (CrashLoopBackOff, ImagePullBackOff, ErrImagePull,
  CreateContainerConfigError, InvalidImageName) — these don't recover on
  their own and shouldn't burn the retry budget.
- Each retry calls `cluster-health.sh` with `RETRY_ON_TRANSIENT=0` so the
  outer loop controls cadence (no double 30s sleeps per attempt).

No k3s or manifest changes. Preflight path is untouched.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

infrastructure/OS_UPDATE_PROCEDURE.md

@@ -84,6 +84,12 @@ Environment variables the scripts expect:
 - Drain timeout: `DRAIN_TIMEOUT_SECONDS` — default `600`
 - Reboot wait: `REBOOT_MAX_WAIT_SECONDS` — default `600`
 - Post-uncordon settle: `POST_UNCORDON_WAIT_SECONDS` — default `180`
+- Post-uncordon bounded health retry (DEV-595):
+  - `HEALTH_MAX_RETRIES` — default `10` (extra attempts after the first)
+  - `HEALTH_RETRY_INTERVAL_SECONDS` — default `30`
+  - Short-circuits immediately on persistent pod errors
+    (`CrashLoopBackOff`, `ImagePullBackOff`, `ErrImagePull`,
+    `CreateContainerConfigError`, `InvalidImageName`).
 
 ---
 

infrastructure/scripts/os-update/README.md

@@ -27,7 +27,7 @@ Scripts that implement the weekly rolling Ubuntu OS-update procedure.
 - Drain with a PDB conflict → uncordon and mark the node `SKIPPED_DRAIN_FAILED`, never `--force`.
 - Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config.
 - kubelet doesn't return `Ready` → hard stop, escalate. Do NOT touch k3s config.
-- Post-node cluster health check red → hard stop, do not proceed to the next node.
+- Post-node cluster health check red → first retry up to `HEALTH_MAX_RETRIES` (default 10) more times at `HEALTH_RETRY_INTERVAL_SECONDS` (default 30s) to let transient reinit windows settle (DEV-595). Short-circuit halt immediately on any `CrashLoopBackOff`/`ImagePullBackOff`/`ErrImagePull`/`CreateContainerConfigError`/`InvalidImageName`. Still red after all attempts → hard stop, do not proceed to the next node.
 - Absolute rule: **no k3s config, no manifests, no PVs, no service files touched** — apt/dpkg only.
 
 ## Typical invocations

infrastructure/scripts/os-update/update-node.sh

@@ -15,6 +15,8 @@
 #   DRAIN_TIMEOUT_SECONDS (default 600)
 #   REBOOT_MAX_WAIT_SECONDS (default 600)
 #   POST_UNCORDON_WAIT_SECONDS (default 180)
+#   HEALTH_MAX_RETRIES (default 10)             — extra post-uncordon health retries
+#   HEALTH_RETRY_INTERVAL_SECONDS (default 30)  — delay between those retries
 #   CONTROL_PLANE_HOST (default 178.105.17.239)
 #   SSH_OPTS (default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new")
 #   ASSUME_YES=1 to skip interactive confirmations
@@ -29,9 +31,15 @@ fi
 DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}"
 REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}"
 POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}"
+HEALTH_MAX_RETRIES="${HEALTH_MAX_RETRIES:-10}"
+HEALTH_RETRY_INTERVAL_SECONDS="${HEALTH_RETRY_INTERVAL_SECONDS:-30}"
 CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
 SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}"
 
+# Pod-level signatures that do not recover on their own — short-circuit the
+# post-uncordon retry loop and halt immediately (DEV-595).
+PERSISTENT_FAILURE_PATTERN='CrashLoopBackOff|ImagePullBackOff|ErrImagePull|CreateContainerConfigError|InvalidImageName'
+
 log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*"; }
 die() { log "FATAL: $*"; exit 1; }
 
@@ -188,11 +196,39 @@ log "[6/7] uncordon $NODE"
 kubectl uncordon "$NODE"
 
 # --- 7. post-node settle ------------------------------------------------------
+# First check at POST_UNCORDON_WAIT_SECONDS, then up to HEALTH_MAX_RETRIES more
+# attempts at HEALTH_RETRY_INTERVAL_SECONDS. Transient yellow signals (Init:*,
+# Running-but-not-Ready, deployment-below-replicas during reschedule) often
+# clear within a minute or two; previously they tripped a FATAL on the first
+# check. A pod in CrashLoopBackOff (or image-pull failure) is persistent and
+# short-circuits the loop immediately. (DEV-595)
 log "[7/7] post-node settle (${POST_UNCORDON_WAIT_SECONDS}s) + health check"
 sleep "$POST_UNCORDON_WAIT_SECONDS"
 
-if RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh"; then
+attempt=1
-  log "=== $NODE update: OK ==="
+max_attempts=$((HEALTH_MAX_RETRIES + 1))
-else
+while :; do
-  die "cluster health failed after updating $NODE — STOP the cycle, do NOT continue"
+  set +e
-fi
+  HEALTH_OUT=$(RETRY_ON_TRANSIENT=0 "$SCRIPT_DIR/cluster-health.sh" 2>&1)
+  HEALTH_RC=$?
+  set -e
+
+  echo "$HEALTH_OUT"
+
+  if [ "$HEALTH_RC" -eq 0 ]; then
+    log "=== $NODE update: OK (health attempt $attempt/$max_attempts) ==="
+    break
+  fi
+
+  if echo "$HEALTH_OUT" | grep -qE "$PERSISTENT_FAILURE_PATTERN"; then
+    die "cluster health failed after updating $NODE with persistent pod error (CrashLoopBackOff/ImagePullBackOff/etc.) on attempt $attempt — STOP the cycle, do NOT continue"
+  fi
+
+  if [ "$attempt" -ge "$max_attempts" ]; then
+    die "cluster health still failing after $attempt attempts (interval ${HEALTH_RETRY_INTERVAL_SECONDS}s) following update of $NODE — STOP the cycle, do NOT continue"
+  fi
+
+  log "  cluster health yellow on attempt $attempt/$max_attempts (no persistent signatures) — retrying in ${HEALTH_RETRY_INTERVAL_SECONDS}s"
+  sleep "$HEALTH_RETRY_INTERVAL_SECONDS"
+  attempt=$((attempt + 1))
+done