diff --git a/infrastructure/OS_UPDATE_PROCEDURE.md b/infrastructure/OS_UPDATE_PROCEDURE.md index 42b63f3..56b2342 100644 --- a/infrastructure/OS_UPDATE_PROCEDURE.md +++ b/infrastructure/OS_UPDATE_PROCEDURE.md @@ -84,6 +84,12 @@ Environment variables the scripts expect: - Drain timeout: `DRAIN_TIMEOUT_SECONDS` — default `600` - Reboot wait: `REBOOT_MAX_WAIT_SECONDS` — default `600` - Post-uncordon settle: `POST_UNCORDON_WAIT_SECONDS` — default `180` +- Post-uncordon bounded health retry (DEV-595): + - `HEALTH_MAX_RETRIES` — default `10` (extra attempts after the first) + - `HEALTH_RETRY_INTERVAL_SECONDS` — default `30` + - Short-circuits immediately on persistent pod errors + (`CrashLoopBackOff`, `ImagePullBackOff`, `ErrImagePull`, + `CreateContainerConfigError`, `InvalidImageName`). --- diff --git a/infrastructure/scripts/os-update/README.md b/infrastructure/scripts/os-update/README.md index 51bc742..bcf86b3 100644 --- a/infrastructure/scripts/os-update/README.md +++ b/infrastructure/scripts/os-update/README.md @@ -27,7 +27,7 @@ Scripts that implement the weekly rolling Ubuntu OS-update procedure. - Drain with a PDB conflict → uncordon and mark the node `SKIPPED_DRAIN_FAILED`, never `--force`. - Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config. - kubelet doesn't return `Ready` → hard stop, escalate. Do NOT touch k3s config. -- Post-node cluster health check red → hard stop, do not proceed to the next node. +- Post-node cluster health check red → first retry up to `HEALTH_MAX_RETRIES` (default 10) more times at `HEALTH_RETRY_INTERVAL_SECONDS` (default 30s) to let transient reinit windows settle (DEV-595). Short-circuit halt immediately on any `CrashLoopBackOff`/`ImagePullBackOff`/`ErrImagePull`/`CreateContainerConfigError`/`InvalidImageName`. Still red after all attempts → hard stop, do not proceed to the next node. - Absolute rule: **no k3s config, no manifests, no PVs, no service files touched** — apt/dpkg only. ## Typical invocations diff --git a/infrastructure/scripts/os-update/update-node.sh b/infrastructure/scripts/os-update/update-node.sh index d72ee64..af7743a 100755 --- a/infrastructure/scripts/os-update/update-node.sh +++ b/infrastructure/scripts/os-update/update-node.sh @@ -15,6 +15,8 @@ # DRAIN_TIMEOUT_SECONDS (default 600) # REBOOT_MAX_WAIT_SECONDS (default 600) # POST_UNCORDON_WAIT_SECONDS (default 180) +# HEALTH_MAX_RETRIES (default 10) — extra post-uncordon health retries +# HEALTH_RETRY_INTERVAL_SECONDS (default 30) — delay between those retries # CONTROL_PLANE_HOST (default 178.105.17.239) # SSH_OPTS (default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new") # ASSUME_YES=1 to skip interactive confirmations @@ -29,9 +31,15 @@ fi DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}" REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}" POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}" +HEALTH_MAX_RETRIES="${HEALTH_MAX_RETRIES:-10}" +HEALTH_RETRY_INTERVAL_SECONDS="${HEALTH_RETRY_INTERVAL_SECONDS:-30}" CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}" SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}" +# Pod-level signatures that do not recover on their own — short-circuit the +# post-uncordon retry loop and halt immediately (DEV-595). +PERSISTENT_FAILURE_PATTERN='CrashLoopBackOff|ImagePullBackOff|ErrImagePull|CreateContainerConfigError|InvalidImageName' + log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*"; } die() { log "FATAL: $*"; exit 1; } @@ -188,11 +196,39 @@ log "[6/7] uncordon $NODE" kubectl uncordon "$NODE" # --- 7. post-node settle ------------------------------------------------------ +# First check at POST_UNCORDON_WAIT_SECONDS, then up to HEALTH_MAX_RETRIES more +# attempts at HEALTH_RETRY_INTERVAL_SECONDS. Transient yellow signals (Init:*, +# Running-but-not-Ready, deployment-below-replicas during reschedule) often +# clear within a minute or two; previously they tripped a FATAL on the first +# check. A pod in CrashLoopBackOff (or image-pull failure) is persistent and +# short-circuits the loop immediately. (DEV-595) log "[7/7] post-node settle (${POST_UNCORDON_WAIT_SECONDS}s) + health check" sleep "$POST_UNCORDON_WAIT_SECONDS" -if RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh"; then - log "=== $NODE update: OK ===" -else - die "cluster health failed after updating $NODE — STOP the cycle, do NOT continue" -fi +attempt=1 +max_attempts=$((HEALTH_MAX_RETRIES + 1)) +while :; do + set +e + HEALTH_OUT=$(RETRY_ON_TRANSIENT=0 "$SCRIPT_DIR/cluster-health.sh" 2>&1) + HEALTH_RC=$? + set -e + + echo "$HEALTH_OUT" + + if [ "$HEALTH_RC" -eq 0 ]; then + log "=== $NODE update: OK (health attempt $attempt/$max_attempts) ===" + break + fi + + if echo "$HEALTH_OUT" | grep -qE "$PERSISTENT_FAILURE_PATTERN"; then + die "cluster health failed after updating $NODE with persistent pod error (CrashLoopBackOff/ImagePullBackOff/etc.) on attempt $attempt — STOP the cycle, do NOT continue" + fi + + if [ "$attempt" -ge "$max_attempts" ]; then + die "cluster health still failing after $attempt attempts (interval ${HEALTH_RETRY_INTERVAL_SECONDS}s) following update of $NODE — STOP the cycle, do NOT continue" + fi + + log " cluster health yellow on attempt $attempt/$max_attempts (no persistent signatures) — retrying in ${HEALTH_RETRY_INTERVAL_SECONDS}s" + sleep "$HEALTH_RETRY_INTERVAL_SECONDS" + attempt=$((attempt + 1)) +done