basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

fix(os-update): accept 401/403 as /livez alive in CP reboot poll (DEV-566)

Browse source 
k3s api-server /livez on this cluster is auth-gated and returns HTTP 401
without a bearer token. The Phase E reboot poll hardcoded HTTP 200, so
every CP reboot would time out at 10 min ("api-server did not return
within 600s") even though cp-2 was actually Ready and etcd-rejoined
within ~90 s.

Any HTTP response from the api-server (200/401/403) means it is up and
serving; only network-level failures (000, connection refused) count as
"not yet back". Added --max-time 5 to keep polling snappy immediately
after the reboot when TCP may accept-then-drop.

Verified during DEV-566 cycle:
  curl -sk -o/dev/null -w '%{http_code}' https://<cp>:6443/livez -> 401
on cp-1, cp-2, cp-3 (all Ready).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO 2026-09-06 01:47:47 +00:00
parent b7e34236db
commit 0e796dff14
1 changed files with 15 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

21
infrastructure/scripts/os-update/update-cp-node.sh
View file

@ -473,15 +473,24 @@ phase_reboot() {
if [ "$DRY_RUN" != "1" ]; then
local deadline=$(( $(date +%s) + REBOOT_MAX_WAIT_SECONDS ))
local code=000
# /livez is auth-gated on this cluster (returns 401 without a bearer token).
# Any HTTP response from the api-server (200/401/403) means it is up and
# serving; only network-level failures (000, connection refused) count as
# "not yet back". curl --max-time keeps the poll cycle snappy after reboot.
while [ $(date +%s) -lt $deadline ]; do
code=$(curl -sk -o /dev/null -w '%{http_code}' "https://$TARGET_HOST:6443/livez" 2>/dev/null || echo 000)
if [ "$code" = "200" ]; then
log " $NODE api-server /livez=200"
break
fi
code=$(curl -sk -o /dev/null -w '%{http_code}' --max-time 5 "https://$TARGET_HOST:6443/livez" 2>/dev/null || echo 000)
case "$code" in
200|401|403)
log " $NODE api-server responding (/livez HTTP $code)"
break
;;
esac
sleep 5
done
[ "$code" = "200" ] || die "$NODE api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s -- escalate; check 'hcloud server describe $NODE' and Hetzner console"
case "$code" in
200|401|403) : ;;
*) die "$NODE api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s (last code=$code) -- escalate; check 'hcloud server describe $NODE' and Hetzner console" ;;
esac
fi
log "-- verify etcd cluster status from peer ($PEER_HOST) -- $NODE should be 'started'"