feat(os-update): reconcile Docker + label after node updates (DEV-499)
The weekly rolling OS-update cycle has been silently stripping docker.io from worker nodes (DEV-498), breaking the Forgejo runner whose hostPath mount for /var/run/docker.sock requires the socket to exist. Fix in two layers: - Defensive pin: update-node.sh now runs `apt-mark manual docker.io` in the apt phase (whenever it is installed) so `apt-get autoremove --purge` cannot silently drop it during subsequent upgrades. - Post-reboot reconciliation: new `ensure-node-docker.sh` installs docker.io if missing, enables + starts the systemd unit, waits for /var/run/docker.sock, and re-applies the `basicstack.de/docker=true` label. Wired into update-node.sh between kubelet-Ready and uncordon. No-op on nodes without the label (safe for cp-1 and the update runner). Verified idempotent against all 5 labeled workers; `apt-mark manual docker.io` now set on every worker (survived across reboots by design). Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
8dffba2c52
commit
19af6918f0
4 changed files with 142 additions and 2 deletions
|
|
@ -205,6 +205,16 @@ kubectl get node "$NODE"
|
|||
|
||||
If Ready never returns True: escalate. Do NOT change k3s config.
|
||||
|
||||
### 5b. Reconcile Docker on labeled nodes (before uncordon)
|
||||
|
||||
For nodes carrying the `basicstack.de/docker=true` label (currently the workers that run the Forgejo runner), ensure `docker.io` is present + running before scheduling resumes. Without this the runner pod comes back `ContainerCreating` because `hostPath` requires the docker socket to exist (see [DEV-498](/DEV/issues/DEV-498) and [DEV-499](/DEV/issues/DEV-499)).
|
||||
|
||||
```bash
|
||||
scripts/os-update/ensure-node-docker.sh "$NODE"
|
||||
```
|
||||
|
||||
This step is a no-op on nodes without the label. `update-node.sh` runs it automatically between step 5 and step 6; the script is also safe to run ad-hoc after any manual OS operation. As a defensive extra measure, the apt phase (step 3) now runs `apt-mark manual docker.io` on any node where it is installed, so `apt-get autoremove --purge` cannot silently strip it.
|
||||
|
||||
### 6. Uncordon
|
||||
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -9,7 +9,8 @@ Scripts that implement the weekly rolling Ubuntu OS-update procedure.
|
|||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `cluster-health.sh` | Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node. |
|
||||
| `update-node.sh <node>` | Drain, apt-update, reboot-if-required, wait for Ready, uncordon, post-node health check. Retriable per-node. |
|
||||
| `update-node.sh <node>` | Drain, apt-update, reboot-if-required, wait for Ready, ensure Docker on labeled nodes, uncordon, post-node health check. Retriable per-node. |
|
||||
| `ensure-node-docker.sh <node>` | Idempotent: on nodes labeled `basicstack.de/docker=true`, install docker.io if missing, `apt-mark manual`, enable+start the docker service, wait for `/var/run/docker.sock`, re-apply the label. No-op on nodes without the label. Called from `update-node.sh` between "kubelet Ready" and "uncordon"; also runnable ad-hoc for recovery. |
|
||||
| `os-update.sh` | Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest. |
|
||||
|
||||
## Order of operations (encoded in `os-update.sh`)
|
||||
|
|
@ -42,6 +43,9 @@ Scripts that implement the weekly rolling Ubuntu OS-update procedure.
|
|||
|
||||
# Restart a partial cycle from a specific node onward.
|
||||
./os-update.sh --start-from k3s-worker-4
|
||||
|
||||
# Ad-hoc: reconcile Docker on a single node (e.g. after emergency ops).
|
||||
./ensure-node-docker.sh k3s-worker-3
|
||||
```
|
||||
|
||||
Logs land in `/tmp/os-update-<UTC-timestamp>/` on the machine that ran the cycle.
|
||||
|
|
|
|||
112
infrastructure/scripts/os-update/ensure-node-docker.sh
Executable file
112
infrastructure/scripts/os-update/ensure-node-docker.sh
Executable file
|
|
@ -0,0 +1,112 @@
|
|||
#!/bin/bash
|
||||
# ensure-node-docker.sh — ensure docker.io is installed, enabled, and running on
|
||||
# a node that carries (or should carry) the `basicstack.de/docker=true` label.
|
||||
#
|
||||
# Runs from the operator machine (or the control plane); needs kubectl and
|
||||
# ssh access to root@<node-ssh-target>.
|
||||
#
|
||||
# Behavior:
|
||||
# - If the node has label `basicstack.de/docker=true`, ensure docker.io is
|
||||
# installed, marked `apt-mark manual`, systemd `docker` is enabled+active,
|
||||
# and /var/run/docker.sock exists.
|
||||
# - Re-apply the label (idempotent) so that returning nodes always end the
|
||||
# step in a known state.
|
||||
# - If the node does NOT carry the label, this is a no-op — do NOT install
|
||||
# docker on nodes that were not designated to run the Forgejo runner.
|
||||
#
|
||||
# Usage:
|
||||
# ensure-node-docker.sh <node-name>
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 — node is either not designated (no label) or Docker is confirmed healthy
|
||||
# 1 — hard failure: label present but docker could not be brought up
|
||||
#
|
||||
# Motivated by DEV-499: the weekly rolling OS update was purging docker.io from
|
||||
# workers, which broke the Forgejo runner (DEV-498). Making Docker part of the
|
||||
# post-reboot reconciliation removes the manual "apt-get install docker.io &&
|
||||
# systemctl enable --now docker && kubectl label node" step.
|
||||
set -euo pipefail
|
||||
|
||||
NODE="${1:-}"
|
||||
if [ -z "$NODE" ]; then
|
||||
echo "usage: $0 <node-name>" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
|
||||
SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}"
|
||||
DOCKER_LABEL_KEY="basicstack.de/docker"
|
||||
|
||||
log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] ensure-docker/$NODE: $*"; }
|
||||
die() { log "FATAL: $*"; exit 1; }
|
||||
|
||||
# node -> ssh target. Keep in sync with update-node.sh::node_ssh_target.
|
||||
node_ssh_target() {
|
||||
case "$1" in
|
||||
k3s-cp-1) echo "root@178.105.17.239" ;;
|
||||
k3s-worker-1) echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
|
||||
k3s-worker-2) echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
|
||||
k3s-worker-3) echo "root@167.233.121.121" ;;
|
||||
k3s-worker-4) echo "root@128.140.3.80" ;;
|
||||
k3s-worker-5) echo "root@167.233.192.86" ;;
|
||||
k3s-update-runner) echo "root@167.233.79.65" ;;
|
||||
*) die "unknown node $1 — update node_ssh_target() in $0" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# --- 1. is this node designated to run Docker? --------------------------------
|
||||
kubectl get node "$NODE" >/dev/null || die "node $NODE not found in cluster"
|
||||
LABEL_VAL=$(kubectl get node "$NODE" \
|
||||
-o jsonpath="{.metadata.labels.${DOCKER_LABEL_KEY//./\\.}}" 2>/dev/null || echo "")
|
||||
|
||||
if [ "$LABEL_VAL" != "true" ]; then
|
||||
log "no ${DOCKER_LABEL_KEY}=true label — skipping Docker reconciliation"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
SSH_TARGET=$(node_ssh_target "$NODE")
|
||||
log "label ${DOCKER_LABEL_KEY}=true present — reconciling docker.io via $SSH_TARGET"
|
||||
|
||||
# --- 2. ensure docker.io on the node -----------------------------------------
|
||||
REMOTE=$(cat <<'REMOTE'
|
||||
set -euo pipefail
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
|
||||
|
||||
# Recover from any half-finished dpkg state before touching apt.
|
||||
dpkg --configure -a >/dev/null 2>&1 || true
|
||||
|
||||
if ! dpkg -s docker.io >/dev/null 2>&1; then
|
||||
echo "docker.io not installed — installing"
|
||||
apt-get update
|
||||
apt-get $APT_OPTS install docker.io
|
||||
else
|
||||
echo "docker.io already installed"
|
||||
fi
|
||||
|
||||
# Keep docker.io out of the reach of apt-get autoremove --purge, which is what
|
||||
# the weekly OS-update cycle runs. Idempotent.
|
||||
apt-mark manual docker.io >/dev/null
|
||||
|
||||
systemctl enable --now docker
|
||||
|
||||
# Wait for the docker socket to appear so the forgejo-runner pod can bind it.
|
||||
for _ in $(seq 1 30); do
|
||||
if [ -S /var/run/docker.sock ]; then break; fi
|
||||
sleep 1
|
||||
done
|
||||
if [ ! -S /var/run/docker.sock ]; then
|
||||
echo "docker.sock missing after enable" >&2
|
||||
systemctl status docker --no-pager | tail -20 >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "docker OK: $(docker version --format '{{.Server.Version}}')"
|
||||
REMOTE
|
||||
)
|
||||
|
||||
ssh $SSH_OPTS $SSH_TARGET "bash -s" <<< "$REMOTE" | sed 's/^/ /'
|
||||
|
||||
# --- 3. re-apply the label (idempotent, in case someone stripped it) ---------
|
||||
kubectl label node "$NODE" "${DOCKER_LABEL_KEY}=true" --overwrite >/dev/null
|
||||
log "docker + label reconciled OK"
|
||||
|
|
@ -101,6 +101,13 @@ fi
|
|||
|
||||
apt-get update
|
||||
|
||||
# Before any upgrade/autoremove pass, pin docker.io as manually-installed on
|
||||
# nodes where it is present. Without this, apt autoremove --purge has, in the
|
||||
# past, taken docker.io out from under the Forgejo runner (DEV-498/DEV-499).
|
||||
if dpkg -s docker.io >/dev/null 2>&1; then
|
||||
apt-mark manual docker.io >/dev/null
|
||||
fi
|
||||
|
||||
# Try upgrade; on broken deps, one attempt at apt-get -f install then retry.
|
||||
if ! apt-get $APT_OPTS upgrade; then
|
||||
echo "upgrade failed, attempting apt-get -f install"
|
||||
|
|
@ -167,6 +174,14 @@ done
|
|||
[ "$READY" = "True" ] || die "kubelet on $NODE never returned Ready — escalate (do NOT change k3s config)"
|
||||
log " Ready=True"
|
||||
|
||||
# --- 5b. ensure Docker on designated nodes ------------------------------------
|
||||
# Do this BEFORE uncordoning so the runner pod's first scheduling attempt
|
||||
# succeeds instead of racing through ContainerCreating. Idempotent no-op on
|
||||
# nodes that are not labeled basicstack.de/docker=true. (DEV-499)
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
log "[5b/7] ensure docker.io on nodes labeled basicstack.de/docker=true"
|
||||
"$SCRIPT_DIR/ensure-node-docker.sh" "$NODE"
|
||||
|
||||
# --- 6. uncordon --------------------------------------------------------------
|
||||
log "[6/7] uncordon $NODE"
|
||||
kubectl uncordon "$NODE"
|
||||
|
|
@ -175,7 +190,6 @@ kubectl uncordon "$NODE"
|
|||
log "[7/7] post-node settle (${POST_UNCORDON_WAIT_SECONDS}s) + health check"
|
||||
sleep "$POST_UNCORDON_WAIT_SECONDS"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
if RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh"; then
|
||||
log "=== $NODE update: OK ==="
|
||||
else
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue