diff --git a/infrastructure/OS_UPDATE_PROCEDURE.md b/infrastructure/OS_UPDATE_PROCEDURE.md new file mode 100644 index 0000000..76ceebf --- /dev/null +++ b/infrastructure/OS_UPDATE_PROCEDURE.md @@ -0,0 +1,309 @@ +# Weekly Rolling Ubuntu OS Update Procedure + +**Purpose:** Keep every k3s node's Ubuntu OS patched (kernel, security, package updates) with a **rolling, zero-downtime** update — one node at a time, drain → update → reboot → verify → uncordon → cluster health check → next node. + +**Audience:** Agents (currently CTO, optionally a dedicated ClusterOps agent). Also usable manually by an operator. + +**Scope — what this procedure does:** +- Runs `apt-get update`, `apt-get -y upgrade`, `apt-get -y dist-upgrade`, `apt-get -y autoremove` on each cluster node. +- Reboots the node if a reboot is required (kernel/libc updates). +- Drains and cordons each node before touching it, uncordons after verification. +- Verifies the node and cluster are healthy before moving on. + +**Scope — what this procedure MUST NOT do:** +- **Never change the Kubernetes / k3s setup.** Do not touch `/etc/systemd/system/k3s*.service*`, `/etc/rancher/k3s/*`, k3s binary version, k3s config, kube-system manifests, network policies, or any manifest under `infrastructure/`, `apps/`, or applied via ArgoCD. +- **Do not upgrade k3s** here. k3s upgrades are handled separately by system-upgrade-controller (see `K3S_OPERATIONS.md` and `k3s-upgrade/`). +- **Do not delete PersistentVolumes, PVCs, or workload manifests.** +- Do not "fix" application-level problems on a node — that's out of scope. Only fix problems in the OS/apt/reboot layer of the current node being updated. Anything else → stop, report, escalate. + +--- + +## Cluster topology (context) + +| Role | Node | Private IP | Public IP | Datacenter | Notes | +|------|------|-----------|-----------|------------|-------| +| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | update LAST | +| worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | | +| worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | | +| worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | | +| worker | k3s-worker-4 | 10.42.1.6 | 128.140.3.80 | nbg1 | | +| worker | k3s-worker-5 | 10.42.1.7 | 167.233.192.86 | fsn1 | | +| runner | k3s-update-runner | 167.233.79.65 | 167.233.79.65 | fsn1 | k3s-upgrade helper, still an updatable node | + +Always re-derive the live list before running — nodes may have been added/removed: + +```bash +ssh root@178.105.17.239 'kubectl get nodes -o wide' +``` + +**Order rule:** update ALL workers first, control plane LAST. Within workers, update in this order to protect stateful workloads: +1. runner + workers that host no PVs (safest — lowest disruption) +2. remaining workers +3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time. +4. **k3s-cp-1 last** — single control plane; the API server goes away during its reboot. + +**Concurrency:** exactly one node at a time. Never in parallel. + +--- + +## Access + +Prereqs are the same as `CLUSTER_ACCESS.md`: +- SSH key for `root` on every node (jump via control plane for private-IP workers). +- `kubectl` available (either from the operator machine, or by SSH-ing to the control plane and using it there). +- `hcloud` CLI configured (only needed for firewall-related recovery — not for normal runs). + +Environment variables the scripts expect: +- `CONTROL_PLANE_HOST` — default `178.105.17.239` +- `CONTROL_PLANE_PRIVATE` — default `10.42.1.1` +- Drain timeout: `DRAIN_TIMEOUT_SECONDS` — default `600` +- Reboot wait: `REBOOT_MAX_WAIT_SECONDS` — default `600` +- Post-uncordon settle: `POST_UNCORDON_WAIT_SECONDS` — default `180` + +--- + +## Preflight (run once per cycle, before touching any node) + +1. **Cluster is currently healthy.** If any of the checks below fail, STOP and open an issue — do not start OS updates on an already-degraded cluster. + ```bash + ssh root@$CONTROL_PLANE_HOST bash -s <<'EOF' + set -e + kubectl get nodes + echo "--- Not-ready nodes:" + kubectl get nodes -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Ready" and .status!="True")) | .metadata.name' + echo "--- Pods not Running/Completed:" + kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded | grep -v 'STATUS' || echo " (none)" + echo "--- Non-Ready pods (Running but not Ready):" + kubectl get pods -A -o json | jq -r '.items[] | select(.status.phase=="Running") | select([.status.conditions[]?|select(.type=="Ready")|.status]|contains(["False"])) | "\(.metadata.namespace)/\(.metadata.name)"' + EOF + ``` + Only proceed if: all nodes `Ready`, no non-Running/Succeeded pods, no Running-but-not-Ready pods (small transient counts are OK — retry once and continue if it clears). + +2. **Snapshot k3s datastore** (control plane only — this is a checkpoint you can restore etcd from if the control-plane reboot goes badly): + ```bash + ssh root@$CONTROL_PLANE_HOST 'k3s etcd-snapshot save --name pre-os-update-$(date +%Y%m%d)' + ssh root@$CONTROL_PLANE_HOST 'k3s etcd-snapshot list | tail -5' + ``` + +3. **List the nodes to update** and derive the ordered plan. Persist to `/tmp/os-update-plan.txt` on the control plane for auditability. See `scripts/os-update/os-update.sh` for the reference implementation of the ordering rule. + +--- + +## Per-node procedure + +Repeat for each node in the ordered plan. The reference implementation is `infrastructure/scripts/os-update/update-node.sh `; the manual steps below are what that script does. + +Throughout: every command's stdout+stderr goes to a per-run log at `/tmp/os-update--.log` on the control plane. Attach the log to the issue at the end. + +### 1. Pre-check the node + +```bash +NODE= +kubectl get node "$NODE" +kubectl describe node "$NODE" | grep -A2 Conditions +``` + +Confirm: `Ready=True`, not already cordoned, no `DiskPressure`/`MemoryPressure`/`PIDPressure`. + +### 2. Cordon and drain + +```bash +kubectl cordon "$NODE" +kubectl drain "$NODE" \ + --ignore-daemonsets \ + --delete-emptydir-data \ + --disable-eviction=false \ + --timeout="${DRAIN_TIMEOUT_SECONDS:-600}s" +``` + +If drain fails on a PodDisruptionBudget: +- Do NOT force-delete pods (breaks HA guarantees). +- Log the blocking PDB, uncordon the node, mark the node as `SKIPPED_PDB` in the plan, and continue with the next node. Escalate the PDB conflict on the issue at the end. + +If drain fails on a lone pod without a controller: +- Do NOT `--force`. Same as above — uncordon, mark `SKIPPED_ORPHAN_POD`, continue. + +### 3. Update the OS + +On the node itself: + +```bash +ssh -o StrictHostKeyChecking=accept-new root@ bash -s <<'REMOTE' +set -euo pipefail +export DEBIAN_FRONTEND=noninteractive +# Refresh package lists +apt-get update +# Configure apt to keep existing config files silently (no interactive prompts) +APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold' +apt-get $APT_OPTS upgrade +apt-get $APT_OPTS dist-upgrade +apt-get $APT_OPTS autoremove --purge +apt-get clean +# Report reboot need +if [ -f /var/run/reboot-required ]; then + echo "REBOOT_REQUIRED=yes" + cat /var/run/reboot-required.pkgs 2>/dev/null || true +else + echo "REBOOT_REQUIRED=no" +fi +REMOTE +``` + +For private-IP workers, target them from the control plane (`ssh -J root@$CONTROL_PLANE_HOST root@10.42.1.X`) or run the whole block after SSH-ing to the control plane first. + +Common OS-only fixes the agent MAY perform on the node if apt fails: +- `dpkg --configure -a` after an interrupted install +- `apt-get -f install` to resolve broken deps +- Free disk with `journalctl --vacuum-time=3d` or `apt-get clean` if `/` is full +- Restart a system service that is stuck (`systemctl restart `) — but NOT `k3s`, `k3s-agent`, `containerd`, `flanneld`, or any container runtime + +**Never**: change k3s config, delete PVs, uninstall packages the OS didn't schedule, edit `/etc/rancher/`, or reinstall k3s. If a fix would touch any of those, stop and escalate. + +### 4. Reboot if required + +If `REBOOT_REQUIRED=yes`: + +```bash +ssh root@ 'systemctl reboot' || true +# Wait until SSH responds again (max REBOOT_MAX_WAIT_SECONDS) +deadline=$(( $(date +%s) + ${REBOOT_MAX_WAIT_SECONDS:-600} )) +while [ $(date +%s) -lt $deadline ]; do + sleep 10 + if ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new root@ 'uptime' 2>/dev/null; then + echo "Node back up"; break + fi +done +``` + +If the node doesn't come back within the timeout: escalate immediately. Do NOT rebuild the node or touch k3s — a rebuild requires the ADD_WORKER_NODE procedure and is a separate approved action. + +### 5. Wait for k3s-agent / k3s to be ready again + +```bash +# Wait for kubelet Ready condition (control plane view) +deadline=$(( $(date +%s) + 300 )) +while [ $(date +%s) -lt $deadline ]; do + READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}') + [ "$READY" = "True" ] && break + sleep 5 +done +kubectl get node "$NODE" +``` + +If Ready never returns True: escalate. Do NOT change k3s config. + +### 6. Uncordon + +```bash +kubectl uncordon "$NODE" +``` + +### 7. Post-node health verification + +Wait for pods to reschedule and settle, then verify: + +```bash +sleep "${POST_UNCORDON_WAIT_SECONDS:-180}" + +# All nodes Ready? +kubectl get nodes +kubectl get nodes -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Ready" and .status!="True")) | .metadata.name' | grep . && { echo "NOT-READY NODES"; exit 1; } || true + +# Any pod not Running/Succeeded? +BAD=$(kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers 2>/dev/null | wc -l) +[ "$BAD" -gt 0 ] && { echo "BAD PODS: $BAD"; kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded; exit 1; } || true + +# Any Deployment/StatefulSet under desired replicas? +kubectl get deploy -A -o json | jq -r '.items[] | select((.status.readyReplicas // 0) < (.spec.replicas // 1)) | "deploy \(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"' +kubectl get sts -A -o json | jq -r '.items[] | select((.status.readyReplicas // 0) < (.spec.replicas // 1)) | "sts \(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)"' +``` + +Only proceed to the next node when ALL of the above are green. If not: +- Give it another 3 minutes and re-check (workloads with large images may still be pulling). +- If still not green: STOP the cycle. Uncordon everything, leave the cluster in a stable state, and open a follow-up issue with the failing workloads. Do NOT proceed to update more nodes. + +### 8. Control plane special handling + +`k3s-cp-1` is the single control plane node. During its reboot: +- The kube-apiserver is unavailable — kubectl commands from the operator machine will error out. +- The `kubectl` waits in step 5/7 must run from a machine that is NOT the control plane, or must be scheduled after the control plane's SSH is back and `curl -k https://localhost:6443/healthz` returns `ok`. +- Skip the drain for DaemonSet pods on the control plane (`--ignore-daemonsets` covers that), but hosted apps that scheduled onto CP (rare — verify with `kubectl get pods -A --field-selector spec.nodeName=k3s-cp-1`) will be evicted. + +--- + +## Finalization + +After all nodes are done: + +1. Final cluster health snapshot (same preflight commands as at the start). +2. Print apt history summary per node so the audit trail has "what changed": + ```bash + for host in ; do + echo "=== $host ===" + ssh root@$host 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60' + done + ``` +3. Optional: prune old etcd snapshots to keep disk in check: + ```bash + ssh root@$CONTROL_PLANE_HOST 'k3s etcd-snapshot list' + # delete anything older than 14 days if desired (manual review) + ``` +4. Attach the per-run logs to the Paperclip issue that triggered this run. +5. Update the issue with: + - Nodes updated (list) + - Nodes skipped (list + reason) + - Any package that required manual intervention + - Reboots performed + - Final `kubectl get nodes` output + +If everything is clean → mark the issue `done`. +If a node was skipped or errored → mark `blocked` with the unblock action, or open a child issue for the specific failure. + +--- + +## Recovery / what to do when it goes wrong + +**Node stuck cordoned after failure:** `kubectl uncordon ` — always leave the cluster back in its normal scheduling state. + +**Node fails to boot after reboot:** +- Check Hetzner console for boot errors (kernel panic, initramfs). +- Try `hcloud server reboot ` from `hcloud` CLI. +- If the node cannot recover: escalate. Do NOT delete or rebuild the Hetzner server without board approval; that is the ADD_WORKER_NODE flow. + +**k3s-agent won't start after reboot:** +- Check `journalctl -u k3s-agent -n 100`. +- Do NOT edit the k3s-agent unit file. Do NOT re-run the k3s installer. +- Escalate. This is out of scope for the OS-update procedure. + +**Pods CrashLoopBackOff after node came back:** +- Not an OS-update problem to fix — the node is healthy, apt succeeded. Leave the node uncordoned, stop the cycle, and open an application-level issue. + +**PDB blocked drain:** +- Never `--force` the drain. Leave the node uncordoned, skip it, note in the report which PDB blocked and which workload owns it. + +**Datastore snapshot restore (last resort — CP only):** +- Only if the control plane is broken beyond repair. See `K3S_OPERATIONS.md` for the `--cluster-reset --cluster-reset-restore-path=` procedure. Requires board approval — do NOT execute unattended. + +--- + +## Automation entry points + +- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from `. Use `--dry-run` to print the plan without touching anything. +- `infrastructure/scripts/os-update/update-node.sh ` — single-node update (all 7 per-node steps). Callable standalone for retry. +- `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure. + +Read the script sources for the exact behavior before running them. They mirror this procedure step for step. + +--- + +## Weekly schedule + +A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this procedure once per week. The routine creates a task whose description points here. The assigned agent reads this document and executes the automation. + +--- + +## Change history + +| Date | Change | By | +|------|--------|-----| +| 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) | diff --git a/infrastructure/OS_UPDATE_ROUTINE.md b/infrastructure/OS_UPDATE_ROUTINE.md new file mode 100644 index 0000000..1659aaa --- /dev/null +++ b/infrastructure/OS_UPDATE_ROUTINE.md @@ -0,0 +1,47 @@ +# Weekly OS Update — Paperclip Routine + +**Owner:** CTO agent (until a dedicated ClusterOps agent is spun up) +**Schedule:** every **Sunday 03:00 Europe/Berlin** (low-traffic window, before Monday operations) +**Companion procedure:** [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md) + +## What the routine does + +Each fire creates a Paperclip task whose description points to `OS_UPDATE_PROCEDURE.md`. The assigned agent: + +1. Checks out the task. +2. Reads the procedure doc. +3. Runs `infrastructure/scripts/os-update/os-update.sh` end to end. +4. Attaches per-run logs (`/tmp/os-update-/`) to the task. +5. Closes the task `done` on success, or `blocked` with a named unblock action on skip/error. + +## Routine configuration (Paperclip) + +- `title`: "Weekly rolling OS update — k3s cluster nodes" +- `assigneeAgentId`: CTO (agent id `4b5f09a2-22d8-4e3d-9ac4-46d008ad1385`) +- `projectId`: `d24057b8-02ca-41cd-b7a0-6151298c6e2c` (BasicStack Phase-2) +- `goalId`: `b4dfe4a8-5c37-4f62-aad9-fa340dcd34a4` (self-hosted Kubernetes at Hetzner) +- `priority`: `medium` +- `concurrencyPolicy`: `coalesce_if_active` (a previous run still open? merge into it, don't stack) +- `catchUpPolicy`: `skip_missed` (no back-fires if Paperclip was down) +- Trigger: `schedule` — cron `0 3 * * 0`, timezone `Europe/Berlin` + +The task description created by each fire is a link to the procedure doc in Forgejo, so the doc is the single source of truth even if the routine metadata drifts. + +## Reassigning to a dedicated agent + +If we later hire a ClusterOps agent, only two things change: + +1. `PATCH /api/routines/{routineId}` — update `assigneeAgentId`. +2. Nothing in `OS_UPDATE_PROCEDURE.md` or the scripts changes; the procedure is agent-neutral. + +## Pausing / disabling + +To pause without losing config: + +```bash +curl -X PATCH -H "Authorization: Bearer $PAPERCLIP_API_KEY" -H "Content-Type: application/json" \ + -d '{"status":"paused"}' \ + "$PAPERCLIP_API_URL/routines/{routineId}" +``` + +Resume with `{"status":"active"}`. diff --git a/infrastructure/README.md b/infrastructure/README.md index c33298d..e775ae7 100644 --- a/infrastructure/README.md +++ b/infrastructure/README.md @@ -2,6 +2,14 @@ This directory contains cluster-wide infrastructure configurations that support all applications. +## Operational procedures (agent-facing) + +- **[OS_UPDATE_PROCEDURE.md](OS_UPDATE_PROCEDURE.md)** — weekly rolling Ubuntu OS update for all cluster nodes (drain → apt → reboot → verify → uncordon → cluster health → next). Never touches k3s config. +- **[OS_UPDATE_ROUTINE.md](OS_UPDATE_ROUTINE.md)** — the Paperclip routine that fires the above weekly. +- **[K3S_OPERATIONS.md](K3S_OPERATIONS.md)** — k3s version upgrades (separate concern from OS updates). +- **[ADD_WORKER_NODE.md](ADD_WORKER_NODE.md)** — adding a worker node. +- **[CLUSTER_ACCESS.md](CLUSTER_ACCESS.md)** — SSH / kubectl access. + ## Structure ### `networking/` diff --git a/infrastructure/scripts/os-update/README.md b/infrastructure/scripts/os-update/README.md new file mode 100644 index 0000000..a9e3fb5 --- /dev/null +++ b/infrastructure/scripts/os-update/README.md @@ -0,0 +1,47 @@ +# OS Update Automation + +Scripts that implement the weekly rolling Ubuntu OS-update procedure. + +**Authoritative doc:** [`../../OS_UPDATE_PROCEDURE.md`](../../OS_UPDATE_PROCEDURE.md) — read it first. The scripts here mirror that procedure step-for-step. + +## Files + +| Script | Purpose | +|--------|---------| +| `cluster-health.sh` | Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node. | +| `update-node.sh ` | Drain, apt-update, reboot-if-required, wait for Ready, uncordon, post-node health check. Retriable per-node. | +| `os-update.sh` | Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest. | + +## Order of operations (encoded in `os-update.sh`) + +1. Workers with no stateful affinity concerns first. +2. `fsn1` workers (potential Stalwart hosts) last among workers — see [`stalwart-datacenter-affinity`](../../K3S_OPERATIONS.md) note. +3. `k3s-cp-1` last (single control plane). +4. One node at a time. Never in parallel. + +## Guardrails the scripts enforce + +- Preflight cluster health failure → refuse to start. +- Drain with a PDB conflict → uncordon and mark the node `SKIPPED_DRAIN_FAILED`, never `--force`. +- Node reboot fails to come back within timeout → hard stop, escalate. Do NOT rebuild the node or touch k3s config. +- kubelet doesn't return `Ready` → hard stop, escalate. Do NOT touch k3s config. +- Post-node cluster health check red → hard stop, do not proceed to the next node. +- Absolute rule: **no k3s config, no manifests, no PVs, no service files touched** — apt/dpkg only. + +## Typical invocations + +```bash +# Dry-run: print the ordered plan, touch nothing. +./os-update.sh --dry-run + +# Full cycle (weekly, triggered by the Paperclip routine). +./os-update.sh + +# Retry a single node (after fixing a manual issue). +./update-node.sh k3s-worker-3 + +# Restart a partial cycle from a specific node onward. +./os-update.sh --start-from k3s-worker-4 +``` + +Logs land in `/tmp/os-update-/` on the machine that ran the cycle. diff --git a/infrastructure/scripts/os-update/cluster-health.sh b/infrastructure/scripts/os-update/cluster-health.sh new file mode 100755 index 0000000..017e42e --- /dev/null +++ b/infrastructure/scripts/os-update/cluster-health.sh @@ -0,0 +1,130 @@ +#!/bin/bash +# cluster-health.sh — verify k3s cluster is fully healthy. +# Exits 0 on green, 1 on any failure. Used as preflight and post-node check by os-update.sh. +# +# Runs kubectl commands against whatever the current KUBECONFIG resolves to; +# invoke via `ssh root@$CONTROL_PLANE_HOST bash -s < cluster-health.sh` to +# check the cluster from an operator machine without local kubeconfig. +# +# Environment overrides: +# RETRY_ON_TRANSIENT=1 — one retry after 30s for non-Ready-but-Running pods +# VERBOSE=1 — dump full failing rows on non-zero exit + +set -euo pipefail + +fail=0 + +echo "=== nodes ===" +kubectl get nodes -o wide + +not_ready=$(kubectl get nodes -o json | jq -r ' + .items[] + | select(.status.conditions[] | select(.type=="Ready" and .status!="True")) + | .metadata.name +' || true) +if [ -n "$not_ready" ]; then + echo "FAIL: nodes not Ready: $not_ready" + fail=1 +fi + +echo +echo "=== node pressure conditions ===" +pressure=$(kubectl get nodes -o json | jq -r ' + .items[] + | . as $n + | .status.conditions[] + | select(.type=="DiskPressure" or .type=="MemoryPressure" or .type=="PIDPressure") + | select(.status=="True") + | "\($n.metadata.name) \(.type)=True" +' || true) +if [ -n "$pressure" ]; then + echo "FAIL: node pressure: $pressure" + fail=1 +else + echo " (none)" +fi + +check_bad_pods() { + kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers 2>/dev/null || true +} +check_notready_pods() { + kubectl get pods -A -o json | jq -r ' + .items[] + | select(.status.phase=="Running") + | select([.status.conditions[]?|select(.type=="Ready")|.status] | contains(["False"])) + | "\(.metadata.namespace)/\(.metadata.name)" + ' 2>/dev/null || true +} + +echo +echo "=== pods not Running/Succeeded ===" +bad_pods=$(check_bad_pods) +if [ -n "$bad_pods" ]; then + if [ "${RETRY_ON_TRANSIENT:-0}" = "1" ]; then + echo " transient? re-checking in 30s..." + sleep 30 + bad_pods=$(check_bad_pods) + fi +fi +if [ -n "$bad_pods" ]; then + echo "FAIL: pods not Running/Succeeded:" + echo "$bad_pods" + fail=1 +else + echo " (none)" +fi + +echo +echo "=== pods Running but not Ready ===" +notready=$(check_notready_pods) +if [ -n "$notready" ] && [ "${RETRY_ON_TRANSIENT:-0}" = "1" ]; then + echo " transient? re-checking in 30s..." + sleep 30 + notready=$(check_notready_pods) +fi +if [ -n "$notready" ]; then + echo "FAIL: pods Running-but-not-Ready:" + echo "$notready" + fail=1 +else + echo " (none)" +fi + +echo +echo "=== deployments below desired replicas ===" +deploy_bad=$(kubectl get deploy -A -o json | jq -r ' + .items[] + | select((.status.readyReplicas // 0) < (.spec.replicas // 1)) + | "\(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)" +' || true) +if [ -n "$deploy_bad" ]; then + echo "FAIL: deployments not fully ready:" + echo "$deploy_bad" + fail=1 +else + echo " (none)" +fi + +echo +echo "=== statefulsets below desired replicas ===" +sts_bad=$(kubectl get sts -A -o json | jq -r ' + .items[] + | select((.status.readyReplicas // 0) < (.spec.replicas // 1)) + | "\(.metadata.namespace)/\(.metadata.name) \(.status.readyReplicas // 0)/\(.spec.replicas)" +' || true) +if [ -n "$sts_bad" ]; then + echo "FAIL: statefulsets not fully ready:" + echo "$sts_bad" + fail=1 +else + echo " (none)" +fi + +if [ "$fail" -ne 0 ]; then + echo + echo "CLUSTER-HEALTH: FAIL" + exit 1 +fi + +echo +echo "CLUSTER-HEALTH: OK" diff --git a/infrastructure/scripts/os-update/os-update.sh b/infrastructure/scripts/os-update/os-update.sh new file mode 100755 index 0000000..ef1e19d --- /dev/null +++ b/infrastructure/scripts/os-update/os-update.sh @@ -0,0 +1,178 @@ +#!/bin/bash +# os-update.sh — full weekly rolling OS-update cycle. +# +# Behavior: +# 1. preflight cluster health (fail-closed) +# 2. take an etcd snapshot +# 3. compute node order (workers first, control plane last; +# Stalwart-hosting fsn1 workers moved to end of workers group) +# 4. call update-node.sh for each node, halting on any failure +# 5. finalization: health snapshot + apt history digest +# +# Usage: +# os-update.sh [--dry-run] [--start-from ] [--only ] +# +# Environment: +# CONTROL_PLANE_HOST (default 178.105.17.239) +# All env vars honored by update-node.sh are honored here as well. +# +# Read OS_UPDATE_PROCEDURE.md alongside this script; the script mirrors it +# step-for-step and the doc is the authoritative reference. +set -euo pipefail + +CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}" +DRY_RUN=0 +START_FROM="" +ONLY="" + +while [ $# -gt 0 ]; do + case "$1" in + --dry-run) DRY_RUN=1; shift ;; + --start-from) START_FROM="$2"; shift 2 ;; + --only) ONLY="$2"; shift 2 ;; + -h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown arg: $1" >&2; exit 2 ;; + esac +done + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +STAMP=$(date -u +%Y%m%dT%H%M%SZ) +LOG_DIR="/tmp/os-update-${STAMP}" +mkdir -p "$LOG_DIR" + +log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOG_DIR/main.log"; } +die() { log "FATAL: $*"; exit 1; } + +log "=== os-update.sh cycle $STAMP ===" +log "log dir: $LOG_DIR" + +# --- 0. sanity checks --------------------------------------------------------- +command -v kubectl >/dev/null || die "kubectl not on PATH" +command -v jq >/dev/null || die "jq not on PATH (needed for health checks)" + +# --- 1. preflight ------------------------------------------------------------- +log "[preflight] cluster health" +if ! RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh" | tee "$LOG_DIR/preflight.log"; then + die "cluster is not healthy at preflight — refuse to start OS updates" +fi + +# --- 2. etcd snapshot --------------------------------------------------------- +if [ "$DRY_RUN" -eq 0 ]; then + log "[preflight] taking k3s etcd snapshot" + ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new root@"$CONTROL_PLANE_HOST" \ + "k3s etcd-snapshot save --name pre-os-update-$(date +%Y%m%d)" | tee "$LOG_DIR/etcd-snapshot.log" || \ + log "WARN: etcd snapshot failed — continuing but this is a risk on CP reboot" +else + log "[preflight] DRY-RUN — skipping etcd snapshot" +fi + +# --- 3. node ordering --------------------------------------------------------- +# Ordering rule: +# - workers before control plane +# - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last +# - k3s-cp-1 always last +STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true) +# If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers. +CP_NAME="k3s-cp-1" + +ALL_NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n') + +workers=() +stalwart_workers=() +for n in $ALL_NODES; do + [ "$n" = "$CP_NAME" ] && continue + if [ -n "$STALWART_NODE" ] && [ "$n" = "$STALWART_NODE" ]; then + stalwart_workers+=("$n") + continue + fi + # Any fsn1 worker is a potential Stalwart host — schedule after non-Stalwart nodes. + loc=$(kubectl get node "$n" -o jsonpath='{.metadata.labels.csi\.hetzner\.cloud/location}' 2>/dev/null || echo "") + if [ "$loc" = "fsn1" ]; then + stalwart_workers+=("$n") + else + workers+=("$n") + fi +done + +ORDER=("${workers[@]}" "${stalwart_workers[@]}" "$CP_NAME") + +if [ -n "$ONLY" ]; then + ORDER=("$ONLY") +elif [ -n "$START_FROM" ]; then + new=() + skip=1 + for n in "${ORDER[@]}"; do + [ "$n" = "$START_FROM" ] && skip=0 + [ $skip -eq 0 ] && new+=("$n") + done + ORDER=("${new[@]}") +fi + +log "[plan] ordered nodes (${#ORDER[@]}): ${ORDER[*]}" +printf '%s\n' "${ORDER[@]}" > "$LOG_DIR/plan.txt" + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY-RUN — plan written, no node touched. Exiting." + exit 0 +fi + +# --- 4. per-node loop --------------------------------------------------------- +updated=() +skipped=() +for n in "${ORDER[@]}"; do + log "====================================================================" + log "==> updating $n" + log "====================================================================" + NODE_LOG="$LOG_DIR/${n}.log" + set +e + "$SCRIPT_DIR/update-node.sh" "$n" 2>&1 | tee "$NODE_LOG" + rc=${PIPESTATUS[0]} + set -e + case $rc in + 0) updated+=("$n") ;; + 3) skipped+=("$n:drain-blocked") ;; + *) die "update-node.sh failed for $n (rc=$rc). Cycle halted. See $NODE_LOG" ;; + esac +done + +# --- 5. finalization ---------------------------------------------------------- +log "====================================================================" +log "==> finalization" +log "====================================================================" + +log "[final] cluster health" +"$SCRIPT_DIR/cluster-health.sh" | tee "$LOG_DIR/final-health.log" || \ + die "final cluster health failed after cycle. Do NOT declare success." + +log "[final] apt history digest" +{ + for n in "${updated[@]}"; do + echo "=== $n ===" + # resolve ssh target via a mini-eval of node_ssh_target-equivalent + case "$n" in + k3s-cp-1) t="root@178.105.17.239" ;; + k3s-worker-1) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;; + k3s-worker-2) t="-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;; + k3s-worker-3) t="root@167.233.121.121" ;; + k3s-worker-4) t="root@128.140.3.80" ;; + k3s-worker-5) t="root@167.233.192.86" ;; + k3s-update-runner) t="root@167.233.79.65" ;; + *) echo " (unknown ssh target)"; continue ;; + esac + ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new $t \ + 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -40' 2>/dev/null \ + || echo " (could not read apt history)" + done +} | tee "$LOG_DIR/apt-history.log" + +log "=== summary ===" +log "updated (${#updated[@]}): ${updated[*]:-none}" +log "skipped (${#skipped[@]}): ${skipped[*]:-none}" +log "logs: $LOG_DIR" + +if [ ${#skipped[@]} -gt 0 ]; then + log "cycle finished with skipped nodes — return code 4 so the caller can escalate" + exit 4 +fi + +log "cycle complete." diff --git a/infrastructure/scripts/os-update/update-node.sh b/infrastructure/scripts/os-update/update-node.sh new file mode 100755 index 0000000..99580f1 --- /dev/null +++ b/infrastructure/scripts/os-update/update-node.sh @@ -0,0 +1,183 @@ +#!/bin/bash +# update-node.sh — drain, apt-update, reboot-if-needed, wait, uncordon a single node. +# +# Usage: +# update-node.sh +# +# Runs from an operator machine (or from the control plane); needs kubectl and +# ssh access to root@. Node → SSH target resolution is in +# `node_ssh_target` below; edit that mapping when you add nodes. +# +# NEVER touches k3s config, k3s services, containerd, or any manifest. +# Only fixes it will attempt: apt/dpkg recovery on the same node (see step 3). +# +# Environment overrides: +# DRAIN_TIMEOUT_SECONDS (default 600) +# REBOOT_MAX_WAIT_SECONDS (default 600) +# POST_UNCORDON_WAIT_SECONDS (default 180) +# CONTROL_PLANE_HOST (default 178.105.17.239) +# SSH_OPTS (default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new") +# ASSUME_YES=1 to skip interactive confirmations +set -euo pipefail + +NODE="${1:-}" +if [ -z "$NODE" ]; then + echo "usage: $0 " >&2 + exit 2 +fi + +DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}" +REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}" +POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}" +CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}" +SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}" + +log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*"; } +die() { log "FATAL: $*"; exit 1; } + +# node -> ssh target. Private-IP workers are reached via ProxyJump through the CP. +node_ssh_target() { + case "$1" in + k3s-cp-1) echo "root@178.105.17.239" ;; + k3s-worker-1) echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;; + k3s-worker-2) echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;; + k3s-worker-3) echo "root@167.233.121.121" ;; + k3s-worker-4) echo "root@128.140.3.80" ;; + k3s-worker-5) echo "root@167.233.192.86" ;; + k3s-update-runner) echo "root@167.233.79.65" ;; + *) die "unknown node $1 — update node_ssh_target() in $0" ;; + esac +} + +SSH_TARGET=$(node_ssh_target "$NODE") + +log "=== update-node.sh $NODE ===" +log "ssh target: $SSH_TARGET" + +# --- 1. pre-check -------------------------------------------------------------- +log "[1/7] pre-check" +kubectl get node "$NODE" >/dev/null || die "node $NODE not found in cluster" +READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}') +[ "$READY" = "True" ] || die "node $NODE is not Ready before we start (Ready=$READY)" + +for cond in DiskPressure MemoryPressure PIDPressure; do + v=$(kubectl get node "$NODE" -o jsonpath="{.status.conditions[?(@.type==\"$cond\")].status}") + [ "$v" = "True" ] && die "node $NODE has $cond=True — refuse to update" +done + +# --- 2. cordon + drain -------------------------------------------------------- +log "[2/7] cordon + drain (timeout ${DRAIN_TIMEOUT_SECONDS}s)" +kubectl cordon "$NODE" + +set +e +kubectl drain "$NODE" \ + --ignore-daemonsets \ + --delete-emptydir-data \ + --timeout="${DRAIN_TIMEOUT_SECONDS}s" +DRAIN_RC=$? +set -e + +if [ $DRAIN_RC -ne 0 ]; then + log "drain FAILED (rc=$DRAIN_RC). Never force. Uncordoning $NODE and marking SKIPPED." + kubectl uncordon "$NODE" || true + echo "SKIPPED_DRAIN_FAILED $NODE" + exit 3 +fi + +# --- 3. apt update on the node ------------------------------------------------ +log "[3/7] apt update/upgrade on $NODE" +REMOTE_APT=$(cat <<'REMOTE' +set -euo pipefail +export DEBIAN_FRONTEND=noninteractive +APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold' + +# Recover from any half-finished dpkg state before touching apt. +if ! dpkg --audit | grep -qE .; then + : +else + echo "dpkg audit reported issues, running dpkg --configure -a" + dpkg --configure -a || true +fi + +apt-get update + +# Try upgrade; on broken deps, one attempt at apt-get -f install then retry. +if ! apt-get $APT_OPTS upgrade; then + echo "upgrade failed, attempting apt-get -f install" + apt-get $APT_OPTS -f install + apt-get $APT_OPTS upgrade +fi + +apt-get $APT_OPTS dist-upgrade +apt-get $APT_OPTS autoremove --purge +apt-get clean + +if [ -f /var/run/reboot-required ]; then + echo "REBOOT_REQUIRED=yes" + echo "REBOOT_REASON</dev/null || echo "(no package list)" + echo "EOF" +else + echo "REBOOT_REQUIRED=no" +fi +REMOTE +) + +APT_OUT=$(ssh $SSH_OPTS $SSH_TARGET "bash -s" <<< "$REMOTE_APT") +echo "$APT_OUT" | sed 's/^/ /' + +if echo "$APT_OUT" | grep -q '^REBOOT_REQUIRED=yes'; then + REBOOT=1 +else + REBOOT=0 +fi + +# --- 4. reboot if required ---------------------------------------------------- +if [ "$REBOOT" -eq 1 ]; then + log "[4/7] reboot required — rebooting $NODE" + ssh $SSH_OPTS $SSH_TARGET 'systemctl reboot' || true + + # Give SSH a moment to actually drop before we start polling. + sleep 15 + + deadline=$(( $(date +%s) + REBOOT_MAX_WAIT_SECONDS )) + while [ $(date +%s) -lt $deadline ]; do + if ssh $SSH_OPTS -o ConnectTimeout=5 $SSH_TARGET 'uptime' >/dev/null 2>&1; then + log " $NODE ssh is back" + break + fi + sleep 10 + done + + if ! ssh $SSH_OPTS -o ConnectTimeout=5 $SSH_TARGET 'uptime' >/dev/null 2>&1; then + die "node $NODE did not return within ${REBOOT_MAX_WAIT_SECONDS}s — escalate" + fi +else + log "[4/7] no reboot needed" +fi + +# --- 5. wait for kubelet Ready ------------------------------------------------ +log "[5/7] wait for kubelet Ready on $NODE" +deadline=$(( $(date +%s) + 300 )) +while [ $(date +%s) -lt $deadline ]; do + READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown) + [ "$READY" = "True" ] && break + sleep 5 +done +[ "$READY" = "True" ] || die "kubelet on $NODE never returned Ready — escalate (do NOT change k3s config)" +log " Ready=True" + +# --- 6. uncordon -------------------------------------------------------------- +log "[6/7] uncordon $NODE" +kubectl uncordon "$NODE" + +# --- 7. post-node settle ------------------------------------------------------ +log "[7/7] post-node settle (${POST_UNCORDON_WAIT_SECONDS}s) + health check" +sleep "$POST_UNCORDON_WAIT_SECONDS" + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +if RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh"; then + log "=== $NODE update: OK ===" +else + die "cluster health failed after updating $NODE — STOP the cycle, do NOT continue" +fi