basicstack/stack.basicstack.de

CTO 3b4146de65 Add weekly rolling OS-update procedure for k3s nodes (DEV-462)

- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update
  procedure (drain -> apt -> reboot -> verify -> uncordon -> health ->
  next). Explicit MUST NOT list around k3s config, PVs, and manifests.
- infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip
  routine (Sun 03:00 Europe/Berlin) that fires this procedure.
- infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh,
  os-update.sh, README. Enforces the same guardrails in code:
  workers-first-then-CP, one node at a time, no --force drains, halts on
  reboot/kubelet/health failure, never touches k3s config.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

2026-08-09 15:55:39 +00:00

2 KiB

Raw Blame History

Weekly OS Update — Paperclip Routine

Owner: CTO agent (until a dedicated ClusterOps agent is spun up) Schedule: every Sunday 03:00 Europe/Berlin (low-traffic window, before Monday operations) Companion procedure: OS_UPDATE_PROCEDURE.md

What the routine does

Each fire creates a Paperclip task whose description points to OS_UPDATE_PROCEDURE.md. The assigned agent:

Checks out the task.
Reads the procedure doc.
Runs infrastructure/scripts/os-update/os-update.sh end to end.
Attaches per-run logs (/tmp/os-update-<stamp>/) to the task.
Closes the task done on success, or blocked with a named unblock action on skip/error.

Routine configuration (Paperclip)

title: "Weekly rolling OS update — k3s cluster nodes"
assigneeAgentId: CTO (agent id 4b5f09a2-22d8-4e3d-9ac4-46d008ad1385)
projectId: d24057b8-02ca-41cd-b7a0-6151298c6e2c (BasicStack Phase-2)
goalId: b4dfe4a8-5c37-4f62-aad9-fa340dcd34a4 (self-hosted Kubernetes at Hetzner)
priority: medium
concurrencyPolicy: coalesce_if_active (a previous run still open? merge into it, don't stack)
catchUpPolicy: skip_missed (no back-fires if Paperclip was down)
Trigger: schedule — cron 0 3 * * 0, timezone Europe/Berlin

The task description created by each fire is a link to the procedure doc in Forgejo, so the doc is the single source of truth even if the routine metadata drifts.

Reassigning to a dedicated agent

If we later hire a ClusterOps agent, only two things change:

PATCH /api/routines/{routineId} — update assigneeAgentId.
Nothing in OS_UPDATE_PROCEDURE.md or the scripts changes; the procedure is agent-neutral.

Pausing / disabling

To pause without losing config:

curl -X PATCH -H "Authorization: Bearer $PAPERCLIP_API_KEY" -H "Content-Type: application/json" \
  -d '{"status":"paused"}' \
  "$PAPERCLIP_API_URL/routines/{routineId}"

Resume with {"status":"active"}.

2 KiB Raw Blame History