3b4146de65
- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update procedure (drain -> apt -> reboot -> verify -> uncordon -> health -> next). Explicit MUST NOT list around k3s config, PVs, and manifests. - infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip routine (Sun 03:00 Europe/Berlin) that fires this procedure. - infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh, os-update.sh, README. Enforces the same guardrails in code: workers-first-then-CP, one node at a time, no --force drains, halts on reboot/kubelet/health failure, never touches k3s config. Co-Authored-By: Paperclip <noreply@paperclip.ing>
47 lines
2 KiB
Markdown
47 lines
2 KiB
Markdown
# Weekly OS Update — Paperclip Routine
|
|
|
|
**Owner:** CTO agent (until a dedicated ClusterOps agent is spun up)
|
|
**Schedule:** every **Sunday 03:00 Europe/Berlin** (low-traffic window, before Monday operations)
|
|
**Companion procedure:** [`OS_UPDATE_PROCEDURE.md`](OS_UPDATE_PROCEDURE.md)
|
|
|
|
## What the routine does
|
|
|
|
Each fire creates a Paperclip task whose description points to `OS_UPDATE_PROCEDURE.md`. The assigned agent:
|
|
|
|
1. Checks out the task.
|
|
2. Reads the procedure doc.
|
|
3. Runs `infrastructure/scripts/os-update/os-update.sh` end to end.
|
|
4. Attaches per-run logs (`/tmp/os-update-<stamp>/`) to the task.
|
|
5. Closes the task `done` on success, or `blocked` with a named unblock action on skip/error.
|
|
|
|
## Routine configuration (Paperclip)
|
|
|
|
- `title`: "Weekly rolling OS update — k3s cluster nodes"
|
|
- `assigneeAgentId`: CTO (agent id `4b5f09a2-22d8-4e3d-9ac4-46d008ad1385`)
|
|
- `projectId`: `d24057b8-02ca-41cd-b7a0-6151298c6e2c` (BasicStack Phase-2)
|
|
- `goalId`: `b4dfe4a8-5c37-4f62-aad9-fa340dcd34a4` (self-hosted Kubernetes at Hetzner)
|
|
- `priority`: `medium`
|
|
- `concurrencyPolicy`: `coalesce_if_active` (a previous run still open? merge into it, don't stack)
|
|
- `catchUpPolicy`: `skip_missed` (no back-fires if Paperclip was down)
|
|
- Trigger: `schedule` — cron `0 3 * * 0`, timezone `Europe/Berlin`
|
|
|
|
The task description created by each fire is a link to the procedure doc in Forgejo, so the doc is the single source of truth even if the routine metadata drifts.
|
|
|
|
## Reassigning to a dedicated agent
|
|
|
|
If we later hire a ClusterOps agent, only two things change:
|
|
|
|
1. `PATCH /api/routines/{routineId}` — update `assigneeAgentId`.
|
|
2. Nothing in `OS_UPDATE_PROCEDURE.md` or the scripts changes; the procedure is agent-neutral.
|
|
|
|
## Pausing / disabling
|
|
|
|
To pause without losing config:
|
|
|
|
```bash
|
|
curl -X PATCH -H "Authorization: Bearer $PAPERCLIP_API_KEY" -H "Content-Type: application/json" \
|
|
-d '{"status":"paused"}' \
|
|
"$PAPERCLIP_API_URL/routines/{routineId}"
|
|
```
|
|
|
|
Resume with `{"status":"active"}`.
|