0164c8cb4e
The 2026-10-01 monthly drill passed but the post-run webhook got HTTP 401 from the home Paperclip. Root cause: the home server's global actorMiddleware (/app/server/src/middleware/auth.ts) rejects every Authorization: Bearer <...> that is not a board API key or an agent JWT before per-route handlers run, so the trigger's own `bearer` signingMode handler is unreachable with a 48-hex random secret. Only publicMcpGatewayProtocolPath is exempted. Fix: - Trigger 6983798e-018c-459d-95ea-4b5959441e2d switched to signingMode: github_hmac via PATCH (secret unchanged). - post_webhook now computes HMAC-SHA256(body, secret) with openssl (already in alpine/k8s) and sends X-Hub-Signature-256 instead of Authorization: Bearer. HMAC headers bypass the Bearer branch of the global middleware entirely. Verified end-to-end: synthetic HMAC-signed body posted to .../fire returned 202 issue_created and spawned DEV-591 under parent DEV-578. SealedSecret is unchanged (same secret value reused as the HMAC key). Co-Authored-By: Paperclip <noreply@paperclip.ing>
81 lines
3.9 KiB
Markdown
81 lines
3.9 KiB
Markdown
# Velero restore drill
|
|
|
|
Monthly Kubernetes CronJob that exercises the Velero backup chain by restoring
|
|
the `openwebui` namespace into a transient `restore-test-<ts>` namespace,
|
|
diffing resource counts against the live namespace, and cleaning up.
|
|
|
|
Wired by [DEV-575](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-575)
|
|
under the [DEV-569](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-569)
|
|
Velero cluster-reconstruction backup layer.
|
|
|
|
## What it does
|
|
|
|
1. Picks the latest `Completed` backup for schedule `openwebui-manifests`
|
|
(env `DRILL_BACKUP_OVERRIDE` short-circuits this — used by the failure
|
|
drill).
|
|
2. Creates a `Restore` CR with namespace mapping
|
|
`openwebui: restore-test-<ts>`.
|
|
3. Waits for the `Restore` to reach a terminal phase (`Completed`,
|
|
`PartiallyFailed`, `Failed`, `FailedValidation`) or times out at 30 min.
|
|
4. Counts core kinds (deployments, statefulsets, services, configmaps, PVCs, …)
|
|
in source vs target and requires strict equality on the manifest-carrying
|
|
kinds to pass.
|
|
5. Deletes the `restore-test-<ts>` namespace and the `Restore` CR.
|
|
6. Writes the final report to ConfigMap `velero/velero-restore-drill-last-result`
|
|
and logs it to stdout.
|
|
7. Optionally POSTs a JSON payload to the Paperclip routine webhook if
|
|
the `velero-restore-drill-webhook` Secret is present. The CronJob uses a
|
|
`hostAliases` entry to route `paperclip.home.imicros.de` at the home
|
|
public IP so the webhook reaches the routine from Hetzner (DEV-579).
|
|
Authentication is `github_hmac` — the script computes an HMAC-SHA256
|
|
of the raw body with the webhook secret as key and sends it as
|
|
`X-Hub-Signature-256: sha256=<hex>`. The matching trigger `signingMode`
|
|
is `github_hmac` on routine `c9d941fa-fb97-4e6a-a305-8e46f296e0cd`.
|
|
Previously it was `bearer`, but the home Paperclip's global actor
|
|
middleware rejects any `Authorization: Bearer …` whose value is not a
|
|
board API key or an agent JWT before the per-trigger handler runs,
|
|
making the bearer path unreachable (DEV-590).
|
|
|
|
## Failure signalling
|
|
|
|
The Job exits non-zero on any failure. The alerting stack (DEV-574) will fire
|
|
`KubeJobFailed` on the `velero-restore-drill-*` job name and route the alert
|
|
through existing Alertmanager receivers. Ops can inspect the last result via:
|
|
|
|
```
|
|
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.result}'
|
|
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.log}'
|
|
```
|
|
|
|
## Manual runs
|
|
|
|
Kick a drill on demand:
|
|
|
|
```
|
|
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-adhoc-$(date -u +%Y%m%d-%H%M%S)
|
|
```
|
|
|
|
Force a failure to test the alert wiring (uses a bogus backup name):
|
|
|
|
```
|
|
kubectl -n velero create job drill-adhoc-fail-$(date -u +%s) \
|
|
--image=alpine/k8s:1.30.5 \
|
|
--dry-run=client -o yaml \
|
|
| kubectl -n velero apply -f - # only if you need a scratch, prefer:
|
|
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-fail-$(date -u +%Y%m%d-%H%M%S) --dry-run=client -o yaml \
|
|
| sed 's/value: "openwebui-manifests"/value: "openwebui-manifests"\n - name: DRILL_BACKUP_OVERRIDE\n value: "no-such-backup"/' \
|
|
| kubectl apply -f -
|
|
```
|
|
|
|
For a simpler failure drill, apply the CronJob-derived Job then patch env before scheduling — see the parent DEV-575 comment for the exact runbook.
|
|
|
|
## RBAC scope
|
|
|
|
- `velero.io/backups,restores`: get/list/create/delete in-cluster (namespace
|
|
scope is enforced by resource type — Backups/Restores live in `velero`).
|
|
- `namespaces`: cluster-wide get/list/create/delete. Cannot be name-scoped in
|
|
RBAC; the drill script enforces the `restore-test-` prefix before ever
|
|
deleting a namespace.
|
|
- `configmaps`: get/list/create/update/patch (only used to write the result
|
|
ConfigMap in `velero` — pod-level namespace scoping via ServiceAccount).
|
|
- Cluster-wide `view` ClusterRole for the counting diff.
|