basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/velero/restore-drill
History
CTO 0164c8cb4e fix(velero-drill): switch webhook from bearer to github_hmac (DEV-590) 
The 2026-10-01 monthly drill passed but the post-run webhook got
HTTP 401 from the home Paperclip. Root cause: the home server's
global actorMiddleware (/app/server/src/middleware/auth.ts) rejects
every Authorization: Bearer <...> that is not a board API key or an
agent JWT before per-route handlers run, so the trigger's own
`bearer` signingMode handler is unreachable with a 48-hex random
secret. Only publicMcpGatewayProtocolPath is exempted.

Fix:
- Trigger 6983798e-018c-459d-95ea-4b5959441e2d switched to
  signingMode: github_hmac via PATCH (secret unchanged).
- post_webhook now computes HMAC-SHA256(body, secret) with openssl
  (already in alpine/k8s) and sends X-Hub-Signature-256 instead of
  Authorization: Bearer. HMAC headers bypass the Bearer branch of
  the global middleware entirely.

Verified end-to-end: synthetic HMAC-signed body posted to
.../fire returned 202 issue_created and spawned DEV-591 under
parent DEV-578. SealedSecret is unchanged (same secret value
reused as the HMAC key).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-01 04:45:51 +00:00
..
00-serviceaccount-rbac.yaml feat(velero): monthly restore-drill CronJob for openwebui (DEV-575) 2026-09-12 18:03:37 +00:00
10-drill-script-configmap.yaml fix(velero-drill): switch webhook from bearer to github_hmac (DEV-590) 2026-10-01 04:45:51 +00:00
20-restore-drill-cronjob.yaml fix(velero-drill): hostAliases to reach paperclip.home webhook from cluster (DEV-579) 2026-09-12 18:15:51 +00:00
30-webhook-sealed.yaml feat(velero): monthly restore-drill CronJob for openwebui (DEV-575) 2026-09-12 18:03:37 +00:00
README.md fix(velero-drill): switch webhook from bearer to github_hmac (DEV-590) 2026-10-01 04:45:51 +00:00

README.md

Velero restore drill

Monthly Kubernetes CronJob that exercises the Velero backup chain by restoring the openwebui namespace into a transient restore-test-<ts> namespace, diffing resource counts against the live namespace, and cleaning up.

Wired by DEV-575 under the DEV-569 Velero cluster-reconstruction backup layer.

What it does

  1. Picks the latest Completed backup for schedule openwebui-manifests (env DRILL_BACKUP_OVERRIDE short-circuits this — used by the failure drill).
  2. Creates a Restore CR with namespace mapping openwebui: restore-test-<ts>.
  3. Waits for the Restore to reach a terminal phase (Completed, PartiallyFailed, Failed, FailedValidation) or times out at 30 min.
  4. Counts core kinds (deployments, statefulsets, services, configmaps, PVCs, …) in source vs target and requires strict equality on the manifest-carrying kinds to pass.
  5. Deletes the restore-test-<ts> namespace and the Restore CR.
  6. Writes the final report to ConfigMap velero/velero-restore-drill-last-result and logs it to stdout.
  7. Optionally POSTs a JSON payload to the Paperclip routine webhook if the velero-restore-drill-webhook Secret is present. The CronJob uses a hostAliases entry to route paperclip.home.imicros.de at the home public IP so the webhook reaches the routine from Hetzner (DEV-579). Authentication is github_hmac — the script computes an HMAC-SHA256 of the raw body with the webhook secret as key and sends it as X-Hub-Signature-256: sha256=<hex>. The matching trigger signingMode is github_hmac on routine c9d941fa-fb97-4e6a-a305-8e46f296e0cd. Previously it was bearer, but the home Paperclip's global actor middleware rejects any Authorization: Bearer … whose value is not a board API key or an agent JWT before the per-trigger handler runs, making the bearer path unreachable (DEV-590).

Failure signalling

The Job exits non-zero on any failure. The alerting stack (DEV-574) will fire KubeJobFailed on the velero-restore-drill-* job name and route the alert through existing Alertmanager receivers. Ops can inspect the last result via:

kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.result}'
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.log}'

Manual runs

Kick a drill on demand:

kubectl -n velero create job --from=cronjob/velero-restore-drill drill-adhoc-$(date -u +%Y%m%d-%H%M%S)

Force a failure to test the alert wiring (uses a bogus backup name):

kubectl -n velero create job drill-adhoc-fail-$(date -u +%s) \
  --image=alpine/k8s:1.30.5 \
  --dry-run=client -o yaml \
  | kubectl -n velero apply -f -   # only if you need a scratch, prefer:
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-fail-$(date -u +%Y%m%d-%H%M%S) --dry-run=client -o yaml \
  | sed 's/value: "openwebui-manifests"/value: "openwebui-manifests"\n            - name: DRILL_BACKUP_OVERRIDE\n              value: "no-such-backup"/' \
  | kubectl apply -f -

For a simpler failure drill, apply the CronJob-derived Job then patch env before scheduling — see the parent DEV-575 comment for the exact runbook.

RBAC scope

  • velero.io/backups,restores: get/list/create/delete in-cluster (namespace scope is enforced by resource type — Backups/Restores live in velero).
  • namespaces: cluster-wide get/list/create/delete. Cannot be name-scoped in RBAC; the drill script enforces the restore-test- prefix before ever deleting a namespace.
  • configmaps: get/list/create/update/patch (only used to write the result ConfigMap in velero — pod-level namespace scoping via ServiceAccount).
  • Cluster-wide view ClusterRole for the counting diff.