basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/velero/restore-drill
History
CTO 765962c936 fix(velero-drill): hostAliases to reach paperclip.home webhook from cluster (DEV-579) 
Public DNS for paperclip.home.imicros.de returns the LAN IP 10.0.0.2, so the
drill's webhook POST was landing with HTTP 000 from Hetzner. The home reverse
proxy at 46.225.155.185 already serves that vhost publicly, so pinning the
hostname to the home public IP in the CronJob's hostAliases lets the webhook
fire the Paperclip routine without any home-network or SealedSecret changes.

Verified end-to-end: bearer POST via --resolve returns HTTP 202 with
status=issue_created (run issue b07fc035 created under DEV-578).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-12 18:15:51 +00:00
..
00-serviceaccount-rbac.yaml feat(velero): monthly restore-drill CronJob for openwebui (DEV-575) 2026-09-12 18:03:37 +00:00
10-drill-script-configmap.yaml feat(velero): monthly restore-drill CronJob for openwebui (DEV-575) 2026-09-12 18:03:37 +00:00
20-restore-drill-cronjob.yaml fix(velero-drill): hostAliases to reach paperclip.home webhook from cluster (DEV-579) 2026-09-12 18:15:51 +00:00
30-webhook-sealed.yaml feat(velero): monthly restore-drill CronJob for openwebui (DEV-575) 2026-09-12 18:03:37 +00:00
README.md fix(velero-drill): hostAliases to reach paperclip.home webhook from cluster (DEV-579) 2026-09-12 18:15:51 +00:00

README.md

Velero restore drill

Monthly Kubernetes CronJob that exercises the Velero backup chain by restoring the openwebui namespace into a transient restore-test-<ts> namespace, diffing resource counts against the live namespace, and cleaning up.

Wired by DEV-575 under the DEV-569 Velero cluster-reconstruction backup layer.

What it does

  1. Picks the latest Completed backup for schedule openwebui-manifests (env DRILL_BACKUP_OVERRIDE short-circuits this — used by the failure drill).
  2. Creates a Restore CR with namespace mapping openwebui: restore-test-<ts>.
  3. Waits for the Restore to reach a terminal phase (Completed, PartiallyFailed, Failed, FailedValidation) or times out at 30 min.
  4. Counts core kinds (deployments, statefulsets, services, configmaps, PVCs, …) in source vs target and requires strict equality on the manifest-carrying kinds to pass.
  5. Deletes the restore-test-<ts> namespace and the Restore CR.
  6. Writes the final report to ConfigMap velero/velero-restore-drill-last-result and logs it to stdout.
  7. Optionally POSTs a JSON payload to the Paperclip routine webhook if the velero-restore-drill-webhook Secret is present. The CronJob uses a hostAliases entry to route paperclip.home.imicros.de at the home public IP so the webhook reaches the routine from Hetzner (DEV-579).

Failure signalling

The Job exits non-zero on any failure. The alerting stack (DEV-574) will fire KubeJobFailed on the velero-restore-drill-* job name and route the alert through existing Alertmanager receivers. Ops can inspect the last result via:

kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.result}'
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.log}'

Manual runs

Kick a drill on demand:

kubectl -n velero create job --from=cronjob/velero-restore-drill drill-adhoc-$(date -u +%Y%m%d-%H%M%S)

Force a failure to test the alert wiring (uses a bogus backup name):

kubectl -n velero create job drill-adhoc-fail-$(date -u +%s) \
  --image=alpine/k8s:1.30.5 \
  --dry-run=client -o yaml \
  | kubectl -n velero apply -f -   # only if you need a scratch, prefer:
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-fail-$(date -u +%Y%m%d-%H%M%S) --dry-run=client -o yaml \
  | sed 's/value: "openwebui-manifests"/value: "openwebui-manifests"\n            - name: DRILL_BACKUP_OVERRIDE\n              value: "no-such-backup"/' \
  | kubectl apply -f -

For a simpler failure drill, apply the CronJob-derived Job then patch env before scheduling — see the parent DEV-575 comment for the exact runbook.

RBAC scope

  • velero.io/backups,restores: get/list/create/delete in-cluster (namespace scope is enforced by resource type — Backups/Restores live in velero).
  • namespaces: cluster-wide get/list/create/delete. Cannot be name-scoped in RBAC; the drill script enforces the restore-test- prefix before ever deleting a namespace.
  • configmaps: get/list/create/update/patch (only used to write the result ConfigMap in velero — pod-level namespace scoping via ServiceAccount).
  • Cluster-wide view ClusterRole for the counting diff.