basicstack/stack.basicstack.de

CTO (Paperclip Agent) ac9f36b1e7 feat(velero): monthly restore-drill CronJob for openwebui (DEV-575)

- CronJob at 0 4 1 * * restores the latest openwebui-manifests backup into
  a transient restore-test-<ts> namespace, diffs core resource-kind counts
  against the live openwebui namespace, and deletes the target namespace.
- ServiceAccount + ClusterRole scoped to velero.io + namespace + configmap
  ops. Script enforces the 'restore-test-' name prefix before deleting a
  namespace. Cluster-wide 'view' bound for the diff.
- Failure surfaces via kube_job_status_failed (DEV-574 alert stack) and via
  ConfigMap velero-restore-drill-last-result written on every run.
- Optional Paperclip webhook Secret is sealed but unreachable from Hetzner
  right now (home Paperclip is LAN-only); wired as optional secretKeyRef so
  the drill runs regardless. Connectivity is a follow-up.
- Argo Application velero-restore-drill points at velero/restore-drill/.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

2026-09-12 18:03:37 +00:00

3.2 KiB

Raw Blame History

Velero restore drill

Monthly Kubernetes CronJob that exercises the Velero backup chain by restoring the openwebui namespace into a transient restore-test-<ts> namespace, diffing resource counts against the live namespace, and cleaning up.

Wired by DEV-575 under the DEV-569 Velero cluster-reconstruction backup layer.

What it does

Picks the latest Completed backup for schedule openwebui-manifests (env DRILL_BACKUP_OVERRIDE short-circuits this — used by the failure drill).
Creates a Restore CR with namespace mapping openwebui: restore-test-<ts>.
Waits for the Restore to reach a terminal phase (Completed, PartiallyFailed, Failed, FailedValidation) or times out at 30 min.
Counts core kinds (deployments, statefulsets, services, configmaps, PVCs, …) in source vs target and requires strict equality on the manifest-carrying kinds to pass.
Deletes the restore-test-<ts> namespace and the Restore CR.
Writes the final report to ConfigMap velero/velero-restore-drill-last-result and logs it to stdout.
Optionally POSTs a JSON payload to the Paperclip routine webhook if the velero-restore-drill-webhook Secret is present (currently unreachable from the cluster — see follow-up ticket).

Failure signalling

The Job exits non-zero on any failure. The alerting stack (DEV-574) will fire KubeJobFailed on the velero-restore-drill-* job name and route the alert through existing Alertmanager receivers. Ops can inspect the last result via:

kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.result}'
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.log}'

Manual runs

Kick a drill on demand:

kubectl -n velero create job --from=cronjob/velero-restore-drill drill-adhoc-$(date -u +%Y%m%d-%H%M%S)

Force a failure to test the alert wiring (uses a bogus backup name):

kubectl -n velero create job drill-adhoc-fail-$(date -u +%s) \
  --image=alpine/k8s:1.30.5 \
  --dry-run=client -o yaml \
  | kubectl -n velero apply -f -   # only if you need a scratch, prefer:
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-fail-$(date -u +%Y%m%d-%H%M%S) --dry-run=client -o yaml \
  | sed 's/value: "openwebui-manifests"/value: "openwebui-manifests"\n            - name: DRILL_BACKUP_OVERRIDE\n              value: "no-such-backup"/' \
  | kubectl apply -f -

For a simpler failure drill, apply the CronJob-derived Job then patch env before scheduling — see the parent DEV-575 comment for the exact runbook.

RBAC scope

velero.io/backups,restores: get/list/create/delete in-cluster (namespace scope is enforced by resource type — Backups/Restores live in velero).
namespaces: cluster-wide get/list/create/delete. Cannot be name-scoped in RBAC; the drill script enforces the restore-test- prefix before ever deleting a namespace.
configmaps: get/list/create/update/patch (only used to write the result ConfigMap in velero — pod-level namespace scoping via ServiceAccount).
Cluster-wide view ClusterRole for the counting diff.

3.2 KiB Raw Blame History