# Velero restore drill

Monthly Kubernetes CronJob that exercises the Velero backup chain by restoring
the `openwebui` namespace into a transient `restore-test-<ts>` namespace,
diffing resource counts against the live namespace, and cleaning up.

Wired by [DEV-575](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-575)
under the [DEV-569](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-569)
Velero cluster-reconstruction backup layer.

## What it does

1. Picks the latest `Completed` backup for schedule `openwebui-manifests`
   (env `DRILL_BACKUP_OVERRIDE` short-circuits this — used by the failure
   drill).
2. Creates a `Restore` CR with namespace mapping
   `openwebui: restore-test-<ts>`.
3. Waits for the `Restore` to reach a terminal phase (`Completed`,
   `PartiallyFailed`, `Failed`, `FailedValidation`) or times out at 30 min.
4. Counts core kinds (deployments, statefulsets, services, configmaps, PVCs, …)
   in source vs target and requires strict equality on the manifest-carrying
   kinds to pass.
5. Deletes the `restore-test-<ts>` namespace and the `Restore` CR.
6. Writes the final report to ConfigMap `velero/velero-restore-drill-last-result`
   and logs it to stdout.
7. Optionally POSTs a JSON payload to the Paperclip routine webhook if
   the `velero-restore-drill-webhook` Secret is present. The CronJob uses a
   `hostAliases` entry to route `paperclip.home.imicros.de` at the home
   public IP so the webhook reaches the routine from Hetzner (DEV-579).
   Authentication is `github_hmac` — the script computes an HMAC-SHA256
   of the raw body with the webhook secret as key and sends it as
   `X-Hub-Signature-256: sha256=<hex>`. The matching trigger `signingMode`
   is `github_hmac` on routine `c9d941fa-fb97-4e6a-a305-8e46f296e0cd`.
   Previously it was `bearer`, but the home Paperclip's global actor
   middleware rejects any `Authorization: Bearer …` whose value is not a
   board API key or an agent JWT before the per-trigger handler runs,
   making the bearer path unreachable (DEV-590).

## Failure signalling

The Job exits non-zero on any failure. The alerting stack (DEV-574) will fire
`KubeJobFailed` on the `velero-restore-drill-*` job name and route the alert
through existing Alertmanager receivers. Ops can inspect the last result via:

```
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.result}'
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.log}'
```

## Manual runs

Kick a drill on demand:

```
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-adhoc-$(date -u +%Y%m%d-%H%M%S)
```

Force a failure to test the alert wiring (uses a bogus backup name):

```
kubectl -n velero create job drill-adhoc-fail-$(date -u +%s) \
  --image=alpine/k8s:1.30.5 \
  --dry-run=client -o yaml \
  | kubectl -n velero apply -f -   # only if you need a scratch, prefer:
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-fail-$(date -u +%Y%m%d-%H%M%S) --dry-run=client -o yaml \
  | sed 's/value: "openwebui-manifests"/value: "openwebui-manifests"\n            - name: DRILL_BACKUP_OVERRIDE\n              value: "no-such-backup"/' \
  | kubectl apply -f -
```

For a simpler failure drill, apply the CronJob-derived Job then patch env before scheduling — see the parent DEV-575 comment for the exact runbook.

## RBAC scope

- `velero.io/backups,restores`: get/list/create/delete in-cluster (namespace
  scope is enforced by resource type — Backups/Restores live in `velero`).
- `namespaces`: cluster-wide get/list/create/delete. Cannot be name-scoped in
  RBAC; the drill script enforces the `restore-test-` prefix before ever
  deleting a namespace.
- `configmaps`: get/list/create/update/patch (only used to write the result
  ConfigMap in `velero` — pod-level namespace scoping via ServiceAccount).
- Cluster-wide `view` ClusterRole for the counting diff.