ac9f36b1e7
- CronJob at 0 4 1 * * restores the latest openwebui-manifests backup into a transient restore-test-<ts> namespace, diffs core resource-kind counts against the live openwebui namespace, and deletes the target namespace. - ServiceAccount + ClusterRole scoped to velero.io + namespace + configmap ops. Script enforces the 'restore-test-' name prefix before deleting a namespace. Cluster-wide 'view' bound for the diff. - Failure surfaces via kube_job_status_failed (DEV-574 alert stack) and via ConfigMap velero-restore-drill-last-result written on every run. - Optional Paperclip webhook Secret is sealed but unreachable from Hetzner right now (home Paperclip is LAN-only); wired as optional secretKeyRef so the drill runs regardless. Connectivity is a follow-up. - Argo Application velero-restore-drill points at velero/restore-drill/. Co-Authored-By: Paperclip <noreply@paperclip.ing>
72 lines
3.2 KiB
Markdown
72 lines
3.2 KiB
Markdown
# Velero restore drill
|
|
|
|
Monthly Kubernetes CronJob that exercises the Velero backup chain by restoring
|
|
the `openwebui` namespace into a transient `restore-test-<ts>` namespace,
|
|
diffing resource counts against the live namespace, and cleaning up.
|
|
|
|
Wired by [DEV-575](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-575)
|
|
under the [DEV-569](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-569)
|
|
Velero cluster-reconstruction backup layer.
|
|
|
|
## What it does
|
|
|
|
1. Picks the latest `Completed` backup for schedule `openwebui-manifests`
|
|
(env `DRILL_BACKUP_OVERRIDE` short-circuits this — used by the failure
|
|
drill).
|
|
2. Creates a `Restore` CR with namespace mapping
|
|
`openwebui: restore-test-<ts>`.
|
|
3. Waits for the `Restore` to reach a terminal phase (`Completed`,
|
|
`PartiallyFailed`, `Failed`, `FailedValidation`) or times out at 30 min.
|
|
4. Counts core kinds (deployments, statefulsets, services, configmaps, PVCs, …)
|
|
in source vs target and requires strict equality on the manifest-carrying
|
|
kinds to pass.
|
|
5. Deletes the `restore-test-<ts>` namespace and the `Restore` CR.
|
|
6. Writes the final report to ConfigMap `velero/velero-restore-drill-last-result`
|
|
and logs it to stdout.
|
|
7. Optionally POSTs a JSON payload to the Paperclip routine webhook if
|
|
the `velero-restore-drill-webhook` Secret is present (currently unreachable
|
|
from the cluster — see follow-up ticket).
|
|
|
|
## Failure signalling
|
|
|
|
The Job exits non-zero on any failure. The alerting stack (DEV-574) will fire
|
|
`KubeJobFailed` on the `velero-restore-drill-*` job name and route the alert
|
|
through existing Alertmanager receivers. Ops can inspect the last result via:
|
|
|
|
```
|
|
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.result}'
|
|
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.log}'
|
|
```
|
|
|
|
## Manual runs
|
|
|
|
Kick a drill on demand:
|
|
|
|
```
|
|
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-adhoc-$(date -u +%Y%m%d-%H%M%S)
|
|
```
|
|
|
|
Force a failure to test the alert wiring (uses a bogus backup name):
|
|
|
|
```
|
|
kubectl -n velero create job drill-adhoc-fail-$(date -u +%s) \
|
|
--image=alpine/k8s:1.30.5 \
|
|
--dry-run=client -o yaml \
|
|
| kubectl -n velero apply -f - # only if you need a scratch, prefer:
|
|
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-fail-$(date -u +%Y%m%d-%H%M%S) --dry-run=client -o yaml \
|
|
| sed 's/value: "openwebui-manifests"/value: "openwebui-manifests"\n - name: DRILL_BACKUP_OVERRIDE\n value: "no-such-backup"/' \
|
|
| kubectl apply -f -
|
|
```
|
|
|
|
For a simpler failure drill, apply the CronJob-derived Job then patch env before scheduling — see the parent DEV-575 comment for the exact runbook.
|
|
|
|
## RBAC scope
|
|
|
|
- `velero.io/backups,restores`: get/list/create/delete in-cluster (namespace
|
|
scope is enforced by resource type — Backups/Restores live in `velero`).
|
|
- `namespaces`: cluster-wide get/list/create/delete. Cannot be name-scoped in
|
|
RBAC; the drill script enforces the `restore-test-` prefix before ever
|
|
deleting a namespace.
|
|
- `configmaps`: get/list/create/update/patch (only used to write the result
|
|
ConfigMap in `velero` — pod-level namespace scoping via ServiceAccount).
|
|
- Cluster-wide `view` ClusterRole for the counting diff.
|