73 lines
3.2 KiB
Markdown
73 lines
3.2 KiB
Markdown
|
# Velero restore drill
|
||
|
|
|
||
|
|
Monthly Kubernetes CronJob that exercises the Velero backup chain by restoring
|
||
|
|
the `openwebui` namespace into a transient `restore-test-<ts>` namespace,
|
||
|
|
diffing resource counts against the live namespace, and cleaning up.
|
||
|
|
|
||
|
|
Wired by [DEV-575](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-575)
|
||
|
|
under the [DEV-569](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-569)
|
||
|
|
Velero cluster-reconstruction backup layer.
|
||
|
|
|
||
|
|
## What it does
|
||
|
|
|
||
|
|
1. Picks the latest `Completed` backup for schedule `openwebui-manifests`
|
||
|
|
(env `DRILL_BACKUP_OVERRIDE` short-circuits this — used by the failure
|
||
|
|
drill).
|
||
|
|
2. Creates a `Restore` CR with namespace mapping
|
||
|
|
`openwebui: restore-test-<ts>`.
|
||
|
|
3. Waits for the `Restore` to reach a terminal phase (`Completed`,
|
||
|
|
`PartiallyFailed`, `Failed`, `FailedValidation`) or times out at 30 min.
|
||
|
|
4. Counts core kinds (deployments, statefulsets, services, configmaps, PVCs, …)
|
||
|
|
in source vs target and requires strict equality on the manifest-carrying
|
||
|
|
kinds to pass.
|
||
|
|
5. Deletes the `restore-test-<ts>` namespace and the `Restore` CR.
|
||
|
|
6. Writes the final report to ConfigMap `velero/velero-restore-drill-last-result`
|
||
|
|
and logs it to stdout.
|
||
|
|
7. Optionally POSTs a JSON payload to the Paperclip routine webhook if
|
||
|
|
the `velero-restore-drill-webhook` Secret is present (currently unreachable
|
||
|
|
from the cluster — see follow-up ticket).
|
||
|
|
|
||
|
|
## Failure signalling
|
||
|
|
|
||
|
|
The Job exits non-zero on any failure. The alerting stack (DEV-574) will fire
|
||
|
|
`KubeJobFailed` on the `velero-restore-drill-*` job name and route the alert
|
||
|
|
through existing Alertmanager receivers. Ops can inspect the last result via:
|
||
|
|
|
||
|
|
```
|
||
|
|
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.result}'
|
||
|
|
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.log}'
|
||
|
|
```
|
||
|
|
|
||
|
|
## Manual runs
|
||
|
|
|
||
|
|
Kick a drill on demand:
|
||
|
|
|
||
|
|
```
|
||
|
|
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-adhoc-$(date -u +%Y%m%d-%H%M%S)
|
||
|
|
```
|
||
|
|
|
||
|
|
Force a failure to test the alert wiring (uses a bogus backup name):
|
||
|
|
|
||
|
|
```
|
||
|
|
kubectl -n velero create job drill-adhoc-fail-$(date -u +%s) \
|
||
|
|
--image=alpine/k8s:1.30.5 \
|
||
|
|
--dry-run=client -o yaml \
|
||
|
|
| kubectl -n velero apply -f - # only if you need a scratch, prefer:
|
||
|
|
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-fail-$(date -u +%Y%m%d-%H%M%S) --dry-run=client -o yaml \
|
||
|
|
| sed 's/value: "openwebui-manifests"/value: "openwebui-manifests"\n - name: DRILL_BACKUP_OVERRIDE\n value: "no-such-backup"/' \
|
||
|
|
| kubectl apply -f -
|
||
|
|
```
|
||
|
|
|
||
|
|
For a simpler failure drill, apply the CronJob-derived Job then patch env before scheduling — see the parent DEV-575 comment for the exact runbook.
|
||
|
|
|
||
|
|
## RBAC scope
|
||
|
|
|
||
|
|
- `velero.io/backups,restores`: get/list/create/delete in-cluster (namespace
|
||
|
|
scope is enforced by resource type — Backups/Restores live in `velero`).
|
||
|
|
- `namespaces`: cluster-wide get/list/create/delete. Cannot be name-scoped in
|
||
|
|
RBAC; the drill script enforces the `restore-test-` prefix before ever
|
||
|
|
deleting a namespace.
|
||
|
|
- `configmaps`: get/list/create/update/patch (only used to write the result
|
||
|
|
ConfigMap in `velero` — pod-level namespace scoping via ServiceAccount).
|
||
|
|
- Cluster-wide `view` ClusterRole for the counting diff.
|