765962c936
Public DNS for paperclip.home.imicros.de returns the LAN IP 10.0.0.2, so the drill's webhook POST was landing with HTTP 000 from Hetzner. The home reverse proxy at 46.225.155.185 already serves that vhost publicly, so pinning the hostname to the home public IP in the CronJob's hostAliases lets the webhook fire the Paperclip routine without any home-network or SealedSecret changes. Verified end-to-end: bearer POST via --resolve returns HTTP 202 with status=issue_created (run issue b07fc035 created under DEV-578). Co-Authored-By: Paperclip <noreply@paperclip.ing>
73 lines
3.3 KiB
Markdown
73 lines
3.3 KiB
Markdown
# Velero restore drill
|
|
|
|
Monthly Kubernetes CronJob that exercises the Velero backup chain by restoring
|
|
the `openwebui` namespace into a transient `restore-test-<ts>` namespace,
|
|
diffing resource counts against the live namespace, and cleaning up.
|
|
|
|
Wired by [DEV-575](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-575)
|
|
under the [DEV-569](https://forgejo.basicstack.de/basicstack/stack.basicstack.de/wiki/DEV-569)
|
|
Velero cluster-reconstruction backup layer.
|
|
|
|
## What it does
|
|
|
|
1. Picks the latest `Completed` backup for schedule `openwebui-manifests`
|
|
(env `DRILL_BACKUP_OVERRIDE` short-circuits this — used by the failure
|
|
drill).
|
|
2. Creates a `Restore` CR with namespace mapping
|
|
`openwebui: restore-test-<ts>`.
|
|
3. Waits for the `Restore` to reach a terminal phase (`Completed`,
|
|
`PartiallyFailed`, `Failed`, `FailedValidation`) or times out at 30 min.
|
|
4. Counts core kinds (deployments, statefulsets, services, configmaps, PVCs, …)
|
|
in source vs target and requires strict equality on the manifest-carrying
|
|
kinds to pass.
|
|
5. Deletes the `restore-test-<ts>` namespace and the `Restore` CR.
|
|
6. Writes the final report to ConfigMap `velero/velero-restore-drill-last-result`
|
|
and logs it to stdout.
|
|
7. Optionally POSTs a JSON payload to the Paperclip routine webhook if
|
|
the `velero-restore-drill-webhook` Secret is present. The CronJob uses a
|
|
`hostAliases` entry to route `paperclip.home.imicros.de` at the home
|
|
public IP so the webhook reaches the routine from Hetzner (DEV-579).
|
|
|
|
## Failure signalling
|
|
|
|
The Job exits non-zero on any failure. The alerting stack (DEV-574) will fire
|
|
`KubeJobFailed` on the `velero-restore-drill-*` job name and route the alert
|
|
through existing Alertmanager receivers. Ops can inspect the last result via:
|
|
|
|
```
|
|
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.result}'
|
|
kubectl -n velero get cm velero-restore-drill-last-result -o jsonpath='{.data.log}'
|
|
```
|
|
|
|
## Manual runs
|
|
|
|
Kick a drill on demand:
|
|
|
|
```
|
|
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-adhoc-$(date -u +%Y%m%d-%H%M%S)
|
|
```
|
|
|
|
Force a failure to test the alert wiring (uses a bogus backup name):
|
|
|
|
```
|
|
kubectl -n velero create job drill-adhoc-fail-$(date -u +%s) \
|
|
--image=alpine/k8s:1.30.5 \
|
|
--dry-run=client -o yaml \
|
|
| kubectl -n velero apply -f - # only if you need a scratch, prefer:
|
|
kubectl -n velero create job --from=cronjob/velero-restore-drill drill-fail-$(date -u +%Y%m%d-%H%M%S) --dry-run=client -o yaml \
|
|
| sed 's/value: "openwebui-manifests"/value: "openwebui-manifests"\n - name: DRILL_BACKUP_OVERRIDE\n value: "no-such-backup"/' \
|
|
| kubectl apply -f -
|
|
```
|
|
|
|
For a simpler failure drill, apply the CronJob-derived Job then patch env before scheduling — see the parent DEV-575 comment for the exact runbook.
|
|
|
|
## RBAC scope
|
|
|
|
- `velero.io/backups,restores`: get/list/create/delete in-cluster (namespace
|
|
scope is enforced by resource type — Backups/Restores live in `velero`).
|
|
- `namespaces`: cluster-wide get/list/create/delete. Cannot be name-scoped in
|
|
RBAC; the drill script enforces the `restore-test-` prefix before ever
|
|
deleting a namespace.
|
|
- `configmaps`: get/list/create/update/patch (only used to write the result
|
|
ConfigMap in `velero` — pod-level namespace scoping via ServiceAccount).
|
|
- Cluster-wide `view` ClusterRole for the counting diff.
|