diff --git a/apps/monitoring/backup-grafana-restic-cronjob.yaml b/apps/monitoring/backup-grafana-restic-cronjob.yaml index 63d5a5d..0968b75 100644 --- a/apps/monitoring/backup-grafana-restic-cronjob.yaml +++ b/apps/monitoring/backup-grafana-restic-cronjob.yaml @@ -44,7 +44,10 @@ spec: kubernetes.io/hostname: k3s-worker-2 containers: - name: restic - image: restic/restic:0.17.3 + image: harbor.basicstack.de/library/restic:0.17.3 + # Mirrored from docker.io/restic/restic:0.17.3 (DEV-493) — + # deterministic ingress via Harbor. Retag procedure in + # docs/monitoring/restic-restore.md § "Tag-bump procedure". env: - name: AWS_ACCESS_KEY_ID valueFrom: diff --git a/apps/monitoring/backup-k8s-resources-cronjob.yaml b/apps/monitoring/backup-k8s-resources-cronjob.yaml index 0eae91a..751135a 100644 --- a/apps/monitoring/backup-k8s-resources-cronjob.yaml +++ b/apps/monitoring/backup-k8s-resources-cronjob.yaml @@ -90,7 +90,10 @@ spec: mountPath: /dump containers: - name: restic - image: restic/restic:0.17.3 + image: harbor.basicstack.de/library/restic:0.17.3 + # Mirrored from docker.io/restic/restic:0.17.3 (DEV-493) — + # deterministic ingress via Harbor. Retag procedure in + # docs/monitoring/restic-restore.md § "Tag-bump procedure". env: - name: AWS_ACCESS_KEY_ID valueFrom: diff --git a/apps/monitoring/backup-loki-restic-cronjob.yaml b/apps/monitoring/backup-loki-restic-cronjob.yaml index 31e67f9..fc66734 100644 --- a/apps/monitoring/backup-loki-restic-cronjob.yaml +++ b/apps/monitoring/backup-loki-restic-cronjob.yaml @@ -53,7 +53,10 @@ spec: topologyKey: kubernetes.io/hostname containers: - name: restic - image: restic/restic:0.17.3 + image: harbor.basicstack.de/library/restic:0.17.3 + # Mirrored from docker.io/restic/restic:0.17.3 (DEV-493) — + # deterministic ingress via Harbor. Retag procedure in + # docs/monitoring/restic-restore.md § "Tag-bump procedure". env: - name: AWS_ACCESS_KEY_ID valueFrom: diff --git a/docs/monitoring/restic-restore.md b/docs/monitoring/restic-restore.md index 772cd14..e07603c 100644 --- a/docs/monitoring/restic-restore.md +++ b/docs/monitoring/restic-restore.md @@ -13,13 +13,14 @@ event, or (b) run the quarterly restore drill that keeps the ## Repo layout -Single bucket, three repo prefixes: +Single bucket, four repo prefixes: -| repo prefix | source | writer CronJob | tag | -| ---------------------------------------------- | ---------------------------------- | ---------------------- | --------------- | -| `restic/loki` | PVC `loki-storage-encrypted` | `backup-loki-restic` | `loki` | -| `restic/grafana` | PVC `grafana-storage` (worker-2) | `backup-grafana-restic`| `grafana` | -| `restic/k8s-resources` | `kubectl get -o yaml` (stdin) | `backup-k8s-resources` | `k8s-resources` | +| repo prefix | source | writer CronJob | tag | +| ---------------------------------------------- | --------------------------------------------- | ---------------------- | --------------- | +| `restic/loki` | PVC `loki-storage-encrypted` | `backup-loki-restic` | `loki` | +| `restic/grafana` | PVC `grafana-storage` (worker-2) | `backup-grafana-restic`| `grafana` | +| `restic/k8s-resources` | `kubectl get -o yaml` (stdin) | `backup-k8s-resources` | `k8s-resources` | +| `restic/prometheus` | PVC `prometheus-data-encrypted` (excl. WAL) | `prometheus-backup` | `prometheus` | Password + endpoint + bucket + AWS creds live in `SealedSecret` `monitoring-s3-backup` (namespace `monitoring`). Keys: @@ -28,8 +29,7 @@ Password + endpoint + bucket + AWS creds live in `SealedSecret` locally, mirrored to Passbolt entry `restic / monitoring backups`. - `access-key`, `secret-key` — Hetzner S3 credentials. - `endpoint` — e.g. `https://hel1.your-objectstorage.com`. -- `bucket` — Hetzner bucket name (also used by the legacy Prometheus - rclone job). +- `bucket` — Hetzner bucket name (all four restic repos share it). Encryption is done **client-side by restic**. Hetzner Object Storage has no SSE-S3 / SSE-KMS ([FAQ](https://docs.hetzner.com/storage/object-storage/faq/general/)); @@ -70,7 +70,7 @@ spec: restartPolicy: Never containers: - name: restic - image: restic/restic:0.17.3 # matches CronJob image + image: harbor.basicstack.de/library/restic:0.17.3 # matches CronJob image (DEV-493) command: ["sleep", "3600"] env: - { name: AWS_ACCESS_KEY_ID, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: access-key } } } @@ -127,6 +127,41 @@ restic restore latest --target /work/grafana --tag grafana Follows the loki pattern (restored tree under `/work/grafana/source/`). +### Prometheus + +```sh +export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus" +restic snapshots +restic restore latest --target /work/prometheus --tag prometheus +find /work/prometheus/source -maxdepth 1 -type d | head +``` + +Restored tree lands under `/work/prometheus/source/…`. **WAL and +`chunks_head/` are excluded on purpose** — see the "compaction race" +notes at the top of `apps/monitoring/prometheus-backup-cronjob.yaml`. +Expect the last ~15 s of ingested samples to be lost on recovery; the +compacted 2h/24h blocks are complete. + +**Integrity check — `promtool tsdb analyze` (required for the DEV-492 +drill).** The snapshot must load cleanly through Prometheus's own +verifier before we call the drill green. + +```sh +kubectl -n restore-drill run promtool-check --rm -it --restart=Never \ + --image=prom/prometheus:v2.53.1 \ + --overrides='{"spec":{"containers":[{"name":"promtool-check","image":"prom/prometheus:v2.53.1","command":["sh","-c","for b in /work/prometheus/source/*/; do echo \"--- $b\"; promtool tsdb analyze /work/prometheus/source \"$(basename $b)\" || exit 1; done"],"volumeMounts":[{"name":"work","mountPath":"/work"}]}],"volumes":[{"name":"work","emptyDir":{}}]}}' +``` + +Simpler drill path (the restore pod already has restic; `promtool` is +not shipped in `restic/restic`, so run it from a `prom/prometheus` +sidecar or an interactive pod that mounts the same `emptyDir` — see +DEV-492 drill entry for the concrete two-pod recipe used on the +first run). + +Exit 0 on every block = pass. A non-zero exit on any block means the +compaction-race mitigation slipped and the snapshot is corrupt. +Re-run the CronJob and re-drill. + ## Integrity checks Two layers. @@ -198,6 +233,145 @@ restic key remove Rotate annually or immediately on suspected compromise. +## Restic image (Harbor mirror) + +All three restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`, +`backup-k8s-resources`) reference a Harbor-hosted copy of upstream to +keep the backup pipeline off the Docker Hub pull path and immune to +upstream retagging: + +- Manifest reference: `harbor.basicstack.de/library/restic:0.17.3` +- Upstream: `docker.io/restic/restic:0.17.3` +- Ticket: [DEV-493](/DEV/issues/DEV-493) + +The `library` project is public, so no `imagePullSecret` is required +on the CronJob pods. + +### Tag-bump procedure + +Run this when we want to move restic to a new pinned tag (e.g. 0.17.3 +→ 0.17.4). Do the mirror push **before** editing manifests so Argo +cannot roll pods onto an unmirrored tag. + +1. **Pick and verify the upstream tag.** Confirm the tag exists on + Docker Hub and (ideally) read the upstream restic release notes for + breaking changes: + + ```sh + curl -s "https://hub.docker.com/v2/repositories/restic/restic/tags/0.17.4" + ``` + +2. **Mirror the image to Harbor** with a one-shot in-cluster crane + Job. Requires an admin (or `library`-scoped robot) Harbor + credential; the Job auth secret is throwaway. + + ```sh + # from the CTO workstation with kubectl + Harbor admin access + NEW_TAG=0.17.4 + ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \ + -o jsonpath='{.data.harborAdminPassword}' | base64 -d) + AUTH=$(printf 'admin:%s' "$ADMIN_PW" | base64 -w0) + cat >/tmp/dc.json <` under + the `restic` container. Bump `` in all three, commit, push. + +6. **Let Argo sync** — the `monitoring` Application picks up the new + manifests. Watch for successful reconciliation: + + ```sh + kubectl -n argocd get application monitoring \ + -o jsonpath='{.status.sync.status}{" "}{.status.health.status}{"\n"}' + ``` + +7. **Trigger one CronJob run to prove the Harbor pull is green.** + Create a manual Job from any of the CronJobs (loki is fine) and + inspect its logs: + + ```sh + kubectl -n monitoring create job --from=cronjob/backup-loki-restic \ + smoke-restic-$(date +%Y%m%d-%H%M%S) + ``` + + The Pod should pull from `harbor.basicstack.de/library/restic:` + (`kubectl describe pod ...` → Events → `Pulling image …`) and + finish with `restic backup /source` output ending in a snapshot id. + +8. **Update this doc's "restore pod" template** if the new tag is + incompatible with the existing restore pod command — the drill pod + image must match the CronJob image, otherwise the restore-drill + won't round-trip. + +**Rollback:** if the new tag misbehaves, edit the three manifests back +to the last-known-good tag, commit, and let Argo re-sync. The old +tag's manifest remains in Harbor until it's explicitly deleted, so +rollback is a manifest change only. + ## Drill log Every drill appends to this section. Include: