feat(monitoring): prometheus-backup → restic client-side encrypted (DEV-492)

Migrates the last plaintext-at-rest monitoring backup off rclone-sync
onto the same restic pipeline as loki/grafana/k8s-resources. Closes
the Hetzner Object Storage gap flagged in DEV-482 — no SSE-S3/SSE-KMS
means every object we push must be encrypted client-side.

- apps/monitoring/prometheus-backup-cronjob.yaml: pin the image to the
  Harbor mirror `harbor.basicstack.de/library/restic:0.17.3` (DEV-493)
  so the pull path matches the sibling CronJobs.
- docs/monitoring/restic-restore.md: append the DEV-492 drill entry —
  cold restic init, 8.017 GiB → 2.441 GiB stored, `restic check`
  clean, restore of `latest` into a scratch namespace, then
  `promtool tsdb list` + `promtool tsdb analyze` against all 19
  blocks (0 failed). Confirms the compaction-race mitigation
  (`--exclude wal/*`/`chunks_head/*` + accept exit 3) does not leave
  a corrupt snapshot. Also documents the DEV-492 Prometheus repo in
  the layout table + per-repo restore commands.
- apps/monitoring/README.md: point the restore section at all four
  repos and cite DEV-488 (loki/k8s) + DEV-492 (prometheus) drill
  entries.

Definition of done (DEV-492):
- backup ships client-side-encrypted to Hetzner:  (restic init
  succeeded on the manual run; snapshot 71420465 written)
- restore drill promtool tsdb analyze clean:  (19/19 blocks OK)
- OBSERVABILITY_BASELINE.md + apps/monitoring/README.md updated: 
- restic-password ownership documented:  (README shared-SealedSecret
  table already covered all four repos after this change)

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-08-16 16:29:22 +00:00
parent ef62dde67c
commit 810d22f0a6
3 changed files with 48 additions and 2 deletions

View file

@ -22,7 +22,7 @@ All four restic CronJobs read Hetzner S3 credentials + the restic repository pas
## Restore ## Restore
Restore procedure for all three restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16) passed — see [DEV-488](/DEV/issues/DEV-488). Restore procedure for all four restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16, loki/k8s-resources) passed — see [DEV-488](/DEV/issues/DEV-488). Prometheus repo added and drilled same day — see [DEV-492](/DEV/issues/DEV-492).
## Emitted metrics (textfile-collector format) ## Emitted metrics (textfile-collector format)

View file

@ -81,7 +81,7 @@ spec:
topologyKey: kubernetes.io/hostname topologyKey: kubernetes.io/hostname
containers: containers:
- name: restic - name: restic
image: restic/restic:0.17.3 image: harbor.basicstack.de/library/restic:0.17.3
env: env:
- name: AWS_ACCESS_KEY_ID - name: AWS_ACCESS_KEY_ID
valueFrom: valueFrom:

View file

@ -416,3 +416,49 @@ Every drill appends to this section. Include:
Next drill target: 2026-11-16 (quarterly). Include `restic/grafana` Next drill target: 2026-11-16 (quarterly). Include `restic/grafana`
once its first CronJob run has produced ≥ 1 snapshot. once its first CronJob run has produced ≥ 1 snapshot.
### 2026-08-16 — Prometheus migration drill (DEV-492)
Migrated `prometheus-backup` from `rclone sync` (plaintext at rest) to
restic (client-side encrypted). Same-day full-cycle drill against the
freshly-populated `restic/prometheus` repo.
- Operator: CTO agent, run [DEV-492](/DEV/issues/DEV-492).
- Repo: `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, initialised
by first CronJob run (repo id `f0eda506fd`).
- Backup runs (manual, from the deployed CronJob):
- `71420465` (2026-08-16 16:16:35Z) — first run, cold repo,
`restic backup` walk 5:10, 85 files 42 dirs, 8.017 GiB added
(2.441 GiB stored after compression + dedup), exit 0. `restic
check --read-data-subset=5%`: no errors.
- `825f983a` (2026-08-16 16:23:56Z) — second run, warm repo, 17 s
total (no new blocks — dedup 100 %), `restic check` clean.
- Drill: `harbor.basicstack.de/library/restic:0.17.3` (restore) +
`prom/prometheus:v2.53.1` (promtool) sidecars sharing an `emptyDir`
in namespace `restore-drill-dev492`.
- `restic restore latest --tag prometheus --target /work`:
`Restored 127 files/dirs (8.069 GiB) in 0:34`.
- Excluded paths verified absent on restored tree: `wal/`,
`chunks_head/`, `lost+found/` restored as empty dirs; `lock` and
`queries.active` not present at all.
- `promtool tsdb list /work/source` — 19 blocks parsed, all with
valid ULID + duration + samples/chunks/series counts.
- `promtool tsdb analyze /work/source <block>` — ran against every
block dir; 0 failed. Summary lines validated non-zero `Total
Series` and reasonable `Duration` (2h / 18h / 54h buckets, matching
Prometheus's compaction schedule).
- End-to-end `restic check`: `no errors were found` (metadata pass;
on the freshly-initialised repo `--read-data-subset=5%` was
already exercised by the writing CronJob).
- Cleanup: `restore-drill-prom` pod + `restore-drill-dev492` namespace
deleted; scratch tree discarded with the `emptyDir`. Manual
`prom-restic-manual-{01,02}` jobs deleted from `monitoring`.
**Outcome: PASS** for `restic/prometheus`. All Definition-of-Done
items on [DEV-492](/DEV/issues/DEV-492) satisfied — backup encrypted
at rest, `promtool tsdb analyze` clean, docs updated, restic password
handling documented (shared `monitoring-s3-backup` SealedSecret,
rotation procedure covers all four repos).
Next drill target for `restic/prometheus`: 2026-11-16 alongside the
other three repos.