diff --git a/apps/monitoring/README.md b/apps/monitoring/README.md index 1b8bd14..8e7215c 100644 --- a/apps/monitoring/README.md +++ b/apps/monitoring/README.md @@ -22,7 +22,7 @@ All four restic CronJobs read Hetzner S3 credentials + the restic repository pas ## Restore -Restore procedure for all three restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16) passed — see [DEV-488](/DEV/issues/DEV-488). +Restore procedure for all four restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16, loki/k8s-resources) passed — see [DEV-488](/DEV/issues/DEV-488). Prometheus repo added and drilled same day — see [DEV-492](/DEV/issues/DEV-492). ## Emitted metrics (textfile-collector format) diff --git a/apps/monitoring/prometheus-backup-cronjob.yaml b/apps/monitoring/prometheus-backup-cronjob.yaml index 705fab2..5c3bd7b 100644 --- a/apps/monitoring/prometheus-backup-cronjob.yaml +++ b/apps/monitoring/prometheus-backup-cronjob.yaml @@ -81,7 +81,7 @@ spec: topologyKey: kubernetes.io/hostname containers: - name: restic - image: restic/restic:0.17.3 + image: harbor.basicstack.de/library/restic:0.17.3 env: - name: AWS_ACCESS_KEY_ID valueFrom: diff --git a/docs/monitoring/restic-restore.md b/docs/monitoring/restic-restore.md index e07603c..f185926 100644 --- a/docs/monitoring/restic-restore.md +++ b/docs/monitoring/restic-restore.md @@ -416,3 +416,49 @@ Every drill appends to this section. Include: Next drill target: 2026-11-16 (quarterly). Include `restic/grafana` once its first CronJob run has produced ≥ 1 snapshot. + +### 2026-08-16 — Prometheus migration drill (DEV-492) + +Migrated `prometheus-backup` from `rclone sync` (plaintext at rest) to +restic (client-side encrypted). Same-day full-cycle drill against the +freshly-populated `restic/prometheus` repo. + +- Operator: CTO agent, run [DEV-492](/DEV/issues/DEV-492). +- Repo: `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, initialised + by first CronJob run (repo id `f0eda506fd`). +- Backup runs (manual, from the deployed CronJob): + - `71420465` (2026-08-16 16:16:35Z) — first run, cold repo, + `restic backup` walk 5:10, 85 files 42 dirs, 8.017 GiB added + (2.441 GiB stored after compression + dedup), exit 0. `restic + check --read-data-subset=5%`: no errors. + - `825f983a` (2026-08-16 16:23:56Z) — second run, warm repo, 17 s + total (no new blocks — dedup 100 %), `restic check` clean. +- Drill: `harbor.basicstack.de/library/restic:0.17.3` (restore) + + `prom/prometheus:v2.53.1` (promtool) sidecars sharing an `emptyDir` + in namespace `restore-drill-dev492`. + - `restic restore latest --tag prometheus --target /work`: + `Restored 127 files/dirs (8.069 GiB) in 0:34`. + - Excluded paths verified absent on restored tree: `wal/`, + `chunks_head/`, `lost+found/` restored as empty dirs; `lock` and + `queries.active` not present at all. + - `promtool tsdb list /work/source` — 19 blocks parsed, all with + valid ULID + duration + samples/chunks/series counts. + - `promtool tsdb analyze /work/source ` — ran against every + block dir; 0 failed. Summary lines validated non-zero `Total + Series` and reasonable `Duration` (2h / 18h / 54h buckets, matching + Prometheus's compaction schedule). + - End-to-end `restic check`: `no errors were found` (metadata pass; + on the freshly-initialised repo `--read-data-subset=5%` was + already exercised by the writing CronJob). +- Cleanup: `restore-drill-prom` pod + `restore-drill-dev492` namespace + deleted; scratch tree discarded with the `emptyDir`. Manual + `prom-restic-manual-{01,02}` jobs deleted from `monitoring`. + +**Outcome: PASS** for `restic/prometheus`. All Definition-of-Done +items on [DEV-492](/DEV/issues/DEV-492) satisfied — backup encrypted +at rest, `promtool tsdb analyze` clean, docs updated, restic password +handling documented (shared `monitoring-s3-backup` SealedSecret, +rotation procedure covers all four repos). + +Next drill target for `restic/prometheus`: 2026-11-16 alongside the +other three repos.