Migrates the last plaintext-at-rest monitoring backup off rclone-sync
onto the same restic pipeline as loki/grafana/k8s-resources. Closes
the Hetzner Object Storage gap flagged in DEV-482 — no SSE-S3/SSE-KMS
means every object we push must be encrypted client-side.
- apps/monitoring/prometheus-backup-cronjob.yaml: pin the image to the
Harbor mirror `harbor.basicstack.de/library/restic:0.17.3` (DEV-493)
so the pull path matches the sibling CronJobs.
- docs/monitoring/restic-restore.md: append the DEV-492 drill entry —
cold restic init, 8.017 GiB → 2.441 GiB stored, `restic check`
clean, restore of `latest` into a scratch namespace, then
`promtool tsdb list` + `promtool tsdb analyze` against all 19
blocks (0 failed). Confirms the compaction-race mitigation
(`--exclude wal/*`/`chunks_head/*` + accept exit 3) does not leave
a corrupt snapshot. Also documents the DEV-492 Prometheus repo in
the layout table + per-repo restore commands.
- apps/monitoring/README.md: point the restore section at all four
repos and cite DEV-488 (loki/k8s) + DEV-492 (prometheus) drill
entries.
Definition of done (DEV-492):
- backup ships client-side-encrypted to Hetzner: ✅ (restic init
succeeded on the manual run; snapshot 71420465 written)
- restore drill promtool tsdb analyze clean: ✅ (19/19 blocks OK)
- OBSERVABILITY_BASELINE.md + apps/monitoring/README.md updated: ✅
- restic-password ownership documented: ✅ (README shared-SealedSecret
table already covered all four repos after this change)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Point the three monitoring restic CronJobs at the Harbor-hosted
mirror at harbor.basicstack.de/library/restic:0.17.3 so the backup
pipeline is not gated by Docker Hub throttling or upstream retag.
- Mirrored docker.io/restic/restic:0.17.3 into the public
`library` project via a one-shot crane copy Job. Digest in
Harbor matches the upstream index (sha256:8f5a62b4…).
- Updated the three CronJobs
(backup-loki-restic, backup-grafana-restic,
backup-k8s-resources) to pull from Harbor.
- Documented the tag-bump procedure in
docs/monitoring/restic-restore.md (§ Tag-bump procedure) and
pointed the restore-drill pod at the same Harbor image so the
restore round-trip stays image-consistent with the writers.
Refs: [DEV-493](/DEV/issues/DEV-493), parent [DEV-482](/DEV/issues/DEV-482).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Cover credentials projection via scratch ns + monitoring-s3-backup
copy, per-repo restore commands (loki / k8s-resources / grafana),
repo self-check + source-parity flow, cleanup, and rotation.
- Record first drill (2026-08-16): loki snapshot 97092888 and
k8s-resources snapshot 9b0155cf restored; sha256 parity vs live
loki-storage-encrypted PVC confirmed for all 4 files; cluster.yaml
sanity-checked. Outcome: PASS.
- Next drill target: 2026-11-16 (quarterly); include restic/grafana
once its first CronJob run has produced >= 1 snapshot.
Unblocks retirement of the legacy backup-volumes CronJob (DEV-489).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Created detailed documentation covering:
- Authentication and SSO login flow
- Adding new applications (Git and UI methods)
- Repository credential rotation procedure
- Emergency recovery procedures (with critical safety warnings)
- Self-management architecture
- Monitoring, maintenance, and troubleshooting
- Backup and disaster recovery
Updated README.md with Argo CD section and links to docs.
Updated docs/README.md to index the new Argo CD documentation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
