Migrates the last plaintext-at-rest monitoring backup off rclone-sync
onto the same restic pipeline as loki/grafana/k8s-resources. Closes
the Hetzner Object Storage gap flagged in DEV-482 — no SSE-S3/SSE-KMS
means every object we push must be encrypted client-side.
- apps/monitoring/prometheus-backup-cronjob.yaml: pin the image to the
Harbor mirror `harbor.basicstack.de/library/restic:0.17.3` (DEV-493)
so the pull path matches the sibling CronJobs.
- docs/monitoring/restic-restore.md: append the DEV-492 drill entry —
cold restic init, 8.017 GiB → 2.441 GiB stored, `restic check`
clean, restore of `latest` into a scratch namespace, then
`promtool tsdb list` + `promtool tsdb analyze` against all 19
blocks (0 failed). Confirms the compaction-race mitigation
(`--exclude wal/*`/`chunks_head/*` + accept exit 3) does not leave
a corrupt snapshot. Also documents the DEV-492 Prometheus repo in
the layout table + per-repo restore commands.
- apps/monitoring/README.md: point the restore section at all four
repos and cite DEV-488 (loki/k8s) + DEV-492 (prometheus) drill
entries.
Definition of done (DEV-492):
- backup ships client-side-encrypted to Hetzner: ✅ (restic init
succeeded on the manual run; snapshot 71420465 written)
- restore drill promtool tsdb analyze clean: ✅ (19/19 blocks OK)
- OBSERVABILITY_BASELINE.md + apps/monitoring/README.md updated: ✅
- restic-password ownership documented: ✅ (README shared-SealedSecret
table already covered all four repos after this change)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The restic pipeline (loki/grafana/k8s-resources) is proven end-to-end
after the DEV-488 restore drill, so the legacy rsync/tar pipeline into
the 100 Gi local-path `backup-storage` PVC is removed.
- Delete `apps/monitoring/backup-volumes-cronjob.yaml`.
- Remove the DEV-483 bridge `nodeSelector: k3s-worker-2` from
`apps/monitoring/loki-deployment.yaml`. Loki's data protection now
runs via `backup-loki-restic`, which follows the pod via podAffinity
regardless of which node the RWO CSI volume attaches on. The
`Recreate` rollout strategy stays — it is unrelated (avoids the
attach-deadlock during a rollout). Resolves the RWO/nodeSelector
attach race that was blocking DEV-478 weekly OS updates.
- Update `apps/monitoring/README.md` to drop the `backup-volumes`
section, link the restic restore runbook, and record the pin
removal.
- Clean stale coexistence comments in the restic/prometheus CronJob
manifests now that the legacy job is gone.
Cluster-side (already applied out-of-band, since these manifests are
`kubectl apply`-based, not Argo-managed):
- `kubectl -n monitoring delete cronjob backup-volumes` -> NotFound.
- `kubectl -n monitoring delete pvc backup-storage` -> gone; local-path
PV `pvc-d0db0ba9-8f89-4f66-9e65-d573ebe1085a` reclaimed automatically
(Delete policy). Two stale pre-DEV-487 `backup-k8s-resources` job
pods that still referenced the PVC were deleted to release the
`pvc-protection` finalizer.
- `kubectl -n monitoring apply -f loki-deployment.yaml` -> Recreate
rollout, new pod Ready in ~60s, no nodeSelector on the new spec.
- No `VolumeAttachment` for the retired PV.
- Restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
`backup-k8s-resources`, `prometheus-backup`) intact.
Pre-delete snapshots retained on k3s-cp-1 under
`/root/dev489-snapshots-20260816T155411Z/` for post-mortem.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The original `rclone delete --min-age 7d` step ate the freshly uploaded
backup: rclone preserves each source file's mtime, and Prometheus TSDB
chunk/block files retain very old mtimes (weeks-old immutable blocks),
so filtering by file age deleted ~72% of the objects immediately after
sync (verified: 8.764 GiB destination reduced to 2.310 GiB / 37 objects
before we noticed).
Replaced the mtime prune with a prefix-name-based prune: every top-level
prefix is `prometheus-YYYYMMDD-HHMMSS`, so we parse the encoded date and
`rclone purge` whole prefixes older than 7 days. This keeps the latest
7 daily snapshots intact regardless of the Prometheus block ages.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
After the DEV-464 split, `monitoring/backup-volumes` only backs up
grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving
Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner
Cloud volume attached to whichever node currently runs the Prometheus
pod (typically k3s-worker-1), so it cannot join the shared backup-volumes
job without provoking Multi-Attach errors.
This introduces a dedicated `monitoring/prometheus-backup` CronJob that:
- Streams `prometheus-data-encrypted` to Hetzner S3 via rclone
(`basicstack-backup/prometheus/prometheus-<DATE>/`).
- Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO
PVC always attaches on the same node.
- Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline.
- Retains 7 days of dated backups (rclone delete --min-age 7d).
- Tolerates the expected TSDB compaction race (Prometheus deletes old
block dirs mid-copy): rclone's non-zero exit from those transient
errors is captured, then success is validated by comparing dest
bytes to source bytes (>= 80% and > 100 MiB floor).
S3 credentials are the same Hetzner Object Storage account used by
`opencloud-backup` and `stalwart-backup`, resealed for the `monitoring`
namespace as `SealedSecret monitoring-s3-backup`.
Verified with a manual job on k3s-worker-1 (see DEV-465 for logs).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
