basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/apps/monitoring
History
CTO Agent a5ceedf4da fix(monitoring): prune Prometheus S3 backups by prefix-date, not file mtime (DEV-465) 
The original `rclone delete --min-age 7d` step ate the freshly uploaded
backup: rclone preserves each source file's mtime, and Prometheus TSDB
chunk/block files retain very old mtimes (weeks-old immutable blocks),
so filtering by file age deleted ~72% of the objects immediately after
sync (verified: 8.764 GiB destination reduced to 2.310 GiB / 37 objects
before we noticed).

Replaced the mtime prune with a prefix-name-based prune: every top-level
prefix is `prometheus-YYYYMMDD-HHMMSS`, so we parse the encoded date and
`rclone purge` whole prefixes older than 7 days. This keeps the latest
7 daily snapshots intact regardless of the Prometheus block ages.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-09 17:40:50 +00:00
..
backup-k8s-resources-cronjob.yaml fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464) 2026-08-09 16:51:30 +00:00
backup-volumes-cronjob.yaml fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464) 2026-08-09 16:51:30 +00:00
prometheus-backup-cronjob.yaml fix(monitoring): prune Prometheus S3 backups by prefix-date, not file mtime (DEV-465) 2026-08-09 17:40:50 +00:00
prometheus-backup-sealed.yaml Add dedicated Prometheus data backup CronJob (DEV-465) 2026-08-09 17:18:21 +00:00
README.md Add dedicated Prometheus data backup CronJob (DEV-465) 2026-08-09 17:18:21 +00:00

README.md

monitoring — backup CronJobs

Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (kubectl apply -f apps/monitoring/). See DEV-464 for the repair context.

  • backup-k8s-resources-cronjob.yaml — daily dump of Kubernetes resources into backup-storage PVC.
  • backup-volumes-cronjob.yaml — daily rsync/tar of Grafana + Loki PVCs into backup-storage. Prometheus data is NOT included here — it lives on a different node (see below).
  • prometheus-backup-cronjob.yaml + prometheus-backup-sealed.yaml — dedicated Prometheus data backup that streams prometheus-data-encrypted to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via podAffinity so the RWO PVC attaches on the same node (DEV-465).

The backup-storage PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for backup-k8s-resources and backup-volumes.

backup-k8s-resources and backup-volumes pin themselves to k3s-worker-2 via nodeSelector because that is the node that holds all destination + source PVCs used there. prometheus-backup follows the Prometheus pod via podAffinity, writing to Hetzner S3 (hetzner-s3:basicstack-backup/prometheus/) so it stays independent of backup-storage.