basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/apps/monitoring/README.md
CTO Agent 63aa116ef1 Add dedicated Prometheus data backup CronJob (DEV-465) 
After the DEV-464 split, `monitoring/backup-volumes` only backs up
grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving
Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner
Cloud volume attached to whichever node currently runs the Prometheus
pod (typically k3s-worker-1), so it cannot join the shared backup-volumes
job without provoking Multi-Attach errors.

This introduces a dedicated `monitoring/prometheus-backup` CronJob that:

- Streams `prometheus-data-encrypted` to Hetzner S3 via rclone
  (`basicstack-backup/prometheus/prometheus-<DATE>/`).
- Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO
  PVC always attaches on the same node.
- Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline.
- Retains 7 days of dated backups (rclone delete --min-age 7d).
- Tolerates the expected TSDB compaction race (Prometheus deletes old
  block dirs mid-copy): rclone's non-zero exit from those transient
  errors is captured, then success is validated by comparing dest
  bytes to source bytes (>= 80% and > 100 MiB floor).

S3 credentials are the same Hetzner Object Storage account used by
`opencloud-backup` and `stalwart-backup`, resealed for the `monitoring`
namespace as `SealedSecret monitoring-s3-backup`.

Verified with a manual job on k3s-worker-1 (see DEV-465 for logs).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-09 17:18:21 +00:00

1.3 KiB
Raw Blame History

monitoring — backup CronJobs

Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (kubectl apply -f apps/monitoring/). See DEV-464 for the repair context.

  • backup-k8s-resources-cronjob.yaml — daily dump of Kubernetes resources into backup-storage PVC.
  • backup-volumes-cronjob.yaml — daily rsync/tar of Grafana + Loki PVCs into backup-storage. Prometheus data is NOT included here — it lives on a different node (see below).
  • prometheus-backup-cronjob.yaml + prometheus-backup-sealed.yaml — dedicated Prometheus data backup that streams prometheus-data-encrypted to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via podAffinity so the RWO PVC attaches on the same node (DEV-465).

The backup-storage PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for backup-k8s-resources and backup-volumes.

backup-k8s-resources and backup-volumes pin themselves to k3s-worker-2 via nodeSelector because that is the node that holds all destination + source PVCs used there. prometheus-backup follows the Prometheus pod via podAffinity, writing to Hetzner S3 (hetzner-s3:basicstack-backup/prometheus/) so it stays independent of backup-storage.