63aa116ef1
After the DEV-464 split, `monitoring/backup-volumes` only backs up grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner Cloud volume attached to whichever node currently runs the Prometheus pod (typically k3s-worker-1), so it cannot join the shared backup-volumes job without provoking Multi-Attach errors. This introduces a dedicated `monitoring/prometheus-backup` CronJob that: - Streams `prometheus-data-encrypted` to Hetzner S3 via rclone (`basicstack-backup/prometheus/prometheus-<DATE>/`). - Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO PVC always attaches on the same node. - Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline. - Retains 7 days of dated backups (rclone delete --min-age 7d). - Tolerates the expected TSDB compaction race (Prometheus deletes old block dirs mid-copy): rclone's non-zero exit from those transient errors is captured, then success is validated by comparing dest bytes to source bytes (>= 80% and > 100 MiB floor). S3 credentials are the same Hetzner Object Storage account used by `opencloud-backup` and `stalwart-backup`, resealed for the `monitoring` namespace as `SealedSecret monitoring-s3-backup`. Verified with a manual job on k3s-worker-1 (see DEV-465 for logs). Co-Authored-By: Paperclip <noreply@paperclip.ing>
11 lines
1.3 KiB
Markdown
11 lines
1.3 KiB
Markdown
# monitoring — backup CronJobs
|
|
|
|
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
|
|
|
|
- `backup-k8s-resources-cronjob.yaml` — daily dump of Kubernetes resources into `backup-storage` PVC.
|
|
- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below).
|
|
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)).
|
|
|
|
The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for `backup-k8s-resources` and `backup-volumes`.
|
|
|
|
`backup-k8s-resources` and `backup-volumes` pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used there. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`.
|