From 63aa116ef104d0572ec6baa783409ebf151b511d Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 9 Aug 2026 17:18:21 +0000 Subject: [PATCH] Add dedicated Prometheus data backup CronJob (DEV-465) After the DEV-464 split, `monitoring/backup-volumes` only backs up grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner Cloud volume attached to whichever node currently runs the Prometheus pod (typically k3s-worker-1), so it cannot join the shared backup-volumes job without provoking Multi-Attach errors. This introduces a dedicated `monitoring/prometheus-backup` CronJob that: - Streams `prometheus-data-encrypted` to Hetzner S3 via rclone (`basicstack-backup/prometheus/prometheus-/`). - Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO PVC always attaches on the same node. - Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline. - Retains 7 days of dated backups (rclone delete --min-age 7d). - Tolerates the expected TSDB compaction race (Prometheus deletes old block dirs mid-copy): rclone's non-zero exit from those transient errors is captured, then success is validated by comparing dest bytes to source bytes (>= 80% and > 100 MiB floor). S3 credentials are the same Hetzner Object Storage account used by `opencloud-backup` and `stalwart-backup`, resealed for the `monitoring` namespace as `SealedSecret monitoring-s3-backup`. Verified with a manual job on k3s-worker-1 (see DEV-465 for logs). Co-Authored-By: Paperclip --- apps/monitoring/README.md | 7 +- .../monitoring/prometheus-backup-cronjob.yaml | 176 ++++++++++++++++++ apps/monitoring/prometheus-backup-sealed.yaml | 19 ++ 3 files changed, 199 insertions(+), 3 deletions(-) create mode 100644 apps/monitoring/prometheus-backup-cronjob.yaml create mode 100644 apps/monitoring/prometheus-backup-sealed.yaml diff --git a/apps/monitoring/README.md b/apps/monitoring/README.md index 8ab565e..f381914 100644 --- a/apps/monitoring/README.md +++ b/apps/monitoring/README.md @@ -3,8 +3,9 @@ Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context. - `backup-k8s-resources-cronjob.yaml` — daily dump of Kubernetes resources into `backup-storage` PVC. -- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data backup is **not** included here; it needs a separate on-node backup (tracked as a follow-up because Prometheus is on a different node than `backup-storage`). +- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below). +- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)). -The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for both jobs. +The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for `backup-k8s-resources` and `backup-volumes`. -Both CronJobs pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used here. +`backup-k8s-resources` and `backup-volumes` pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used there. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`. diff --git a/apps/monitoring/prometheus-backup-cronjob.yaml b/apps/monitoring/prometheus-backup-cronjob.yaml new file mode 100644 index 0000000..80f8ee2 --- /dev/null +++ b/apps/monitoring/prometheus-backup-cronjob.yaml @@ -0,0 +1,176 @@ +--- +# Prometheus data backup (DEV-465). +# +# Prometheus data lives on the RWO PVC `prometheus-data-encrypted` in +# namespace `monitoring`. That PVC is mounted by the Prometheus pod which +# currently lives on k3s-worker-1, and the Hetzner CSI volume can only be +# attached to one node at a time. The shared `backup-volumes` CronJob (see +# `apps/monitoring/backup-volumes-cronjob.yaml`) is pinned to k3s-worker-2 +# (where grafana + loki live) and therefore cannot back up Prometheus. +# +# This CronJob co-schedules with the Prometheus pod via podAffinity, so it +# lands on whichever node currently holds `prometheus-data-encrypted`. The +# PVC is mounted read-only alongside the running Prometheus pod (RWO permits +# additional read-only mounts on the same node) and streamed to Hetzner S3 +# via rclone under `basicstack-backup/prometheus/prometheus-/`. Old +# snapshots are pruned after 7 days. +apiVersion: batch/v1 +kind: CronJob +metadata: + name: prometheus-backup + namespace: monitoring + labels: + app: backup + type: prometheus +spec: + schedule: "30 3 * * *" # daily 03:30, offset from backup-volumes (03:00) + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + metadata: + labels: + app: backup + type: prometheus + spec: + backoffLimit: 2 + activeDeadlineSeconds: 3600 # 60 min hard cap; Prometheus data is ~9GB + template: + metadata: + labels: + app: backup + type: prometheus + spec: + restartPolicy: OnFailure + # Co-schedule with the Prometheus pod so the RWO PVC attaches on + # the same node. This survives Prometheus being rescheduled to a + # different worker (the backup follows). + affinity: + podAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app + operator: In + values: + - prometheus + topologyKey: kubernetes.io/hostname + containers: + - name: prometheus-backup + image: rclone/rclone:1.68 + command: + - /bin/sh + - -c + - | + set -eu + DATE=$(date +%Y%m%d-%H%M%S) + echo "=== Prometheus backup started at $(date) (target prefix: prometheus-${DATE}) ===" + + mkdir -p /root/.config/rclone + cat > /root/.config/rclone/rclone.conf </dev/null | cut -f1 || echo 0) + SOURCE_HUMAN=$(du -sh /source 2>/dev/null | cut -f1 || echo unknown) + echo "Source /source size: ${SOURCE_HUMAN} (${SOURCE_BYTES} bytes)" + + # Prometheus TSDB is largely immutable chunk files plus an + # append-only WAL. rclone sync is safe with the database + # live: on restore the WAL is replayed. HOWEVER Prometheus + # compacts blocks every ~2h and deletes their source dirs, + # which races with the copy and produces "no such file or + # directory" errors mid-run. Those are expected and do + # not indicate data loss — the compacted successor blocks + # are picked up on the same or the next daily run. We + # therefore do not fail the job on rclone's non-zero exit + # from those transient errors; instead we validate the + # backup by comparing destination size to source (must be + # >= 80% of source bytes and > 100 MiB). + DEST="hetzner-s3:${S3_BUCKET}/prometheus/prometheus-${DATE}/" + echo "Streaming Prometheus data to ${DEST} ..." + RCLONE_EXIT=0 + rclone sync /source "${DEST}" \ + --transfers 4 \ + --checkers 4 \ + --stats 30s \ + --stats-log-level NOTICE \ + --s3-chunk-size 32M \ + --s3-upload-concurrency 4 \ + --retries 3 \ + --retries-sleep 30s || RCLONE_EXIT=$? + echo "rclone sync exit code: ${RCLONE_EXIT}" + + echo "Measuring destination size..." + DEST_BYTES=$(rclone size "${DEST}" --json 2>/dev/null | \ + sed -n 's/.*"bytes":\s*\([0-9]\+\).*/\1/p' | head -1) + DEST_BYTES=${DEST_BYTES:-0} + DEST_HUMAN=$(rclone size "${DEST}" 2>/dev/null | \ + grep -oE 'Total size:.*' || echo "Total size: unknown") + echo "Destination bytes: ${DEST_BYTES}" + echo "Destination summary: ${DEST_HUMAN}" + + MIN_ACCEPTABLE=$(( SOURCE_BYTES * 80 / 100 )) + FLOOR=104857600 # 100 MiB absolute floor + echo "Acceptance threshold: dest >= ${MIN_ACCEPTABLE} bytes and > ${FLOOR} bytes" + if [ "${DEST_BYTES}" -lt "${FLOOR}" ]; then + echo "ERROR: destination is below hard floor (100 MiB) — backup failed." + exit 2 + fi + if [ "${DEST_BYTES}" -lt "${MIN_ACCEPTABLE}" ]; then + echo "ERROR: destination is < 80% of source (${DEST_BYTES} < ${MIN_ACCEPTABLE}) — backup incomplete." + exit 3 + fi + echo "OK: destination size acceptable." + + echo "Pruning prometheus backups older than 7 days..." + rclone delete "hetzner-s3:${S3_BUCKET}/prometheus/" \ + --min-age 7d || echo "prune step reported errors (continuing)" + rclone rmdirs "hetzner-s3:${S3_BUCKET}/prometheus/" --leave-root || true + + echo "Post-run inventory (prometheus/ prefixes):" + rclone lsd "hetzner-s3:${S3_BUCKET}/prometheus/" || true + + echo "=== Prometheus backup completed at $(date) ===" + env: + - name: S3_ACCESS_KEY + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: access-key + - name: S3_SECRET_KEY + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: secret-key + - name: S3_ENDPOINT + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: endpoint + - name: S3_BUCKET + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: bucket + volumeMounts: + - name: prometheus-data + mountPath: /source + readOnly: true + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 1000m + memory: 512Mi + volumes: + - name: prometheus-data + persistentVolumeClaim: + claimName: prometheus-data-encrypted diff --git a/apps/monitoring/prometheus-backup-sealed.yaml b/apps/monitoring/prometheus-backup-sealed.yaml new file mode 100644 index 0000000..1b30c2e --- /dev/null +++ b/apps/monitoring/prometheus-backup-sealed.yaml @@ -0,0 +1,19 @@ +--- +apiVersion: bitnami.com/v1alpha1 +kind: SealedSecret +metadata: + creationTimestamp: null + name: monitoring-s3-backup + namespace: monitoring +spec: + encryptedData: + access-key: AgBf1XxCF3/Uv1vVlJHyTSEMONqWBqPfSOax8jJJHy3NOpgGga3SsxDfAbPXe2rBOYSgGzRZcaT8TcNg8Ng1ipv5Dhu7LzmRb6v6fW0NimrHSU/dtzzJSIR7jQ1sf+Qn4CPS73ReDRvUJ7g4o57ClZIVFBk9oPUDJYdjtg9kAM5Me0ZyqVp4Cnyj2TRXYcQPM+RN1r/PXqK4MhlUsmhsbQCPYwocM0liqv0T8L5xR1g2845oMaMx2nmLs1c1R8nCRugK8IeBKf/CCKLYg8yTqWKS2arrfxm+dkl1mrqLdOgnXrqeeo/63lT1sbd06ub86PJr83MXWW0yh1Uc4r7DX7xK/JIGJ4w+jtQa0vRutYANDcdeVlzjVl+ghoxxw/ZizYVQ/mUFCc+ApJD9u709mI39xof77U5yYBRhnZEExRseL756APubmCc0tPfUSNA1GmMhiYaMnyWu6F5Okreto/03AgI3t7lQm9SPRRM7hcHETG/QC8PrZ0NZEN/A2Nm/ZzwDG4HPDaNxH+IV90f5X4wuh6Mh2lffidOix88rkaizQAeVY5gt8tvfSfyfRc5RoOOov+fvUVCqJHnWaD1A2lKScVeCgLv8/0CJTyR0zM6cg2XN8z0HvU/sahfZrG23fVs3E/IFM5D7//CDp+GfcwQOCBZKPLvJtTRh5nA5EbXQusGEaMOTBiiGG0IJKPejKcPRWS1g+owu+G5R6QQjZgNDoP0cPg== + bucket: AgCYDCrQalif3+XZYhe4Xxyypg2OMGvzdNyBmHzMaP+ou1g2Q0wIjH6S1+YpPu+JgmaATZVISoSZ+OIl0mNOibiFMr0kFjuMhvKRt9Pq05+6axnqnJ17U2GNsZQtY3Ic93qxkkhJKCOQ87mvAJ5AIVBc3pAqfkD5YYTllgnSKwUwyaWK3iX3hxuKLYDvsQ2WTtcc/gsnlCkm1dwPE4E7DFt4wh+jQJRD314Mt7KSpvPBpz7yVVFFX+xRKCfRxjcwshUSUI5OgagfOQvsivkxlL0oSqA9veX2MJpd0SJH0T1Y0na0evkJitxxe+ECwM4FluK/iwW4cdUu+EpQIdd5otY9C5b8e8PJp8ah1k1hYzXdX+j975MeyXWMUn/NgXfyT2CUef6hmmmImialrSYsHT72bz3xpAnZ/oQ/0qcZpNje2oJeokNgkoiC0E5XRnfwEF9jL3XL+aB6gQ3BD9YRptaxLLxOQNj51Oe82vVsOnUz+utzJO1DtqpF+8I0lvHfgcUhFQytUuLscilfBQO/ru8wh47ZI+LRiS3qCZp2Pqi9+r7Y7jJKcbb2awWp6lgrrqgLMvnbVMuvxBi2hUy0C+EkP8tayRJDzz72utx8FY/v0FWoRDl9BAi8kn11+kbbo8a2OXe+20UkmNzunz5H0ijIxhJWzVh/oLuuLFCS0YJtJNjuqUB0VctAFcR4p5Agua8LA9+73HIBsHBYexzmvEZ6Ng== + endpoint: AgCV6YtzikWYMRwHZM1tT8epq6XiSVqhbxMhS7htVjVUfdR4V7/KIcNExxSBul+TU+RCq3S7fppL7CxKbI/IsEdirRMj2PvESXEmzLY/xId7+wJFEdxeXxn59cbV2MfCyysKIwzsVXbmzO66yGaeMKU2oyqJVYu+00L9xS0RbtQRbi0UksMxtw0PhZVMee0aHfu9jynWI3lmPzC/EPRuwxQqBIjYM2v2JCDmYjP0fsBSZSJc+SnAaNmmIf7ypcFwy14qIF7C9tIg02lBx+Sg30oMPjn600AbyF6RmZ+mdclp/Qp+zXOUsbMym0Lu9BjbM7G/z0ExpR5sgE/6+05gJhPF7MuHkrYIMo9LHWWdgzNTnjikSTGEeuFzzRlThPMoUfVi/S7KhG9NtQRnNqEoVCGGW6tt5sygXVE/lWSnmmd//4jXvF9ZJHU+5LrjL7UfVIYNH4y7aYtXpoTgmUe85wWqeMdqr1EiSVueoSwCggGITYoXoKYfERfij8ln/xnX1Kt2xxuyLtpDOczBYZK6LLwTpaXVVe+lO/xBgevWaoEz6si2hyj+glluMQWxPsOnPwUfRi5qqlQxgMZK/9aALYhPLquc16iMG+4FdtoOUfQ0XlLvurHsbCt9lUy3YrsmpDhv42FfamVlcK+gMz7mqYhNhd3fVjxCiUuwvzd01QHcx8ikv48YOnDuRiweRszbPNvlaK2KH6fo7b4l9mmaa4W+MaxZCxG3KSMOafwRmyusEUVNWw== + secret-key: AgBItdfzRgcgJvK5HALDnkbGH1DveAgcmS+Q4f16VwX1KY8TzGWpjoaOdsLCwzTWOFME5xjrRyOwwAcdtP65ccWlEZvvoS4kpqv49G1ebblD1wmApnpa8vD63Rd028vUYFF3CnwntX2USxF6t4rD68sZqqzMDyuG9Av93HAKVMNdx9V4AJ45nvfF2WR7ohk37NhQifOdKuelgRV6ahl3/fm/XPRGzf+u7OER8jll9n/ywE0mSkoKBXocbkQ2Xrn+utzplh9BoR9KSPTdnmI8ZXUkDTR9p+7PN4r/HheEdTUG0aaefdEDCbdDlv1EW84l5KzhY4v9g0DW8GQQdyHWi3sMm0hSGFg9NV/Vs1HM9FZZCyjS+mzR4CbuHgkY9mvj+Svn7oS+Fy5d24s1ekmPheakBffXKy2Q2CLMRGNymjUIBQsDwoGC8o4IF+yPyDcw/OvXlr0lruxfRM6IN0oxgRmLAQiqwRzsZb3RPgILEZI9VXKxzCBEuCO7fE2pC05K/mujhUutWmJ5tQ0NFzP1mhNK59cAIVMl7y7Llq5/GZ5ax+voP0c3VandXJeaSRr6O40vyzfCMgpgEgRs9t2WfxCmVYEN2NWpFv07JNICWzJX40PLT9n/euRXzSmJld6UfjzRteaTfmo+OxFczJ6Zl8XdC+6x6wO0Nk0hVYo27s2LH4o0MsM84x9Z3StCCGl77/NnyavpxfDkj3qJQ4BgPtCw26tHTt9DIWYfgeAVqvoRZVlg8YxM3N1k + template: + metadata: + creationTimestamp: null + name: monitoring-s3-backup + namespace: monitoring + type: Opaque