Add dedicated Prometheus data backup CronJob (DEV-465)

After the DEV-464 split, `monitoring/backup-volumes` only backs up
grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving
Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner
Cloud volume attached to whichever node currently runs the Prometheus
pod (typically k3s-worker-1), so it cannot join the shared backup-volumes
job without provoking Multi-Attach errors.

This introduces a dedicated `monitoring/prometheus-backup` CronJob that:

- Streams `prometheus-data-encrypted` to Hetzner S3 via rclone
  (`basicstack-backup/prometheus/prometheus-<DATE>/`).
- Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO
  PVC always attaches on the same node.
- Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline.
- Retains 7 days of dated backups (rclone delete --min-age 7d).
- Tolerates the expected TSDB compaction race (Prometheus deletes old
  block dirs mid-copy): rclone's non-zero exit from those transient
  errors is captured, then success is validated by comparing dest
  bytes to source bytes (>= 80% and > 100 MiB floor).

S3 credentials are the same Hetzner Object Storage account used by
`opencloud-backup` and `stalwart-backup`, resealed for the `monitoring`
namespace as `SealedSecret monitoring-s3-backup`.

Verified with a manual job on k3s-worker-1 (see DEV-465 for logs).

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/monitoring/README.md

@@ -3,8 +3,9 @@
 Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
 
 - `backup-k8s-resources-cronjob.yaml` — daily dump of Kubernetes resources into `backup-storage` PVC.
-- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data backup is **not** included here; it needs a separate on-node backup (tracked as a follow-up because Prometheus is on a different node than `backup-storage`).
+- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below).
+- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)).
 
-The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for both jobs.
+The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for `backup-k8s-resources` and `backup-volumes`.
 
-Both CronJobs pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used here.
+`backup-k8s-resources` and `backup-volumes` pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used there. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`.

apps/monitoring/prometheus-backup-cronjob.yaml

@@ -0,0 +1,176 @@
+---
+# Prometheus data backup (DEV-465).
+#
+# Prometheus data lives on the RWO PVC `prometheus-data-encrypted` in
+# namespace `monitoring`. That PVC is mounted by the Prometheus pod which
+# currently lives on k3s-worker-1, and the Hetzner CSI volume can only be
+# attached to one node at a time. The shared `backup-volumes` CronJob (see
+# `apps/monitoring/backup-volumes-cronjob.yaml`) is pinned to k3s-worker-2
+# (where grafana + loki live) and therefore cannot back up Prometheus.
+#
+# This CronJob co-schedules with the Prometheus pod via podAffinity, so it
+# lands on whichever node currently holds `prometheus-data-encrypted`. The
+# PVC is mounted read-only alongside the running Prometheus pod (RWO permits
+# additional read-only mounts on the same node) and streamed to Hetzner S3
+# via rclone under `basicstack-backup/prometheus/prometheus-<DATE>/`. Old
+# snapshots are pruned after 7 days.
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: prometheus-backup
+  namespace: monitoring
+  labels:
+    app: backup
+    type: prometheus
+spec:
+  schedule: "30 3 * * *"  # daily 03:30, offset from backup-volumes (03:00)
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    metadata:
+      labels:
+        app: backup
+        type: prometheus
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 3600  # 60 min hard cap; Prometheus data is ~9GB
+      template:
+        metadata:
+          labels:
+            app: backup
+            type: prometheus
+        spec:
+          restartPolicy: OnFailure
+          # Co-schedule with the Prometheus pod so the RWO PVC attaches on
+          # the same node. This survives Prometheus being rescheduled to a
+          # different worker (the backup follows).
+          affinity:
+            podAffinity:
+              requiredDuringSchedulingIgnoredDuringExecution:
+                - labelSelector:
+                    matchExpressions:
+                      - key: app
+                        operator: In
+                        values:
+                          - prometheus
+                  topologyKey: kubernetes.io/hostname
+          containers:
+            - name: prometheus-backup
+              image: rclone/rclone:1.68
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  DATE=$(date +%Y%m%d-%H%M%S)
+                  echo "=== Prometheus backup started at $(date) (target prefix: prometheus-${DATE}) ==="
+
+                  mkdir -p /root/.config/rclone
+                  cat > /root/.config/rclone/rclone.conf <<EOC
+                  [hetzner-s3]
+                  type = s3
+                  provider = Other
+                  access_key_id = ${S3_ACCESS_KEY}
+                  secret_access_key = ${S3_SECRET_KEY}
+                  endpoint = ${S3_ENDPOINT}
+                  acl = private
+                  EOC
+
+                  SOURCE_BYTES=$(du -sb /source 2>/dev/null | cut -f1 || echo 0)
+                  SOURCE_HUMAN=$(du -sh /source 2>/dev/null | cut -f1 || echo unknown)
+                  echo "Source /source size: ${SOURCE_HUMAN} (${SOURCE_BYTES} bytes)"
+
+                  # Prometheus TSDB is largely immutable chunk files plus an
+                  # append-only WAL. rclone sync is safe with the database
+                  # live: on restore the WAL is replayed. HOWEVER Prometheus
+                  # compacts blocks every ~2h and deletes their source dirs,
+                  # which races with the copy and produces "no such file or
+                  # directory" errors mid-run. Those are expected and do
+                  # not indicate data loss — the compacted successor blocks
+                  # are picked up on the same or the next daily run. We
+                  # therefore do not fail the job on rclone's non-zero exit
+                  # from those transient errors; instead we validate the
+                  # backup by comparing destination size to source (must be
+                  # >= 80% of source bytes and > 100 MiB).
+                  DEST="hetzner-s3:${S3_BUCKET}/prometheus/prometheus-${DATE}/"
+                  echo "Streaming Prometheus data to ${DEST} ..."
+                  RCLONE_EXIT=0
+                  rclone sync /source "${DEST}" \
+                    --transfers 4 \
+                    --checkers 4 \
+                    --stats 30s \
+                    --stats-log-level NOTICE \
+                    --s3-chunk-size 32M \
+                    --s3-upload-concurrency 4 \
+                    --retries 3 \
+                    --retries-sleep 30s || RCLONE_EXIT=$?
+                  echo "rclone sync exit code: ${RCLONE_EXIT}"
+
+                  echo "Measuring destination size..."
+                  DEST_BYTES=$(rclone size "${DEST}" --json 2>/dev/null | \
+                    sed -n 's/.*"bytes":\s*\([0-9]\+\).*/\1/p' | head -1)
+                  DEST_BYTES=${DEST_BYTES:-0}
+                  DEST_HUMAN=$(rclone size "${DEST}" 2>/dev/null | \
+                    grep -oE 'Total size:.*' || echo "Total size: unknown")
+                  echo "Destination bytes: ${DEST_BYTES}"
+                  echo "Destination summary: ${DEST_HUMAN}"
+
+                  MIN_ACCEPTABLE=$(( SOURCE_BYTES * 80 / 100 ))
+                  FLOOR=104857600 # 100 MiB absolute floor
+                  echo "Acceptance threshold: dest >= ${MIN_ACCEPTABLE} bytes and > ${FLOOR} bytes"
+                  if [ "${DEST_BYTES}" -lt "${FLOOR}" ]; then
+                    echo "ERROR: destination is below hard floor (100 MiB) — backup failed."
+                    exit 2
+                  fi
+                  if [ "${DEST_BYTES}" -lt "${MIN_ACCEPTABLE}" ]; then
+                    echo "ERROR: destination is < 80% of source (${DEST_BYTES} < ${MIN_ACCEPTABLE}) — backup incomplete."
+                    exit 3
+                  fi
+                  echo "OK: destination size acceptable."
+
+                  echo "Pruning prometheus backups older than 7 days..."
+                  rclone delete "hetzner-s3:${S3_BUCKET}/prometheus/" \
+                    --min-age 7d || echo "prune step reported errors (continuing)"
+                  rclone rmdirs "hetzner-s3:${S3_BUCKET}/prometheus/" --leave-root || true
+
+                  echo "Post-run inventory (prometheus/ prefixes):"
+                  rclone lsd "hetzner-s3:${S3_BUCKET}/prometheus/" || true
+
+                  echo "=== Prometheus backup completed at $(date) ==="
+              env:
+                - name: S3_ACCESS_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: access-key
+                - name: S3_SECRET_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: secret-key
+                - name: S3_ENDPOINT
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: endpoint
+                - name: S3_BUCKET
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: bucket
+              volumeMounts:
+                - name: prometheus-data
+                  mountPath: /source
+                  readOnly: true
+              resources:
+                requests:
+                  cpu: 100m
+                  memory: 128Mi
+                limits:
+                  cpu: 1000m
+                  memory: 512Mi
+          volumes:
+            - name: prometheus-data
+              persistentVolumeClaim:
+                claimName: prometheus-data-encrypted

apps/monitoring/prometheus-backup-sealed.yaml

@@ -0,0 +1,19 @@
+---
+apiVersion: bitnami.com/v1alpha1
+kind: SealedSecret
+metadata:
+  creationTimestamp: null
+  name: monitoring-s3-backup
+  namespace: monitoring
+spec:
+  encryptedData:
+    access-key: AgBf1XxCF3/Uv1vVlJHyTSEMONqWBqPfSOax8jJJHy3NOpgGga3SsxDfAbPXe2rBOYSgGzRZcaT8TcNg8Ng1ipv5Dhu7LzmRb6v6fW0NimrHSU/dtzzJSIR7jQ1sf+Qn4CPS73ReDRvUJ7g4o57ClZIVFBk9oPUDJYdjtg9kAM5Me0ZyqVp4Cnyj2TRXYcQPM+RN1r/PXqK4MhlUsmhsbQCPYwocM0liqv0T8L5xR1g2845oMaMx2nmLs1c1R8nCRugK8IeBKf/CCKLYg8yTqWKS2arrfxm+dkl1mrqLdOgnXrqeeo/63lT1sbd06ub86PJr83MXWW0yh1Uc4r7DX7xK/JIGJ4w+jtQa0vRutYANDcdeVlzjVl+ghoxxw/ZizYVQ/mUFCc+ApJD9u709mI39xof77U5yYBRhnZEExRseL756APubmCc0tPfUSNA1GmMhiYaMnyWu6F5Okreto/03AgI3t7lQm9SPRRM7hcHETG/QC8PrZ0NZEN/A2Nm/ZzwDG4HPDaNxH+IV90f5X4wuh6Mh2lffidOix88rkaizQAeVY5gt8tvfSfyfRc5RoOOov+fvUVCqJHnWaD1A2lKScVeCgLv8/0CJTyR0zM6cg2XN8z0HvU/sahfZrG23fVs3E/IFM5D7//CDp+GfcwQOCBZKPLvJtTRh5nA5EbXQusGEaMOTBiiGG0IJKPejKcPRWS1g+owu+G5R6QQjZgNDoP0cPg==
+    bucket: AgCYDCrQalif3+XZYhe4Xxyypg2OMGvzdNyBmHzMaP+ou1g2Q0wIjH6S1+YpPu+JgmaATZVISoSZ+OIl0mNOibiFMr0kFjuMhvKRt9Pq05+6axnqnJ17U2GNsZQtY3Ic93qxkkhJKCOQ87mvAJ5AIVBc3pAqfkD5YYTllgnSKwUwyaWK3iX3hxuKLYDvsQ2WTtcc/gsnlCkm1dwPE4E7DFt4wh+jQJRD314Mt7KSpvPBpz7yVVFFX+xRKCfRxjcwshUSUI5OgagfOQvsivkxlL0oSqA9veX2MJpd0SJH0T1Y0na0evkJitxxe+ECwM4FluK/iwW4cdUu+EpQIdd5otY9C5b8e8PJp8ah1k1hYzXdX+j975MeyXWMUn/NgXfyT2CUef6hmmmImialrSYsHT72bz3xpAnZ/oQ/0qcZpNje2oJeokNgkoiC0E5XRnfwEF9jL3XL+aB6gQ3BD9YRptaxLLxOQNj51Oe82vVsOnUz+utzJO1DtqpF+8I0lvHfgcUhFQytUuLscilfBQO/ru8wh47ZI+LRiS3qCZp2Pqi9+r7Y7jJKcbb2awWp6lgrrqgLMvnbVMuvxBi2hUy0C+EkP8tayRJDzz72utx8FY/v0FWoRDl9BAi8kn11+kbbo8a2OXe+20UkmNzunz5H0ijIxhJWzVh/oLuuLFCS0YJtJNjuqUB0VctAFcR4p5Agua8LA9+73HIBsHBYexzmvEZ6Ng==
+    endpoint: AgCV6YtzikWYMRwHZM1tT8epq6XiSVqhbxMhS7htVjVUfdR4V7/KIcNExxSBul+TU+RCq3S7fppL7CxKbI/IsEdirRMj2PvESXEmzLY/xId7+wJFEdxeXxn59cbV2MfCyysKIwzsVXbmzO66yGaeMKU2oyqJVYu+00L9xS0RbtQRbi0UksMxtw0PhZVMee0aHfu9jynWI3lmPzC/EPRuwxQqBIjYM2v2JCDmYjP0fsBSZSJc+SnAaNmmIf7ypcFwy14qIF7C9tIg02lBx+Sg30oMPjn600AbyF6RmZ+mdclp/Qp+zXOUsbMym0Lu9BjbM7G/z0ExpR5sgE/6+05gJhPF7MuHkrYIMo9LHWWdgzNTnjikSTGEeuFzzRlThPMoUfVi/S7KhG9NtQRnNqEoVCGGW6tt5sygXVE/lWSnmmd//4jXvF9ZJHU+5LrjL7UfVIYNH4y7aYtXpoTgmUe85wWqeMdqr1EiSVueoSwCggGITYoXoKYfERfij8ln/xnX1Kt2xxuyLtpDOczBYZK6LLwTpaXVVe+lO/xBgevWaoEz6si2hyj+glluMQWxPsOnPwUfRi5qqlQxgMZK/9aALYhPLquc16iMG+4FdtoOUfQ0XlLvurHsbCt9lUy3YrsmpDhv42FfamVlcK+gMz7mqYhNhd3fVjxCiUuwvzd01QHcx8ikv48YOnDuRiweRszbPNvlaK2KH6fo7b4l9mmaa4W+MaxZCxG3KSMOafwRmyusEUVNWw==
+    secret-key: AgBItdfzRgcgJvK5HALDnkbGH1DveAgcmS+Q4f16VwX1KY8TzGWpjoaOdsLCwzTWOFME5xjrRyOwwAcdtP65ccWlEZvvoS4kpqv49G1ebblD1wmApnpa8vD63Rd028vUYFF3CnwntX2USxF6t4rD68sZqqzMDyuG9Av93HAKVMNdx9V4AJ45nvfF2WR7ohk37NhQifOdKuelgRV6ahl3/fm/XPRGzf+u7OER8jll9n/ywE0mSkoKBXocbkQ2Xrn+utzplh9BoR9KSPTdnmI8ZXUkDTR9p+7PN4r/HheEdTUG0aaefdEDCbdDlv1EW84l5KzhY4v9g0DW8GQQdyHWi3sMm0hSGFg9NV/Vs1HM9FZZCyjS+mzR4CbuHgkY9mvj+Svn7oS+Fy5d24s1ekmPheakBffXKy2Q2CLMRGNymjUIBQsDwoGC8o4IF+yPyDcw/OvXlr0lruxfRM6IN0oxgRmLAQiqwRzsZb3RPgILEZI9VXKxzCBEuCO7fE2pC05K/mujhUutWmJ5tQ0NFzP1mhNK59cAIVMl7y7Llq5/GZ5ax+voP0c3VandXJeaSRr6O40vyzfCMgpgEgRs9t2WfxCmVYEN2NWpFv07JNICWzJX40PLT9n/euRXzSmJld6UfjzRteaTfmo+OxFczJ6Zl8XdC+6x6wO0Nk0hVYo27s2LH4o0MsM84x9Z3StCCGl77/NnyavpxfDkj3qJQ4BgPtCw26tHTt9DIWYfgeAVqvoRZVlg8YxM3N1k
+  template:
+    metadata:
+      creationTimestamp: null
+      name: monitoring-s3-backup
+      namespace: monitoring
+    type: Opaque