feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)

Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.

- apps/observability/patches/node-exporter-textfile-collector.yaml:
  strategic-merge patch on the kube-prometheus-stack node-exporter DS
  that adds `--collector.textfile.directory=/host/textfile_collector`
  and mounts `/var/lib/node_exporter/textfile_collector` read-only.
  Chart isn't tracked in ArgoCD, so we keep the patch under version
  control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
  swap `emptyDir` /metrics for a `hostPath` on the same directory
  (`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
  never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
  for all four freshness alerts so a stale `.prom` left on a node the
  job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
  the on-node directory, atomic write, cross-node staleness rationale.

Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/monitoring/README.md

@@ -5,12 +5,12 @@ Manifests recording the cluster-side monitoring backup CronJobs that were previo
 - `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
 - `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
 - `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
-- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)). Note: still uses `rclone sync` (plaintext at rest on Hetzner). Migration to `restic` or `rclone crypt` is filed as a follow-up.
+- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block.
-- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos, plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490)).
+- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the four restic repos (loki/grafana/k8s-resources/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)).
 
 ## Shared SealedSecret
 
-All three restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys:
+All four restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys:
 
 | Key              | Purpose                                                                                                          |
 |------------------|------------------------------------------------------------------------------------------------------------------|
@@ -18,7 +18,7 @@ All three restic CronJobs read Hetzner S3 credentials + the restic repository pa
 | `secret-key`     | Hetzner Object Storage secret access key                                                                         |
 | `endpoint`       | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`)                                                        |
 | `bucket`         | Bucket name (single bucket, per-prefix repos)                                                                    |
-| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. |
+| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all four repos (loki/grafana/k8s-resources/prometheus) — rotating rewrites the key file on every repo. |
 
 ## Restore
 
@@ -26,21 +26,24 @@ Restore procedure for all three restic repos: [`docs/monitoring/restic-restore.m
 
 ## Emitted metrics (textfile-collector format)
 
-Each restic CronJob writes to `/metrics/backup_<kind>.prom` inside the pod's emptyDir. Once node-exporter's textfile-collector path is wired up (follow-up), these become scrapeable and the alerts in `backup-restic-alerts.yaml` evaluate against live data.
+Each restic CronJob writes `/metrics/backup_<kind>.prom` (atomic — `.prom.tmp` + `mv`) into a `hostPath` volume mounted at the node-exporter textfile-collector directory (`/var/lib/node_exporter/textfile_collector`). The kube-prometheus-stack node-exporter DaemonSet has `--collector.textfile.directory=/host/textfile_collector` enabled ([DEV-494](/DEV/issues/DEV-494), applied via [`apps/observability/patches/node-exporter-textfile-collector.yaml`](../observability/patches/node-exporter-textfile-collector.yaml)) and surfaces those samples in Prometheus.
 
 | Metric                                     | Emitted by                                                                       |
-|--------------------------------------------|---------------------------------------------------------------------|
+|--------------------------------------------|----------------------------------------------------------------------------------|
-| `backup_<kind>_success` (0/1)              | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml`                |
+| `backup_<kind>_success` (0/1)              | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` |
 | `backup_<kind>_timestamp_seconds`          | same                                                                             |
 | `backup_<kind>_check_status` (exit code)   | same — from `restic check --read-data-subset=5%`                                 |
-| `restic_repo_size_bytes{repo="<kind>"}`    | same — from `restic stats --json --mode raw-data` (added in DEV-490)|
+| `backup_prometheus_backup_status`          | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) |
+| `restic_repo_size_bytes{repo="<kind>"}`    | same — from `restic stats --json --mode raw-data` (added in DEV-490)             |
+
+**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods, `k8s-resources` is unpinned), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.
 
 ## Alerts
 
-`backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`):
+`backup-restic-alerts.yaml` defines nine alerts (all `severity: warning`):
 
-- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` — `time() - backup_<kind>_timestamp_seconds > 28h`. Daily schedule + 4 h grace.
+- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
-- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` — `backup_<kind>_check_status != 0`.
+- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
 - `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions.
 
 All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally:
@@ -52,4 +55,4 @@ promtool check rules /tmp/backup-restic-rules.yaml
 promtool test rules  apps/monitoring/backup-restic-alerts.test.yaml
 ```
 
-The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` and `prometheus-backup` remain unpinned.
+The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` has no PVC dep and stays unpinned. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers.

apps/monitoring/backup-grafana-restic-cronjob.yaml

@@ -118,15 +118,16 @@ spec:
                   REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
                   echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
 
-                  # Textfile-collector metrics; identical wiring to the loki
+                  # Textfile-collector metrics; identical wiring to the
-                  # sibling. Scrapeable once node-exporter's textfile
+                  # loki sibling. See that file for the atomic-write
-                  # collector path is enabled — tracked in DEV-482.
+                  # rationale (DEV-494).
                   {
                     echo "backup_grafana_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
                     echo "backup_grafana_timestamp_seconds $(date +%s)"
                     echo "backup_grafana_check_status ${CHECK_STATUS}"
                     echo "restic_repo_size_bytes{repo=\"grafana\"} ${REPO_SIZE_BYTES}"
-                  } > /metrics/backup_grafana.prom
+                  } > /metrics/backup_grafana.prom.tmp
+                  mv /metrics/backup_grafana.prom.tmp /metrics/backup_grafana.prom
 
                   echo "=== backup-grafana-restic finished at $(date -u +%FT%TZ) ==="
                   exit ${CHECK_STATUS}
@@ -150,6 +151,10 @@ spec:
               persistentVolumeClaim:
                 claimName: grafana-storage
             - name: metrics
-              emptyDir: {}
+              hostPath:
+                # node-exporter's textfile-collector directory
+                # (DEV-494). See sibling loki cronjob for detail.
+                path: /var/lib/node_exporter/textfile_collector
+                type: DirectoryOrCreate
             - name: cache
               emptyDir: {}

apps/monitoring/backup-k8s-resources-cronjob.yaml

@@ -165,14 +165,15 @@ spec:
                   echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
 
                   # Textfile-collector metrics; identical wiring to the
-                  # loki/grafana siblings. Scrapeable once node-exporter's
+                  # loki/grafana siblings. See loki cronjob for the
-                  # textfile collector path is enabled — tracked in DEV-482.
+                  # atomic-write rationale (DEV-494).
                   {
                     echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
                     echo "backup_k8s_resources_timestamp_seconds $(date +%s)"
                     echo "backup_k8s_resources_check_status ${CHECK_STATUS}"
                     echo "restic_repo_size_bytes{repo=\"k8s-resources\"} ${REPO_SIZE_BYTES}"
-                  } > /metrics/backup_k8s_resources.prom
+                  } > /metrics/backup_k8s_resources.prom.tmp
+                  mv /metrics/backup_k8s_resources.prom.tmp /metrics/backup_k8s_resources.prom
 
                   echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ==="
                   exit ${CHECK_STATUS}
@@ -195,6 +196,10 @@ spec:
             - name: dump
               emptyDir: {}
             - name: metrics
-              emptyDir: {}
+              hostPath:
+                # node-exporter's textfile-collector directory
+                # (DEV-494). See sibling loki cronjob for detail.
+                path: /var/lib/node_exporter/textfile_collector
+                type: DirectoryOrCreate
             - name: cache
               emptyDir: {}

apps/monitoring/backup-loki-restic-cronjob.yaml

@@ -134,17 +134,19 @@ spec:
                   REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
                   echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
 
-                  # Textfile-collector metrics. Written into an emptyDir per
+                  # Textfile-collector metrics. Written to the shared
-                  # the current pattern used by the other backup CronJobs.
+                  # host directory that node-exporter's textfile collector
-                  # Once a node-exporter textfile collector path is wired
+                  # scrapes (DEV-494). Atomic write: build the file with a
-                  # up, these become scrapeable — see the follow-up notes
+                  # `.tmp` extension (ignored by node-exporter) and rename
-                  # in DEV-482.
+                  # into place, so a mid-write read never surfaces a
+                  # truncated sample.
                   {
                     echo "backup_loki_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
                     echo "backup_loki_timestamp_seconds $(date +%s)"
                     echo "backup_loki_check_status ${CHECK_STATUS}"
                     echo "restic_repo_size_bytes{repo=\"loki\"} ${REPO_SIZE_BYTES}"
-                  } > /metrics/backup_loki.prom
+                  } > /metrics/backup_loki.prom.tmp
+                  mv /metrics/backup_loki.prom.tmp /metrics/backup_loki.prom
 
                   echo "=== backup-loki-restic finished at $(date -u +%FT%TZ) ==="
                   exit ${CHECK_STATUS}
@@ -173,6 +175,13 @@ spec:
               persistentVolumeClaim:
                 claimName: loki-storage-encrypted
             - name: metrics
-              emptyDir: {}
+              hostPath:
+                # node-exporter's textfile-collector directory
+                # (DEV-494). `DirectoryOrCreate` lets kubelet create
+                # the dir on the current node if it does not yet
+                # exist — nodes were pre-created out-of-band, this
+                # is a safety net.
+                path: /var/lib/node_exporter/textfile_collector
+                type: DirectoryOrCreate
             - name: cache
               emptyDir: {}

apps/monitoring/backup-restic-alerts.test.yaml

@@ -37,6 +37,8 @@ tests:
         values: '-104400x60'
       - series: 'backup_k8s_resources_timestamp_seconds'
         values: '-104400x60'
+      - series: 'backup_prometheus_timestamp_seconds'
+        values: '-104400x60'
     alert_rule_test:
       - eval_time: 30m
         alertname: BackupLokiStale
@@ -83,6 +85,21 @@ tests:
                 `backup_k8s_resources_timestamp_seconds` sample in
                 more than 28 hours. Expected daily at 02:00 UTC.
                 Runbook: docs/monitoring/restic-restore.md
+      - eval_time: 30m
+        alertname: BackupPrometheusStale
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: prometheus
+            exp_annotations:
+              summary: "Prometheus restic backup is stale (>28h)"
+              description: |
+                `prometheus-backup` has not written a fresh
+                `backup_prometheus_timestamp_seconds` sample in more
+                than 28 hours. Expected daily at 03:30 UTC.
+                Runbook: docs/monitoring/restic-restore.md
 
   # ---------------------------------------------------------------------------
   # 2. Freshness alerts stay silent when timestamp is fresh (<28h).
@@ -97,6 +114,8 @@ tests:
         values: '-3600x60'
       - series: 'backup_k8s_resources_timestamp_seconds'
         values: '-3600x60'
+      - series: 'backup_prometheus_timestamp_seconds'
+        values: '-3600x60'
     alert_rule_test:
       - eval_time: 30m
         alertname: BackupLokiStale
@@ -107,6 +126,9 @@ tests:
       - eval_time: 30m
         alertname: BackupK8sResourcesStale
         exp_alerts: []
+      - eval_time: 30m
+        alertname: BackupPrometheusStale
+        exp_alerts: []
 
   # ---------------------------------------------------------------------------
   # 3. Check-status alerts fire when the metric is non-zero.
@@ -119,6 +141,8 @@ tests:
         values: '2x30'
       - series: 'backup_k8s_resources_check_status'
         values: '1x30'
+      - series: 'backup_prometheus_check_status'
+        values: '1x30'
     alert_rule_test:
       - eval_time: 15m
         alertname: BackupLokiCheckFailed
@@ -163,6 +187,21 @@ tests:
                 exit code 1 on the k8s-resources
                 repository.
                 Runbook: docs/monitoring/restic-restore.md
+      - eval_time: 15m
+        alertname: BackupPrometheusCheckFailed
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: prometheus
+            exp_annotations:
+              summary: "restic check failed on prometheus repo"
+              description: |
+                `restic check --read-data-subset=5%` returned
+                exit code 1 on the prometheus
+                repository.
+                Runbook: docs/monitoring/restic-restore.md
 
   # ---------------------------------------------------------------------------
   # 4. Check-status alerts stay silent on 0.
@@ -175,6 +214,8 @@ tests:
         values: '0x30'
       - series: 'backup_k8s_resources_check_status'
         values: '0x30'
+      - series: 'backup_prometheus_check_status'
+        values: '0x30'
     alert_rule_test:
       - eval_time: 15m
         alertname: BackupLokiCheckFailed
@@ -185,6 +226,9 @@ tests:
       - eval_time: 15m
         alertname: BackupK8sResourcesCheckFailed
         exp_alerts: []
+      - eval_time: 15m
+        alertname: BackupPrometheusCheckFailed
+        exp_alerts: []
 
   # ---------------------------------------------------------------------------
   # 5. Repo-size alert fires when >20 GiB (21474836480 bytes).

apps/monitoring/backup-restic-alerts.yaml

@@ -14,11 +14,19 @@
 # by the same CronJob step (see backup-*-restic-cronjob.yaml,
 # `restic stats --json --mode raw-data`).
 #
-# The metrics are written to an emptyDir at /metrics per job.
+# The metrics are written to the node's textfile-collector directory
-# They become scrapeable once node-exporter's textfile collector
+# (`/var/lib/node_exporter/textfile_collector`) and scraped by the
-# path is wired up — tracked as a follow-up. `promtool test rules`
+# kube-prometheus-stack node-exporter DaemonSet (DEV-494).
-# against apps/monitoring/backup-restic-alerts.test.yaml verifies
+#
-# the rules with synthetic samples in the meantime.
+# Because a CronJob may run on a different worker across days (loki
+# backup follows the loki pod; k8s-resources is unpinned), stale
+# `.prom` files can linger on nodes the job has since left. That
+# would leave a per-node series with an old timestamp/check-status
+# indefinitely. The alerts below aggregate across instances so a
+# single fresh sample from the node where the job currently runs is
+# enough to keep the freshness alert quiet, and check/size alerts
+# fire when *any* node reports a bad value (which is the correct
+# behaviour — a recent failure is still a signal).
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
@@ -34,7 +42,10 @@ spec:
       interval: 60s
       rules:
         - alert: BackupLokiStale
-          expr: time() - backup_loki_timestamp_seconds > 28 * 3600
+          # `max()` collapses per-node samples so a stale `.prom` file
+          # left behind on a node the loki backup no longer runs on
+          # does not fire this alert; the freshest sample wins.
+          expr: time() - max(backup_loki_timestamp_seconds) > 28 * 3600
           for: 15m
           labels:
             severity: warning
@@ -49,7 +60,7 @@ spec:
               28 hours. Expected daily at 03:00 UTC.
               Runbook: docs/monitoring/restic-restore.md
         - alert: BackupGrafanaStale
-          expr: time() - backup_grafana_timestamp_seconds > 28 * 3600
+          expr: time() - max(backup_grafana_timestamp_seconds) > 28 * 3600
           for: 15m
           labels:
             severity: warning
@@ -64,7 +75,7 @@ spec:
               than 28 hours. Expected daily at 03:15 UTC.
               Runbook: docs/monitoring/restic-restore.md
         - alert: BackupK8sResourcesStale
-          expr: time() - backup_k8s_resources_timestamp_seconds > 28 * 3600
+          expr: time() - max(backup_k8s_resources_timestamp_seconds) > 28 * 3600
           for: 15m
           labels:
             severity: warning
@@ -78,6 +89,21 @@ spec:
               `backup_k8s_resources_timestamp_seconds` sample in
               more than 28 hours. Expected daily at 02:00 UTC.
               Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupPrometheusStale
+          expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600
+          for: 15m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: prometheus
+          annotations:
+            summary: "Prometheus restic backup is stale (>28h)"
+            description: |
+              `prometheus-backup` has not written a fresh
+              `backup_prometheus_timestamp_seconds` sample in more
+              than 28 hours. Expected daily at 03:30 UTC.
+              Runbook: docs/monitoring/restic-restore.md
     - name: backup-restic.integrity
       interval: 60s
       rules:
@@ -124,6 +150,21 @@ spec:
               exit code {{ $value }} on the k8s-resources
               repository.
               Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupPrometheusCheckFailed
+          expr: backup_prometheus_check_status != 0
+          for: 5m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: prometheus
+          annotations:
+            summary: "restic check failed on prometheus repo"
+            description: |
+              `restic check --read-data-subset=5%` returned
+              exit code {{ $value }} on the prometheus
+              repository.
+              Runbook: docs/monitoring/restic-restore.md
     - name: backup-restic.size
       interval: 60s
       rules:

apps/monitoring/prometheus-backup-cronjob.yaml

@@ -1,16 +1,43 @@
 ---
-# Prometheus data backup (DEV-465).
+# Prometheus data backup via restic to Hetzner Object Storage
+# (DEV-492, DEV-482 Option 4). Replaces the DEV-465 rclone-sync job so
+# every monitoring backup ships client-side-encrypted; Hetzner Object
+# Storage has no SSE-S3/SSE-KMS, so the previous plaintext-at-rest
+# object layout was the only remaining gap.
 #
-# Prometheus data lives on the RWO PVC `prometheus-data-encrypted` in
+# Streams the RWO PVC `prometheus-data-encrypted` (mounted read-only)
-# namespace `monitoring`. That PVC is mounted by the Prometheus pod, and
+# into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, a client-side
-# the Hetzner CSI volume can only be attached to one node at a time.
+# encrypted restic repository (tag=`prometheus`, host=`k3s`).
 #
-# This CronJob co-schedules with the Prometheus pod via podAffinity, so it
+# Node scheduling matches the previous job: podAffinity co-schedules
-# lands on whichever node currently holds `prometheus-data-encrypted`. The
+# with the Prometheus pod (app=prometheus, topology
-# PVC is mounted read-only alongside the running Prometheus pod (RWO permits
+# kubernetes.io/hostname). Hetzner CSI RWO permits additional read-only
-# additional read-only mounts on the same node) and streamed to Hetzner S3
+# mounts on the node that holds the PVC's VolumeAttachment, so this
-# via rclone under `basicstack-backup/prometheus/prometheus-<DATE>/`. Old
+# survives Prometheus being rescheduled to a different worker.
-# snapshots are pruned after 7 days.
+#
+# Prometheus TSDB compaction race
+# --------------------------------
+# Prometheus rewrites the on-disk store roughly every 2 h: it creates a
+# new block dir, then deletes the source dirs. restic walks the source
+# tree once and may catch a file that disappeared mid-walk; restic
+# 0.17.3 exits 3 ("at least one source file could not be read") in
+# that case, and the snapshot excludes only the missing file. The
+# next daily run picks up the successor block, so the race is not a
+# data-loss risk — but we must not treat exit 3 as a hard failure, or
+# the daily job will alert-flap.
+#
+# Mitigation:
+#   - exclude `wal/*` (WAL is replayed from a fresh instance on
+#     restart; we accept losing the last ~15 s of ingested samples
+#     rather than snapshotting a moving segment)
+#   - exclude `chunks_head/*` (in-memory head block; ephemeral, would
+#     be rebuilt from WAL which we do not keep)
+#   - exclude Prometheus lock/scratch files (`lock`, `queries.active`,
+#     `*.tmp`, `lost+found/*`)
+#   - treat restic exit code 3 as a soft warning (log, continue);
+#     any other non-zero exit is still fatal
+#   - restore drill re-runs `promtool tsdb analyze` against every
+#     block so a corrupted snapshot is caught end-to-end
 apiVersion: batch/v1
 kind: CronJob
 metadata:
@@ -19,8 +46,9 @@ metadata:
   labels:
     app: backup
     type: prometheus
+    backend: restic
 spec:
-  schedule: "30 3 * * *"  # daily 03:30, offset from the loki/grafana restic runs
+  schedule: "30 3 * * *"  # daily 03:30, offset from loki/grafana/k8s-resources
   concurrencyPolicy: Forbid
   successfulJobsHistoryLimit: 3
   failedJobsHistoryLimit: 3
@@ -29,19 +57,18 @@ spec:
       labels:
         app: backup
         type: prometheus
+        backend: restic
     spec:
       backoffLimit: 2
-      activeDeadlineSeconds: 3600  # 60 min hard cap; Prometheus data is ~9GB
+      activeDeadlineSeconds: 3600
       template:
         metadata:
           labels:
             app: backup
             type: prometheus
+            backend: restic
         spec:
           restartPolicy: OnFailure
-          # Co-schedule with the Prometheus pod so the RWO PVC attaches on
-          # the same node. This survives Prometheus being rescheduled to a
-          # different worker (the backup follows).
           affinity:
             podAffinity:
               requiredDuringSchedulingIgnoredDuringExecution:
@@ -53,134 +80,24 @@ spec:
                           - prometheus
                   topologyKey: kubernetes.io/hostname
           containers:
-            - name: prometheus-backup
+            - name: restic
-              image: rclone/rclone:1.68
+              image: restic/restic:0.17.3
-              command:
-                - /bin/sh
-                - -c
-                - |
-                  set -eu
-                  DATE=$(date +%Y%m%d-%H%M%S)
-                  echo "=== Prometheus backup started at $(date) (target prefix: prometheus-${DATE}) ==="
-
-                  mkdir -p /root/.config/rclone
-                  cat > /root/.config/rclone/rclone.conf <<EOC
-                  [hetzner-s3]
-                  type = s3
-                  provider = Other
-                  access_key_id = ${S3_ACCESS_KEY}
-                  secret_access_key = ${S3_SECRET_KEY}
-                  endpoint = ${S3_ENDPOINT}
-                  acl = private
-                  EOC
-
-                  SOURCE_BYTES=$(du -sb /source 2>/dev/null | cut -f1 || echo 0)
-                  SOURCE_HUMAN=$(du -sh /source 2>/dev/null | cut -f1 || echo unknown)
-                  echo "Source /source size: ${SOURCE_HUMAN} (${SOURCE_BYTES} bytes)"
-
-                  # Prometheus TSDB is largely immutable chunk files plus an
-                  # append-only WAL. rclone sync is safe with the database
-                  # live: on restore the WAL is replayed. HOWEVER Prometheus
-                  # compacts blocks every ~2h and deletes their source dirs,
-                  # which races with the copy and produces "no such file or
-                  # directory" errors mid-run. Those are expected and do
-                  # not indicate data loss — the compacted successor blocks
-                  # are picked up on the same or the next daily run. We
-                  # therefore do not fail the job on rclone's non-zero exit
-                  # from those transient errors; instead we validate the
-                  # backup by comparing destination size to source (must be
-                  # >= 80% of source bytes and > 100 MiB).
-                  DEST="hetzner-s3:${S3_BUCKET}/prometheus/prometheus-${DATE}/"
-                  echo "Streaming Prometheus data to ${DEST} ..."
-                  RCLONE_EXIT=0
-                  rclone sync /source "${DEST}" \
-                    --transfers 4 \
-                    --checkers 4 \
-                    --stats 30s \
-                    --stats-log-level NOTICE \
-                    --s3-chunk-size 32M \
-                    --s3-upload-concurrency 4 \
-                    --retries 3 \
-                    --retries-sleep 30s || RCLONE_EXIT=$?
-                  echo "rclone sync exit code: ${RCLONE_EXIT}"
-
-                  echo "Measuring destination size..."
-                  DEST_BYTES=$(rclone size "${DEST}" --json 2>/dev/null | \
-                    sed -n 's/.*"bytes":\s*\([0-9]\+\).*/\1/p' | head -1)
-                  DEST_BYTES=${DEST_BYTES:-0}
-                  DEST_HUMAN=$(rclone size "${DEST}" 2>/dev/null | \
-                    grep -oE 'Total size:.*' || echo "Total size: unknown")
-                  echo "Destination bytes: ${DEST_BYTES}"
-                  echo "Destination summary: ${DEST_HUMAN}"
-
-                  MIN_ACCEPTABLE=$(( SOURCE_BYTES * 80 / 100 ))
-                  FLOOR=104857600 # 100 MiB absolute floor
-                  echo "Acceptance threshold: dest >= ${MIN_ACCEPTABLE} bytes and > ${FLOOR} bytes"
-                  if [ "${DEST_BYTES}" -lt "${FLOOR}" ]; then
-                    echo "ERROR: destination is below hard floor (100 MiB) — backup failed."
-                    exit 2
-                  fi
-                  if [ "${DEST_BYTES}" -lt "${MIN_ACCEPTABLE}" ]; then
-                    echo "ERROR: destination is < 80% of source (${DEST_BYTES} < ${MIN_ACCEPTABLE}) — backup incomplete."
-                    exit 3
-                  fi
-                  echo "OK: destination size acceptable."
-
-                  # Prune by prefix name, NOT by file mtime. rclone preserves
-                  # each source file's mtime on upload, and Prometheus TSDB
-                  # chunk files retain very old mtimes (weeks-old blocks),
-                  # so `rclone delete --min-age 7d` would eat the just-
-                  # uploaded backup. Every top-level prefix is named
-                  # `prometheus-YYYYMMDD-HHMMSS`, so we compare the encoded
-                  # date to a 7-day threshold and purge whole prefixes.
-                  echo "Pruning prometheus backup prefixes older than 7 days..."
-                  # BusyBox date lacks GNU's `-d "7 days ago"` and BSD's `-v -7d`,
-                  # so compute the cutoff via @epoch which BusyBox does support.
-                  CUTOFF_EPOCH=$(( $(date +%s) - 7 * 86400 ))
-                  CUTOFF=$(date -d "@${CUTOFF_EPOCH}" +%Y%m%d)
-                  echo "Cutoff (delete prefixes with date < ${CUTOFF}):"
-                  KEEP=0
-                  PRUNE=0
-                  for PREFIX in $(rclone lsf --dirs-only "hetzner-s3:${S3_BUCKET}/prometheus/" 2>/dev/null | sed 's:/$::'); do
-                    case "${PREFIX}" in
-                      prometheus-*)
-                        PDATE=$(echo "${PREFIX}" | sed -n 's/^prometheus-\([0-9]\{8\}\)-.*/\1/p')
-                        if [ -z "${PDATE}" ]; then
-                          echo "  SKIP ${PREFIX} (unparseable name)"
-                          continue
-                        fi
-                        if [ "${PDATE}" -lt "${CUTOFF}" ]; then
-                          echo "  PURGE ${PREFIX} (date ${PDATE} < ${CUTOFF})"
-                          rclone purge "hetzner-s3:${S3_BUCKET}/prometheus/${PREFIX}" \
-                            || echo "  WARN purge failed for ${PREFIX} (continuing)"
-                          PRUNE=$((PRUNE + 1))
-                        else
-                          echo "  KEEP  ${PREFIX} (date ${PDATE})"
-                          KEEP=$((KEEP + 1))
-                        fi
-                        ;;
-                      *)
-                        echo "  SKIP ${PREFIX} (not a prometheus- prefix)"
-                        ;;
-                    esac
-                  done
-                  echo "Prune summary: kept ${KEEP}, purged ${PRUNE}."
-
-                  echo "Post-run inventory (prometheus/ prefixes):"
-                  rclone lsd "hetzner-s3:${S3_BUCKET}/prometheus/" || true
-
-                  echo "=== Prometheus backup completed at $(date) ==="
               env:
-                - name: S3_ACCESS_KEY
+                - name: AWS_ACCESS_KEY_ID
                   valueFrom:
                     secretKeyRef:
                       name: monitoring-s3-backup
                       key: access-key
-                - name: S3_SECRET_KEY
+                - name: AWS_SECRET_ACCESS_KEY
                   valueFrom:
                     secretKeyRef:
                       name: monitoring-s3-backup
                       key: secret-key
+                - name: RESTIC_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: restic-password
                 - name: S3_ENDPOINT
                   valueFrom:
                     secretKeyRef:
@@ -191,18 +108,121 @@ spec:
                     secretKeyRef:
                       name: monitoring-s3-backup
                       key: bucket
+                - name: RESTIC_REPOSITORY
+                  value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/prometheus"
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  echo "=== backup-prometheus-restic started at $(date -u +%FT%TZ) ==="
+                  echo "Repository: ${RESTIC_REPOSITORY}"
+
+                  # First-run tolerance: init if the repo isn't there yet.
+                  if restic snapshots >/dev/null 2>&1; then
+                    echo "Repo exists, skipping init."
+                  else
+                    echo "Repo missing, initialising..."
+                    restic init
+                  fi
+
+                  # See "Prometheus TSDB compaction race" in the file
+                  # header. WAL + chunks_head are excluded on purpose;
+                  # exit code 3 (source file vanished mid-walk during a
+                  # compaction) is accepted and reported, any other
+                  # non-zero exit is fatal.
+                  echo "--- restic backup /source (exclude wal/chunks_head + lock files) ---"
+                  BACKUP_STATUS=0
+                  restic backup /source \
+                    --tag prometheus \
+                    --host k3s \
+                    --exclude 'wal/*' \
+                    --exclude 'chunks_head/*' \
+                    --exclude 'lock' \
+                    --exclude 'queries.active' \
+                    --exclude 'lost+found/*' \
+                    --exclude '*.tmp' || BACKUP_STATUS=$?
+                  echo "restic backup exit: ${BACKUP_STATUS}"
+                  if [ "${BACKUP_STATUS}" -eq 0 ]; then
+                    echo "backup: all files captured cleanly"
+                  elif [ "${BACKUP_STATUS}" -eq 3 ]; then
+                    echo "backup: exit 3 (source files vanished mid-walk) — expected under Prometheus compaction, continuing"
+                  else
+                    echo "backup: FATAL exit ${BACKUP_STATUS} (not compaction-race)"
+                    exit ${BACKUP_STATUS}
+                  fi
+
+                  echo "--- restic forget/prune ---"
+                  restic forget --tag prometheus \
+                    --keep-daily 7 \
+                    --keep-weekly 4 \
+                    --keep-monthly 6 \
+                    --prune
+
+                  echo "--- restic check --read-data-subset=5% ---"
+                  CHECK_STATUS=0
+                  restic check --read-data-subset=5% || CHECK_STATUS=$?
+                  echo "restic check exit: ${CHECK_STATUS}"
+
+                  echo "--- restic stats (repo size) ---"
+                  REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
+                    | grep -oE '"total_size":[0-9]+' \
+                    | head -1 \
+                    | cut -d: -f2)
+                  REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
+                  echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
+
+                  # Textfile-collector metrics. Same hostPath pattern as
+                  # the loki/grafana/k8s-resources siblings (DEV-494) —
+                  # written atomically via `.tmp` + rename so a mid-write
+                  # read never surfaces a truncated sample.
+                  # backup_prometheus_success rolls in both stages: the
+                  # backup step (accepting exit 3) and restic check.
+                  BACKUP_OK=0
+                  if [ "${BACKUP_STATUS}" -eq 0 ] || [ "${BACKUP_STATUS}" -eq 3 ]; then
+                    BACKUP_OK=1
+                  fi
+                  SUCCESS=0
+                  if [ "${BACKUP_OK}" -eq 1 ] && [ "${CHECK_STATUS}" -eq 0 ]; then
+                    SUCCESS=1
+                  fi
+                  {
+                    echo "backup_prometheus_success ${SUCCESS}"
+                    echo "backup_prometheus_timestamp_seconds $(date +%s)"
+                    echo "backup_prometheus_check_status ${CHECK_STATUS}"
+                    echo "backup_prometheus_backup_status ${BACKUP_STATUS}"
+                    echo "restic_repo_size_bytes{repo=\"prometheus\"} ${REPO_SIZE_BYTES}"
+                  } > /metrics/backup_prometheus.prom.tmp
+                  mv /metrics/backup_prometheus.prom.tmp /metrics/backup_prometheus.prom
+
+                  echo "=== backup-prometheus-restic finished at $(date -u +%FT%TZ) ==="
+                  exit ${CHECK_STATUS}
               volumeMounts:
                 - name: prometheus-data
                   mountPath: /source
                   readOnly: true
+                - name: metrics
+                  mountPath: /metrics
+                - name: cache
+                  mountPath: /root/.cache/restic
               resources:
+                # Prometheus TSDB is ~8-10 GiB; give restic room to
+                # burst during pack/check but keep steady-state small.
                 requests:
                   cpu: 100m
-                  memory: 128Mi
+                  memory: 256Mi
                 limits:
-                  cpu: 1000m
+                  cpu: 1500m
-                  memory: 512Mi
+                  memory: 1Gi
           volumes:
             - name: prometheus-data
               persistentVolumeClaim:
                 claimName: prometheus-data-encrypted
+            - name: metrics
+              hostPath:
+                # node-exporter's textfile-collector directory
+                # (DEV-494). See sibling loki cronjob for detail.
+                path: /var/lib/node_exporter/textfile_collector
+                type: DirectoryOrCreate
+            - name: cache
+              emptyDir: {}

apps/observability/patches/README.md

@@ -0,0 +1,38 @@
+# observability patches
+
+Strategic-merge patches applied on top of Helm-managed observability
+resources. Each file is idempotent (re-applying is a no-op) and is
+reasserted by hand rather than by a controller, so re-run after any
+`helm upgrade` of the affected release.
+
+## `node-exporter-textfile-collector.yaml` (DEV-494)
+
+Enables the node-exporter textfile collector on the
+`kube-prometheus-stack-prometheus-node-exporter` DaemonSet by:
+
+1. adding `--collector.textfile.directory=/host/textfile_collector`
+   to the container args, and
+2. mounting the host directory `/var/lib/node_exporter/textfile_collector`
+   read-only at `/host/textfile_collector` (`hostPath` type
+   `DirectoryOrCreate`, kubelet creates it on nodes where the
+   directory does not exist yet).
+
+The four monitoring backup CronJobs in `apps/monitoring/` write their
+textfile-collector `.prom` files into that same host directory, so the
+metrics surface in Prometheus via node-exporter's normal scrape.
+
+Apply / re-apply:
+
+```bash
+kubectl -n observability patch daemonset \
+  kube-prometheus-stack-prometheus-node-exporter \
+  --type=strategic \
+  --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
+kubectl -n observability rollout status daemonset \
+  kube-prometheus-stack-prometheus-node-exporter
+```
+
+The kube-prometheus-stack chart is not currently tracked in ArgoCD;
+if it moves under GitOps, fold these values into the chart values as
+`prometheus-node-exporter.extraArgs` + `.extraHostVolumeMounts`
+instead of maintaining this patch.

apps/observability/patches/node-exporter-textfile-collector.yaml

@@ -0,0 +1,70 @@
+---
+# Strategic-merge patch enabling node-exporter's textfile collector on
+# the kube-prometheus-stack node-exporter DaemonSet (DEV-494).
+#
+# The chart is Helm-managed (release `kube-prometheus-stack` in
+# namespace `observability`, chart kube-prometheus-stack-86.2.2 /
+# prometheus-node-exporter-4.55.0) and is NOT currently tracked in
+# ArgoCD, so a direct DaemonSet patch is the pragmatic wiring path.
+# The change:
+#
+#   1. adds the `--collector.textfile.directory=/host/textfile_collector`
+#      arg to the node-exporter container, and
+#   2. mounts the host directory `/var/lib/node_exporter/textfile_collector`
+#      read-only at `/host/textfile_collector` (type DirectoryOrCreate so
+#      kubelet creates the dir on nodes where it does not yet exist).
+#
+# Apply with:
+#   kubectl -n observability patch daemonset \
+#     kube-prometheus-stack-prometheus-node-exporter \
+#     --type=strategic \
+#     --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
+#
+# If the Helm release is ever `helm upgrade`d without folding these
+# values into the chart values, this patch will be reverted — re-apply
+# it after the upgrade (or move it into a repo-owned values file).
+spec:
+  template:
+    spec:
+      containers:
+        - name: node-exporter
+          args:
+            - --path.procfs=/host/proc
+            - --path.sysfs=/host/sys
+            - --path.rootfs=/host/root
+            - --path.udev.data=/host/root/run/udev/data
+            - --web.listen-address=[$(HOST_IP)]:9100
+            - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
+            - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$
+            - --collector.textfile.directory=/host/textfile_collector
+          volumeMounts:
+            - mountPath: /host/proc
+              name: proc
+              readOnly: true
+            - mountPath: /host/sys
+              name: sys
+              readOnly: true
+            - mountPath: /host/root
+              mountPropagation: HostToContainer
+              name: root
+              readOnly: true
+            - mountPath: /host/textfile_collector
+              name: textfile-collector
+              readOnly: true
+      volumes:
+        - hostPath:
+            path: /proc
+            type: ""
+          name: proc
+        - hostPath:
+            path: /sys
+            type: ""
+          name: sys
+        - hostPath:
+            path: /
+            type: ""
+          name: root
+        - hostPath:
+            path: /var/lib/node_exporter/textfile_collector
+            type: DirectoryOrCreate
+          name: textfile-collector