From 7368283d76466f481ef35f280ffc831f96276e11 Mon Sep 17 00:00:00 2001 From: "CTO (Paperclip Agent)" Date: Sat, 12 Sep 2026 18:24:00 +0000 Subject: [PATCH] retire(monitoring): remove backup-k8s-resources CronJob + restic alerts (DEV-576) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Velero cluster-all now covers cluster-manifest backup with two successive Completed runs verified and one restore drill (openwebui) green (DEV-575). - Delete apps/monitoring/backup-k8s-resources-cronjob.yaml - Drop BackupK8sResourcesStale + BackupK8sResourcesCheckFailed from backup-restic-alerts.yaml and matching series/cases from the promtool test file (all 9 remaining rules pass `promtool test rules`) - Update apps/monitoring/README.md + prometheus-backup-cronjob comments - Runbook lives in BookStack: EN https://books.basicstack.de/books/it-operations-manual/page/velero-backup-restore-runbook DE https://books.basicstack.de/books/it-betriebshandbuch/page/velero-backup-und-restore-runbook Rollback: revert this commit — Argo restores the CronJob + alerts and restic k8s-resources runs resume at the next 02:00 UTC schedule. Co-Authored-By: Paperclip --- apps/monitoring/README.md | 17 +- .../backup-k8s-resources-cronjob.yaml | 205 ------------------ .../monitoring/backup-restic-alerts.test.yaml | 44 ---- apps/monitoring/backup-restic-alerts.yaml | 32 +-- .../monitoring/prometheus-backup-cronjob.yaml | 4 +- 5 files changed, 11 insertions(+), 291 deletions(-) delete mode 100644 apps/monitoring/backup-k8s-resources-cronjob.yaml diff --git a/apps/monitoring/README.md b/apps/monitoring/README.md index d4d16e2..15f5a4e 100644 --- a/apps/monitoring/README.md +++ b/apps/monitoring/README.md @@ -2,11 +2,10 @@ Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context. -- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.19.1` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)). - `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)). - `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)). - `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block. -- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the four restic repos (loki/grafana/k8s-resources/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)). +- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos (loki/grafana/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)). The `k8s-resources` restic repo was superseded by Velero cluster-reconstruction backups and retired in [DEV-576](/DEV/issues/DEV-576). ## Shared SealedSecret @@ -18,7 +17,7 @@ All four restic CronJobs read Hetzner S3 credentials + the restic repository pas | `secret-key` | Hetzner Object Storage secret access key | | `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) | | `bucket` | Bucket name (single bucket, per-prefix repos) | -| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all four repos (loki/grafana/k8s-resources/prometheus) — rotating rewrites the key file on every repo. | +| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all three remaining repos (loki/grafana/prometheus) — rotating rewrites the key file on every repo. | ## Restore @@ -30,20 +29,20 @@ Each restic CronJob writes `/metrics/backup_.prom` (atomic — `.prom.tmp` | Metric | Emitted by | |--------------------------------------------|----------------------------------------------------------------------------------| -| `backup__success` (0/1) | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` | +| `backup__success` (0/1) | `backup-{loki,grafana}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` | | `backup__timestamp_seconds` | same | | `backup__check_status` (exit code) | same — from `restic check --read-data-subset=5%` | | `backup_prometheus_backup_status` | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) | | `restic_repo_size_bytes{repo=""}` | same — from `restic stats --json --mode raw-data` (added in DEV-490) | -**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods, `k8s-resources` is unpinned), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node. +**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node. ## Alerts -`backup-restic-alerts.yaml` defines nine alerts (all `severity: warning`): +`backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`): -- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` / `BackupPrometheusStale` — `time() - max(backup__timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire. -- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` / `BackupPrometheusCheckFailed` — `backup__check_status != 0`. +- `BackupLokiStale` / `BackupGrafanaStale` / `BackupPrometheusStale` — `time() - max(backup__timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire. +- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupPrometheusCheckFailed` — `backup__check_status != 0`. - `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions. All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally: @@ -55,4 +54,4 @@ promtool check rules /tmp/backup-restic-rules.yaml promtool test rules apps/monitoring/backup-restic-alerts.test.yaml ``` -The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` has no PVC dep and stays unpinned. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers. +The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers. The `backup-k8s-resources` CronJob and its restic repo were retired in [DEV-576](/DEV/issues/DEV-576) after Velero `cluster-all` took over cluster-manifest coverage. diff --git a/apps/monitoring/backup-k8s-resources-cronjob.yaml b/apps/monitoring/backup-k8s-resources-cronjob.yaml deleted file mode 100644 index 36f064f..0000000 --- a/apps/monitoring/backup-k8s-resources-cronjob.yaml +++ /dev/null @@ -1,205 +0,0 @@ ---- -# Kubernetes-resource backup via restic to Hetzner Object Storage -# (DEV-487, DEV-482 Option 4). Step 4 of the Option 4 rollout. -# -# Streams a concatenated YAML dump of cluster-scoped and per-namespace -# resources through `restic backup --stdin` into -# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources`. No PVC mount -# (drops the local-path `backup-storage` dependency), no node pin. -# -# Two-container pattern: -# 1. `kubectl-dump` init container (`alpine/k8s:1.29.4`) writes -# /dump/cluster.yaml into an emptyDir. Uses `serviceAccountName: -# backup-sa` (unchanged from the legacy job). -# 2. `restic` main container (`restic/restic:0.19.1`, matches the -# loki/grafana siblings) reads that file on stdin and streams it -# into the restic repo with `--stdin-filename cluster.yaml`. -apiVersion: batch/v1 -kind: CronJob -metadata: - name: backup-k8s-resources - namespace: monitoring - labels: - app: backup - type: k8s-resources - backend: restic -spec: - schedule: "0 2 * * *" - concurrencyPolicy: Forbid - successfulJobsHistoryLimit: 3 - failedJobsHistoryLimit: 3 - jobTemplate: - metadata: - labels: - app: backup - type: k8s-resources - backend: restic - spec: - backoffLimit: 2 - activeDeadlineSeconds: 3600 - template: - metadata: - labels: - app: backup - type: k8s-resources - backend: restic - spec: - restartPolicy: OnFailure - serviceAccountName: backup-sa - initContainers: - - name: kubectl-dump - image: alpine/k8s:1.29.4 - command: - - /bin/sh - - -c - - | - set -eu - echo "=== kubectl-dump started at $(date -u +%FT%TZ) ===" - DUMP=/dump/cluster.yaml - : > "${DUMP}" - - echo "--- namespaces ---" - kubectl get namespaces -o yaml >> "${DUMP}" - echo "---" >> "${DUMP}" - - echo "--- cluster-scoped resources ---" - kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \ - -o yaml >> "${DUMP}" - echo "---" >> "${DUMP}" - - echo "--- namespaced resources ---" - for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do - echo " ns=${ns}" - kubectl get \ - configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \ - -n "${ns}" -o yaml >> "${DUMP}" 2>/dev/null || true - echo "---" >> "${DUMP}" - done - - echo "dump size: $(wc -c < ${DUMP}) bytes" - echo "=== kubectl-dump finished at $(date -u +%FT%TZ) ===" - resources: - requests: - cpu: 50m - memory: 128Mi - limits: - cpu: 500m - memory: 512Mi - volumeMounts: - - name: dump - mountPath: /dump - containers: - - name: restic - image: harbor.basicstack.de/library/restic:0.19.1 - # Mirrored from docker.io/restic/restic:0.19.1 (DEV-493) — - # deterministic ingress via Harbor. Retag procedure in - # docs/monitoring/restic-restore.md § "Tag-bump procedure". - env: - - name: AWS_ACCESS_KEY_ID - valueFrom: - secretKeyRef: - name: monitoring-s3-backup - key: access-key - - name: AWS_SECRET_ACCESS_KEY - valueFrom: - secretKeyRef: - name: monitoring-s3-backup - key: secret-key - - name: RESTIC_PASSWORD - valueFrom: - secretKeyRef: - name: monitoring-s3-backup - key: restic-password - - name: S3_ENDPOINT - valueFrom: - secretKeyRef: - name: monitoring-s3-backup - key: endpoint - - name: S3_BUCKET - valueFrom: - secretKeyRef: - name: monitoring-s3-backup - key: bucket - - name: RESTIC_REPOSITORY - value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/k8s-resources" - command: - - /bin/sh - - -c - - | - set -eu - echo "=== backup-k8s-resources-restic started at $(date -u +%FT%TZ) ===" - echo "Repository: ${RESTIC_REPOSITORY}" - - # First-run tolerance: init if the repo isn't there yet. - if restic snapshots >/dev/null 2>&1; then - echo "Repo exists, skipping init." - else - echo "Repo missing, initialising..." - restic init - fi - - echo "--- restic backup --stdin cluster.yaml ---" - restic backup --stdin \ - --stdin-filename cluster.yaml \ - --tag k8s-resources \ - --host k3s < /dump/cluster.yaml - - echo "--- restic forget/prune ---" - restic forget --tag k8s-resources \ - --keep-daily 7 \ - --keep-weekly 4 \ - --keep-monthly 6 \ - --prune - - echo "--- restic check --read-data-subset=5% ---" - CHECK_STATUS=0 - restic check --read-data-subset=5% || CHECK_STATUS=$? - echo "restic check exit: ${CHECK_STATUS}" - - echo "--- restic stats (repo size) ---" - REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \ - | grep -oE '"total_size":[0-9]+' \ - | head -1 \ - | cut -d: -f2) - REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0} - echo "restic repo size: ${REPO_SIZE_BYTES} bytes" - - # Textfile-collector metrics; identical wiring to the - # loki/grafana siblings. See loki cronjob for the - # atomic-write rationale (DEV-494). - { - echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)" - echo "backup_k8s_resources_timestamp_seconds $(date +%s)" - echo "backup_k8s_resources_check_status ${CHECK_STATUS}" - echo "restic_repo_size_bytes{repo=\"k8s-resources\"} ${REPO_SIZE_BYTES}" - } > /metrics/backup_k8s_resources.prom.tmp - mv /metrics/backup_k8s_resources.prom.tmp /metrics/backup_k8s_resources.prom - - echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ===" - exit ${CHECK_STATUS} - volumeMounts: - - name: dump - mountPath: /dump - readOnly: true - - name: metrics - mountPath: /metrics - - name: cache - mountPath: /root/.cache/restic - resources: - requests: - cpu: 100m - memory: 128Mi - limits: - cpu: 1500m - memory: 1Gi - volumes: - - name: dump - emptyDir: {} - - name: metrics - hostPath: - # node-exporter's textfile-collector directory - # (DEV-494). See sibling loki cronjob for detail. - path: /var/lib/node_exporter/textfile_collector - type: DirectoryOrCreate - - name: cache - emptyDir: {} diff --git a/apps/monitoring/backup-restic-alerts.test.yaml b/apps/monitoring/backup-restic-alerts.test.yaml index a961aa8..f13a3b6 100644 --- a/apps/monitoring/backup-restic-alerts.test.yaml +++ b/apps/monitoring/backup-restic-alerts.test.yaml @@ -35,8 +35,6 @@ tests: values: '-104400x60' - series: 'backup_grafana_timestamp_seconds' values: '-104400x60' - - series: 'backup_k8s_resources_timestamp_seconds' - values: '-104400x60' - series: 'backup_prometheus_timestamp_seconds' values: '-104400x60' alert_rule_test: @@ -70,21 +68,6 @@ tests: `backup_grafana_timestamp_seconds` sample in more than 28 hours. Expected daily at 03:15 UTC. Runbook: docs/monitoring/restic-restore.md - - eval_time: 30m - alertname: BackupK8sResourcesStale - exp_alerts: - - exp_labels: - severity: warning - service: monitoring - component: backup - repo: k8s-resources - exp_annotations: - summary: "K8s-resources restic backup is stale (>28h)" - description: | - `backup-k8s-resources` has not written a fresh - `backup_k8s_resources_timestamp_seconds` sample in - more than 28 hours. Expected daily at 02:00 UTC. - Runbook: docs/monitoring/restic-restore.md - eval_time: 30m alertname: BackupPrometheusStale exp_alerts: @@ -112,8 +95,6 @@ tests: values: '-3600x60' - series: 'backup_grafana_timestamp_seconds' values: '-3600x60' - - series: 'backup_k8s_resources_timestamp_seconds' - values: '-3600x60' - series: 'backup_prometheus_timestamp_seconds' values: '-3600x60' alert_rule_test: @@ -123,9 +104,6 @@ tests: - eval_time: 30m alertname: BackupGrafanaStale exp_alerts: [] - - eval_time: 30m - alertname: BackupK8sResourcesStale - exp_alerts: [] - eval_time: 30m alertname: BackupPrometheusStale exp_alerts: [] @@ -139,8 +117,6 @@ tests: values: '1x30' - series: 'backup_grafana_check_status' values: '2x30' - - series: 'backup_k8s_resources_check_status' - values: '1x30' - series: 'backup_prometheus_check_status' values: '1x30' alert_rule_test: @@ -172,21 +148,6 @@ tests: `restic check --read-data-subset=5%` returned exit code 2 on the grafana repository. Runbook: docs/monitoring/restic-restore.md - - eval_time: 15m - alertname: BackupK8sResourcesCheckFailed - exp_alerts: - - exp_labels: - severity: warning - service: monitoring - component: backup - repo: k8s-resources - exp_annotations: - summary: "restic check failed on k8s-resources repo" - description: | - `restic check --read-data-subset=5%` returned - exit code 1 on the k8s-resources - repository. - Runbook: docs/monitoring/restic-restore.md - eval_time: 15m alertname: BackupPrometheusCheckFailed exp_alerts: @@ -212,8 +173,6 @@ tests: values: '0x30' - series: 'backup_grafana_check_status' values: '0x30' - - series: 'backup_k8s_resources_check_status' - values: '0x30' - series: 'backup_prometheus_check_status' values: '0x30' alert_rule_test: @@ -223,9 +182,6 @@ tests: - eval_time: 15m alertname: BackupGrafanaCheckFailed exp_alerts: [] - - eval_time: 15m - alertname: BackupK8sResourcesCheckFailed - exp_alerts: [] - eval_time: 15m alertname: BackupPrometheusCheckFailed exp_alerts: [] diff --git a/apps/monitoring/backup-restic-alerts.yaml b/apps/monitoring/backup-restic-alerts.yaml index a30ab03..e0364ef 100644 --- a/apps/monitoring/backup-restic-alerts.yaml +++ b/apps/monitoring/backup-restic-alerts.yaml @@ -19,7 +19,7 @@ # kube-prometheus-stack node-exporter DaemonSet (DEV-494). # # Because a CronJob may run on a different worker across days (loki -# backup follows the loki pod; k8s-resources is unpinned), stale +# backup follows the loki pod), stale # `.prom` files can linger on nodes the job has since left. That # would leave a per-node series with an old timestamp/check-status # indefinitely. The alerts below aggregate across instances so a @@ -74,21 +74,6 @@ spec: `backup_grafana_timestamp_seconds` sample in more than 28 hours. Expected daily at 03:15 UTC. Runbook: docs/monitoring/restic-restore.md - - alert: BackupK8sResourcesStale - expr: time() - max(backup_k8s_resources_timestamp_seconds) > 28 * 3600 - for: 15m - labels: - severity: warning - service: monitoring - component: backup - repo: k8s-resources - annotations: - summary: "K8s-resources restic backup is stale (>28h)" - description: | - `backup-k8s-resources` has not written a fresh - `backup_k8s_resources_timestamp_seconds` sample in - more than 28 hours. Expected daily at 02:00 UTC. - Runbook: docs/monitoring/restic-restore.md - alert: BackupPrometheusStale expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600 for: 15m @@ -150,21 +135,6 @@ spec: `restic check --read-data-subset=5%` returned exit code {{ $value }} on the grafana repository. Runbook: docs/monitoring/restic-restore.md - - alert: BackupK8sResourcesCheckFailed - expr: backup_k8s_resources_check_status != 0 - for: 5m - labels: - severity: warning - service: monitoring - component: backup - repo: k8s-resources - annotations: - summary: "restic check failed on k8s-resources repo" - description: | - `restic check --read-data-subset=5%` returned - exit code {{ $value }} on the k8s-resources - repository. - Runbook: docs/monitoring/restic-restore.md - alert: BackupPrometheusCheckFailed expr: backup_prometheus_check_status != 0 for: 5m diff --git a/apps/monitoring/prometheus-backup-cronjob.yaml b/apps/monitoring/prometheus-backup-cronjob.yaml index 17f6ebb..b4a6c91 100644 --- a/apps/monitoring/prometheus-backup-cronjob.yaml +++ b/apps/monitoring/prometheus-backup-cronjob.yaml @@ -48,7 +48,7 @@ metadata: type: prometheus backend: restic spec: - schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana/k8s-resources + schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 @@ -173,7 +173,7 @@ spec: echo "restic repo size: ${REPO_SIZE_BYTES} bytes" # Textfile-collector metrics. Same hostPath pattern as - # the loki/grafana/k8s-resources siblings (DEV-494) — + # the loki/grafana siblings (DEV-494) — # written atomically via `.tmp` + rename so a mid-write # read never surfaces a truncated sample. # backup_prometheus_success rolls in both stages: the