From 550d1276f75b761ccd22cd8064211559b391fb95 Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 16 Aug 2026 16:07:55 +0000 Subject: [PATCH] feat(monitoring): Prometheus backup-freshness + restic alerts (DEV-490) - New PrometheusRule apps/monitoring/backup-restic-alerts.yaml with seven warning-level rules: * BackupLokiStale / BackupGrafanaStale / BackupK8sResourcesStale -> time() - backup__timestamp_seconds > 28h * BackupLokiCheckFailed / BackupGrafanaCheckFailed / BackupK8sResourcesCheckFailed -> backup__check_status != 0 * ResticRepoOversize -> restic_repo_size_bytes > 20 GiB (per-repo) - Emit restic_repo_size_bytes{repo=""} from all three restic CronJobs (loki, grafana, k8s-resources) via `restic stats --json --mode raw-data` so ResticRepoOversize has data to match once the textfile-collector scrape path is wired. - Companion promtool unit test backup-restic-alerts.test.yaml with five scenarios (fresh/stale, check pass/fail, oversize) -- verified locally with promtool 2.53.1: SUCCESS. - README.md: document the three restic CronJobs, the shared SealedSecret keys (access-key/secret-key/endpoint/bucket/ restic-password), the emitted textfile metrics, and the alert list. Co-Authored-By: Paperclip --- apps/monitoring/README.md | 45 +++- .../backup-grafana-restic-cronjob.yaml | 9 + .../backup-k8s-resources-cronjob.yaml | 9 + .../backup-loki-restic-cronjob.yaml | 13 ++ .../monitoring/backup-restic-alerts.test.yaml | 216 ++++++++++++++++++ apps/monitoring/backup-restic-alerts.yaml | 145 ++++++++++++ 6 files changed, 436 insertions(+), 1 deletion(-) create mode 100644 apps/monitoring/backup-restic-alerts.test.yaml create mode 100644 apps/monitoring/backup-restic-alerts.yaml diff --git a/apps/monitoring/README.md b/apps/monitoring/README.md index 01675ea..1875988 100644 --- a/apps/monitoring/README.md +++ b/apps/monitoring/README.md @@ -5,8 +5,51 @@ Manifests recording the cluster-side monitoring backup CronJobs that were previo - `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)). - `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)). - `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)). -- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)). +- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)). Note: still uses `rclone sync` (plaintext at rest on Hetzner). Migration to `restic` or `rclone crypt` is filed as a follow-up. +- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos, plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490)). + +## Shared SealedSecret + +All three restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys: + +| Key | Purpose | +|------------------|------------------------------------------------------------------------------------------------------------------| +| `access-key` | Hetzner Object Storage access key ID | +| `secret-key` | Hetzner Object Storage secret access key | +| `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) | +| `bucket` | Bucket name (single bucket, per-prefix repos) | +| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. | + +## Restore Restore procedure for all three restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16) passed — see [DEV-488](/DEV/issues/DEV-488). +## Emitted metrics (textfile-collector format) + +Each restic CronJob writes to `/metrics/backup_.prom` inside the pod's emptyDir. Once node-exporter's textfile-collector path is wired up (follow-up), these become scrapeable and the alerts in `backup-restic-alerts.yaml` evaluate against live data. + +| Metric | Emitted by | +|--------------------------------------------|---------------------------------------------------------------------| +| `backup__success` (0/1) | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml` | +| `backup__timestamp_seconds` | same | +| `backup__check_status` (exit code) | same — from `restic check --read-data-subset=5%` | +| `restic_repo_size_bytes{repo=""}` | same — from `restic stats --json --mode raw-data` (added in DEV-490)| + +## Alerts + +`backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`): + +- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` — `time() - backup__timestamp_seconds > 28h`. Daily schedule + 4 h grace. +- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` — `backup__check_status != 0`. +- `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions. + +All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally: + +```bash +awk '/^spec:/{f=1;next} f{sub(/^ /,"");print}' \ + apps/monitoring/backup-restic-alerts.yaml > /tmp/backup-restic-rules.yaml +promtool check rules /tmp/backup-restic-rules.yaml +promtool test rules apps/monitoring/backup-restic-alerts.test.yaml +``` + The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` and `prometheus-backup` remain unpinned. diff --git a/apps/monitoring/backup-grafana-restic-cronjob.yaml b/apps/monitoring/backup-grafana-restic-cronjob.yaml index c888c8a..63d5a5d 100644 --- a/apps/monitoring/backup-grafana-restic-cronjob.yaml +++ b/apps/monitoring/backup-grafana-restic-cronjob.yaml @@ -107,6 +107,14 @@ spec: restic check --read-data-subset=5% || CHECK_STATUS=$? echo "restic check exit: ${CHECK_STATUS}" + echo "--- restic stats (repo size) ---" + REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \ + | grep -oE '"total_size":[0-9]+' \ + | head -1 \ + | cut -d: -f2) + REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0} + echo "restic repo size: ${REPO_SIZE_BYTES} bytes" + # Textfile-collector metrics; identical wiring to the loki # sibling. Scrapeable once node-exporter's textfile # collector path is enabled — tracked in DEV-482. @@ -114,6 +122,7 @@ spec: echo "backup_grafana_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)" echo "backup_grafana_timestamp_seconds $(date +%s)" echo "backup_grafana_check_status ${CHECK_STATUS}" + echo "restic_repo_size_bytes{repo=\"grafana\"} ${REPO_SIZE_BYTES}" } > /metrics/backup_grafana.prom echo "=== backup-grafana-restic finished at $(date -u +%FT%TZ) ===" diff --git a/apps/monitoring/backup-k8s-resources-cronjob.yaml b/apps/monitoring/backup-k8s-resources-cronjob.yaml index 3d7c723..0eae91a 100644 --- a/apps/monitoring/backup-k8s-resources-cronjob.yaml +++ b/apps/monitoring/backup-k8s-resources-cronjob.yaml @@ -153,6 +153,14 @@ spec: restic check --read-data-subset=5% || CHECK_STATUS=$? echo "restic check exit: ${CHECK_STATUS}" + echo "--- restic stats (repo size) ---" + REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \ + | grep -oE '"total_size":[0-9]+' \ + | head -1 \ + | cut -d: -f2) + REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0} + echo "restic repo size: ${REPO_SIZE_BYTES} bytes" + # Textfile-collector metrics; identical wiring to the # loki/grafana siblings. Scrapeable once node-exporter's # textfile collector path is enabled — tracked in DEV-482. @@ -160,6 +168,7 @@ spec: echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)" echo "backup_k8s_resources_timestamp_seconds $(date +%s)" echo "backup_k8s_resources_check_status ${CHECK_STATUS}" + echo "restic_repo_size_bytes{repo=\"k8s-resources\"} ${REPO_SIZE_BYTES}" } > /metrics/backup_k8s_resources.prom echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ===" diff --git a/apps/monitoring/backup-loki-restic-cronjob.yaml b/apps/monitoring/backup-loki-restic-cronjob.yaml index 8274191..31e67f9 100644 --- a/apps/monitoring/backup-loki-restic-cronjob.yaml +++ b/apps/monitoring/backup-loki-restic-cronjob.yaml @@ -119,6 +119,18 @@ spec: restic check --read-data-subset=5% || CHECK_STATUS=$? echo "restic check exit: ${CHECK_STATUS}" + echo "--- restic stats (repo size) ---" + # `restic stats --json --mode raw-data` prints e.g. + # {"total_size":123,"total_file_count":45,...}. Extract + # total_size without jq (not present in the restic image) + # via grep/cut; fall back to 0 on empty output. + REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \ + | grep -oE '"total_size":[0-9]+' \ + | head -1 \ + | cut -d: -f2) + REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0} + echo "restic repo size: ${REPO_SIZE_BYTES} bytes" + # Textfile-collector metrics. Written into an emptyDir per # the current pattern used by the other backup CronJobs. # Once a node-exporter textfile collector path is wired @@ -128,6 +140,7 @@ spec: echo "backup_loki_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)" echo "backup_loki_timestamp_seconds $(date +%s)" echo "backup_loki_check_status ${CHECK_STATUS}" + echo "restic_repo_size_bytes{repo=\"loki\"} ${REPO_SIZE_BYTES}" } > /metrics/backup_loki.prom echo "=== backup-loki-restic finished at $(date -u +%FT%TZ) ===" diff --git a/apps/monitoring/backup-restic-alerts.test.yaml b/apps/monitoring/backup-restic-alerts.test.yaml new file mode 100644 index 0000000..095fdbf --- /dev/null +++ b/apps/monitoring/backup-restic-alerts.test.yaml @@ -0,0 +1,216 @@ +--- +# promtool test file for backup-restic-alerts.yaml (DEV-490 DoD). +# +# Extract the rules from the PrometheusRule wrapper first: +# awk '/^spec:/{f=1;next} f{sub(/^ /,"");print}' \ +# apps/monitoring/backup-restic-alerts.yaml > /tmp/rules.yaml +# Then run: +# promtool test rules apps/monitoring/backup-restic-alerts.test.yaml +# (this file uses `rule_files: [/tmp/rules.yaml]` — pass the extracted +# path explicitly). +rule_files: + - /tmp/backup-restic-rules.yaml + +# Evaluation cadence for the test scenarios. The alerts' `for:` uses 5m/15m/30m, +# so we run 60m of series and evaluate at 60m so all alerts have had time to +# settle into the firing state. +evaluation_interval: 1m + +tests: + # --------------------------------------------------------------------------- + # 1. Freshness alerts fire when timestamp is stale (>28h old). + # --------------------------------------------------------------------------- + - interval: 1m + input_series: + # Timestamp value = "current time - 29h" in Unix seconds. During the + # test, promtool's clock starts at the Unix epoch (0). We fix the + # value to a constant far in the past so `time() - value > 28*3600` + # for the entire test run. + # + # Concretely: time() returns the sample timestamp in seconds. Over a + # 60-minute test starting at t=0, time() runs 0 .. 3600. A constant + # value of -104400 (=-29h) keeps `time() - value` >= 29h across the + # whole window, safely above the 28h threshold. + - series: 'backup_loki_timestamp_seconds' + values: '-104400x60' + - series: 'backup_grafana_timestamp_seconds' + values: '-104400x60' + - series: 'backup_k8s_resources_timestamp_seconds' + values: '-104400x60' + alert_rule_test: + - eval_time: 30m + alertname: BackupLokiStale + exp_alerts: + - exp_labels: + severity: warning + service: monitoring + component: backup + repo: loki + exp_annotations: + summary: "Loki restic backup is stale (>28h)" + description: | + `backup-loki-restic` has not written a fresh + `backup_loki_timestamp_seconds` sample in more than + 28 hours. Expected daily at 03:00 UTC. + Runbook: docs/monitoring/restic-restore.md + - eval_time: 30m + alertname: BackupGrafanaStale + exp_alerts: + - exp_labels: + severity: warning + service: monitoring + component: backup + repo: grafana + exp_annotations: + summary: "Grafana restic backup is stale (>28h)" + description: | + `backup-grafana-restic` has not written a fresh + `backup_grafana_timestamp_seconds` sample in more + than 28 hours. Expected daily at 03:15 UTC. + Runbook: docs/monitoring/restic-restore.md + - eval_time: 30m + alertname: BackupK8sResourcesStale + exp_alerts: + - exp_labels: + severity: warning + service: monitoring + component: backup + repo: k8s-resources + exp_annotations: + summary: "K8s-resources restic backup is stale (>28h)" + description: | + `backup-k8s-resources` has not written a fresh + `backup_k8s_resources_timestamp_seconds` sample in + more than 28 hours. Expected daily at 02:00 UTC. + Runbook: docs/monitoring/restic-restore.md + + # --------------------------------------------------------------------------- + # 2. Freshness alerts stay silent when timestamp is fresh (<28h). + # --------------------------------------------------------------------------- + - interval: 1m + input_series: + # Value = -3600 (=-1h) → time() - value stays around 1-2h, well below + # the 28h threshold across the run. + - series: 'backup_loki_timestamp_seconds' + values: '-3600x60' + - series: 'backup_grafana_timestamp_seconds' + values: '-3600x60' + - series: 'backup_k8s_resources_timestamp_seconds' + values: '-3600x60' + alert_rule_test: + - eval_time: 30m + alertname: BackupLokiStale + exp_alerts: [] + - eval_time: 30m + alertname: BackupGrafanaStale + exp_alerts: [] + - eval_time: 30m + alertname: BackupK8sResourcesStale + exp_alerts: [] + + # --------------------------------------------------------------------------- + # 3. Check-status alerts fire when the metric is non-zero. + # --------------------------------------------------------------------------- + - interval: 1m + input_series: + - series: 'backup_loki_check_status' + values: '1x30' + - series: 'backup_grafana_check_status' + values: '2x30' + - series: 'backup_k8s_resources_check_status' + values: '1x30' + alert_rule_test: + - eval_time: 15m + alertname: BackupLokiCheckFailed + exp_alerts: + - exp_labels: + severity: warning + service: monitoring + component: backup + repo: loki + exp_annotations: + summary: "restic check failed on loki repo" + description: | + `restic check --read-data-subset=5%` returned + exit code 1 on the loki repository. + Runbook: docs/monitoring/restic-restore.md + - eval_time: 15m + alertname: BackupGrafanaCheckFailed + exp_alerts: + - exp_labels: + severity: warning + service: monitoring + component: backup + repo: grafana + exp_annotations: + summary: "restic check failed on grafana repo" + description: | + `restic check --read-data-subset=5%` returned + exit code 2 on the grafana repository. + Runbook: docs/monitoring/restic-restore.md + - eval_time: 15m + alertname: BackupK8sResourcesCheckFailed + exp_alerts: + - exp_labels: + severity: warning + service: monitoring + component: backup + repo: k8s-resources + exp_annotations: + summary: "restic check failed on k8s-resources repo" + description: | + `restic check --read-data-subset=5%` returned + exit code 1 on the k8s-resources + repository. + Runbook: docs/monitoring/restic-restore.md + + # --------------------------------------------------------------------------- + # 4. Check-status alerts stay silent on 0. + # --------------------------------------------------------------------------- + - interval: 1m + input_series: + - series: 'backup_loki_check_status' + values: '0x30' + - series: 'backup_grafana_check_status' + values: '0x30' + - series: 'backup_k8s_resources_check_status' + values: '0x30' + alert_rule_test: + - eval_time: 15m + alertname: BackupLokiCheckFailed + exp_alerts: [] + - eval_time: 15m + alertname: BackupGrafanaCheckFailed + exp_alerts: [] + - eval_time: 15m + alertname: BackupK8sResourcesCheckFailed + exp_alerts: [] + + # --------------------------------------------------------------------------- + # 5. Repo-size alert fires when >20 GiB (21474836480 bytes). + # --------------------------------------------------------------------------- + - interval: 1m + input_series: + # 25 GiB = 26843545600 bytes. + - series: 'restic_repo_size_bytes{repo="loki"}' + values: '26843545600x45' + # 15 GiB = 16106127360 bytes (below threshold). + - series: 'restic_repo_size_bytes{repo="grafana"}' + values: '16106127360x45' + alert_rule_test: + - eval_time: 40m + alertname: ResticRepoOversize + exp_alerts: + - exp_labels: + severity: warning + service: monitoring + component: backup + repo: loki + exp_annotations: + summary: "restic repository loki exceeds 20 GiB" + description: | + Repository loki is + 25GiB, above the 20 GiB + guard. Baseline is <5 GiB per repo. Investigate + retention/prune (see forget flags in the CronJob) + and dedup effectiveness. diff --git a/apps/monitoring/backup-restic-alerts.yaml b/apps/monitoring/backup-restic-alerts.yaml new file mode 100644 index 0000000..d3fc386 --- /dev/null +++ b/apps/monitoring/backup-restic-alerts.yaml @@ -0,0 +1,145 @@ +--- +# Prometheus alerting rules for the restic-based monitoring backups +# (DEV-490 / DEV-482 Option 4). Freshness alerts fire when a +# CronJob has not emitted its `backup__timestamp_seconds` +# metric within 28h (schedules are daily; 28h gives one missed +# run + 4h grace before we page). +# +# Check-status alerts fire on the first non-zero exit from +# `restic check --read-data-subset=5%`. +# +# Repo-size alerts fire when a repository exceeds 20 GiB — the +# baseline is expected < 5 GiB per repo. The alert catches +# retention/prune bugs and runaway growth. The metric is emitted +# by the same CronJob step (see backup-*-restic-cronjob.yaml, +# `restic stats --json --mode raw-data`). +# +# The metrics are written to an emptyDir at /metrics per job. +# They become scrapeable once node-exporter's textfile collector +# path is wired up — tracked as a follow-up. `promtool test rules` +# against apps/monitoring/backup-restic-alerts.test.yaml verifies +# the rules with synthetic samples in the meantime. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: backup-restic-alerts + namespace: monitoring + labels: + app: backup + backend: restic + release: kube-prometheus-stack +spec: + groups: + - name: backup-restic.freshness + interval: 60s + rules: + - alert: BackupLokiStale + expr: time() - backup_loki_timestamp_seconds > 28 * 3600 + for: 15m + labels: + severity: warning + service: monitoring + component: backup + repo: loki + annotations: + summary: "Loki restic backup is stale (>28h)" + description: | + `backup-loki-restic` has not written a fresh + `backup_loki_timestamp_seconds` sample in more than + 28 hours. Expected daily at 03:00 UTC. + Runbook: docs/monitoring/restic-restore.md + - alert: BackupGrafanaStale + expr: time() - backup_grafana_timestamp_seconds > 28 * 3600 + for: 15m + labels: + severity: warning + service: monitoring + component: backup + repo: grafana + annotations: + summary: "Grafana restic backup is stale (>28h)" + description: | + `backup-grafana-restic` has not written a fresh + `backup_grafana_timestamp_seconds` sample in more + than 28 hours. Expected daily at 03:15 UTC. + Runbook: docs/monitoring/restic-restore.md + - alert: BackupK8sResourcesStale + expr: time() - backup_k8s_resources_timestamp_seconds > 28 * 3600 + for: 15m + labels: + severity: warning + service: monitoring + component: backup + repo: k8s-resources + annotations: + summary: "K8s-resources restic backup is stale (>28h)" + description: | + `backup-k8s-resources` has not written a fresh + `backup_k8s_resources_timestamp_seconds` sample in + more than 28 hours. Expected daily at 02:00 UTC. + Runbook: docs/monitoring/restic-restore.md + - name: backup-restic.integrity + interval: 60s + rules: + - alert: BackupLokiCheckFailed + expr: backup_loki_check_status != 0 + for: 5m + labels: + severity: warning + service: monitoring + component: backup + repo: loki + annotations: + summary: "restic check failed on loki repo" + description: | + `restic check --read-data-subset=5%` returned + exit code {{ $value }} on the loki repository. + Runbook: docs/monitoring/restic-restore.md + - alert: BackupGrafanaCheckFailed + expr: backup_grafana_check_status != 0 + for: 5m + labels: + severity: warning + service: monitoring + component: backup + repo: grafana + annotations: + summary: "restic check failed on grafana repo" + description: | + `restic check --read-data-subset=5%` returned + exit code {{ $value }} on the grafana repository. + Runbook: docs/monitoring/restic-restore.md + - alert: BackupK8sResourcesCheckFailed + expr: backup_k8s_resources_check_status != 0 + for: 5m + labels: + severity: warning + service: monitoring + component: backup + repo: k8s-resources + annotations: + summary: "restic check failed on k8s-resources repo" + description: | + `restic check --read-data-subset=5%` returned + exit code {{ $value }} on the k8s-resources + repository. + Runbook: docs/monitoring/restic-restore.md + - name: backup-restic.size + interval: 60s + rules: + - alert: ResticRepoOversize + # 20 GiB = 20 * 1024^3 = 21474836480 bytes. + expr: restic_repo_size_bytes > 21474836480 + for: 30m + labels: + severity: warning + service: monitoring + component: backup + annotations: + summary: "restic repository {{ $labels.repo }} exceeds 20 GiB" + description: | + Repository {{ $labels.repo }} is + {{ $value | humanize1024 }}B, above the 20 GiB + guard. Baseline is <5 GiB per repo. Investigate + retention/prune (see forget flags in the CronJob) + and dedup effectiveness.