--- # promtool test file for backup-restic-alerts.yaml (DEV-490 DoD). # # Extract the rules from the PrometheusRule wrapper first: # awk '/^spec:/{f=1;next} f{sub(/^ /,"");print}' \ # apps/monitoring/backup-restic-alerts.yaml > /tmp/rules.yaml # Then run: # promtool test rules apps/monitoring/backup-restic-alerts.test.yaml # (this file uses `rule_files: [/tmp/rules.yaml]` — pass the extracted # path explicitly). rule_files: - /tmp/backup-restic-rules.yaml # Evaluation cadence for the test scenarios. The alerts' `for:` uses 5m/15m/30m, # so we run 60m of series and evaluate at 60m so all alerts have had time to # settle into the firing state. evaluation_interval: 1m tests: # --------------------------------------------------------------------------- # 1. Freshness alerts fire when timestamp is stale (>28h old). # --------------------------------------------------------------------------- - interval: 1m input_series: # Timestamp value = "current time - 29h" in Unix seconds. During the # test, promtool's clock starts at the Unix epoch (0). We fix the # value to a constant far in the past so `time() - value > 28*3600` # for the entire test run. # # Concretely: time() returns the sample timestamp in seconds. Over a # 60-minute test starting at t=0, time() runs 0 .. 3600. A constant # value of -104400 (=-29h) keeps `time() - value` >= 29h across the # whole window, safely above the 28h threshold. - series: 'backup_loki_timestamp_seconds' values: '-104400x60' - series: 'backup_grafana_timestamp_seconds' values: '-104400x60' - series: 'backup_k8s_resources_timestamp_seconds' values: '-104400x60' - series: 'backup_prometheus_timestamp_seconds' values: '-104400x60' alert_rule_test: - eval_time: 30m alertname: BackupLokiStale exp_alerts: - exp_labels: severity: warning service: monitoring component: backup repo: loki exp_annotations: summary: "Loki restic backup is stale (>28h)" description: | `backup-loki-restic` has not written a fresh `backup_loki_timestamp_seconds` sample in more than 28 hours. Expected daily at 03:00 UTC. Runbook: docs/monitoring/restic-restore.md - eval_time: 30m alertname: BackupGrafanaStale exp_alerts: - exp_labels: severity: warning service: monitoring component: backup repo: grafana exp_annotations: summary: "Grafana restic backup is stale (>28h)" description: | `backup-grafana-restic` has not written a fresh `backup_grafana_timestamp_seconds` sample in more than 28 hours. Expected daily at 03:15 UTC. Runbook: docs/monitoring/restic-restore.md - eval_time: 30m alertname: BackupK8sResourcesStale exp_alerts: - exp_labels: severity: warning service: monitoring component: backup repo: k8s-resources exp_annotations: summary: "K8s-resources restic backup is stale (>28h)" description: | `backup-k8s-resources` has not written a fresh `backup_k8s_resources_timestamp_seconds` sample in more than 28 hours. Expected daily at 02:00 UTC. Runbook: docs/monitoring/restic-restore.md - eval_time: 30m alertname: BackupPrometheusStale exp_alerts: - exp_labels: severity: warning service: monitoring component: backup repo: prometheus exp_annotations: summary: "Prometheus restic backup is stale (>28h)" description: | `prometheus-backup` has not written a fresh `backup_prometheus_timestamp_seconds` sample in more than 28 hours. Expected daily at 03:30 UTC. Runbook: docs/monitoring/restic-restore.md # --------------------------------------------------------------------------- # 2. Freshness alerts stay silent when timestamp is fresh (<28h). # --------------------------------------------------------------------------- - interval: 1m input_series: # Value = -3600 (=-1h) → time() - value stays around 1-2h, well below # the 28h threshold across the run. - series: 'backup_loki_timestamp_seconds' values: '-3600x60' - series: 'backup_grafana_timestamp_seconds' values: '-3600x60' - series: 'backup_k8s_resources_timestamp_seconds' values: '-3600x60' - series: 'backup_prometheus_timestamp_seconds' values: '-3600x60' alert_rule_test: - eval_time: 30m alertname: BackupLokiStale exp_alerts: [] - eval_time: 30m alertname: BackupGrafanaStale exp_alerts: [] - eval_time: 30m alertname: BackupK8sResourcesStale exp_alerts: [] - eval_time: 30m alertname: BackupPrometheusStale exp_alerts: [] # --------------------------------------------------------------------------- # 3. Check-status alerts fire when the metric is non-zero. # --------------------------------------------------------------------------- - interval: 1m input_series: - series: 'backup_loki_check_status' values: '1x30' - series: 'backup_grafana_check_status' values: '2x30' - series: 'backup_k8s_resources_check_status' values: '1x30' - series: 'backup_prometheus_check_status' values: '1x30' alert_rule_test: - eval_time: 15m alertname: BackupLokiCheckFailed exp_alerts: - exp_labels: severity: warning service: monitoring component: backup repo: loki exp_annotations: summary: "restic check failed on loki repo" description: | `restic check --read-data-subset=5%` returned exit code 1 on the loki repository. Runbook: docs/monitoring/restic-restore.md - eval_time: 15m alertname: BackupGrafanaCheckFailed exp_alerts: - exp_labels: severity: warning service: monitoring component: backup repo: grafana exp_annotations: summary: "restic check failed on grafana repo" description: | `restic check --read-data-subset=5%` returned exit code 2 on the grafana repository. Runbook: docs/monitoring/restic-restore.md - eval_time: 15m alertname: BackupK8sResourcesCheckFailed exp_alerts: - exp_labels: severity: warning service: monitoring component: backup repo: k8s-resources exp_annotations: summary: "restic check failed on k8s-resources repo" description: | `restic check --read-data-subset=5%` returned exit code 1 on the k8s-resources repository. Runbook: docs/monitoring/restic-restore.md - eval_time: 15m alertname: BackupPrometheusCheckFailed exp_alerts: - exp_labels: severity: warning service: monitoring component: backup repo: prometheus exp_annotations: summary: "restic check failed on prometheus repo" description: | `restic check --read-data-subset=5%` returned exit code 1 on the prometheus repository. Runbook: docs/monitoring/restic-restore.md # --------------------------------------------------------------------------- # 4. Check-status alerts stay silent on 0. # --------------------------------------------------------------------------- - interval: 1m input_series: - series: 'backup_loki_check_status' values: '0x30' - series: 'backup_grafana_check_status' values: '0x30' - series: 'backup_k8s_resources_check_status' values: '0x30' - series: 'backup_prometheus_check_status' values: '0x30' alert_rule_test: - eval_time: 15m alertname: BackupLokiCheckFailed exp_alerts: [] - eval_time: 15m alertname: BackupGrafanaCheckFailed exp_alerts: [] - eval_time: 15m alertname: BackupK8sResourcesCheckFailed exp_alerts: [] - eval_time: 15m alertname: BackupPrometheusCheckFailed exp_alerts: [] # --------------------------------------------------------------------------- # 5. Repo-size alert fires when >20 GiB (21474836480 bytes). # --------------------------------------------------------------------------- - interval: 1m input_series: # 25 GiB = 26843545600 bytes. - series: 'restic_repo_size_bytes{repo="loki"}' values: '26843545600x45' # 15 GiB = 16106127360 bytes (below threshold). - series: 'restic_repo_size_bytes{repo="grafana"}' values: '16106127360x45' alert_rule_test: - eval_time: 40m alertname: ResticRepoOversize exp_alerts: - exp_labels: severity: warning service: monitoring component: backup repo: loki exp_annotations: summary: "restic repository loki exceeds 20 GiB" description: | Repository loki is 25GiB, above the 20 GiB guard. Baseline is <5 GiB per repo. Investigate retention/prune (see forget flags in the CronJob) and dedup effectiveness.