550d1276f7
- New PrometheusRule apps/monitoring/backup-restic-alerts.yaml with
seven warning-level rules:
* BackupLokiStale / BackupGrafanaStale / BackupK8sResourcesStale
-> time() - backup_<kind>_timestamp_seconds > 28h
* BackupLokiCheckFailed / BackupGrafanaCheckFailed /
BackupK8sResourcesCheckFailed -> backup_<kind>_check_status != 0
* ResticRepoOversize -> restic_repo_size_bytes > 20 GiB (per-repo)
- Emit restic_repo_size_bytes{repo="<kind>"} from all three restic
CronJobs (loki, grafana, k8s-resources) via `restic stats --json
--mode raw-data` so ResticRepoOversize has data to match once the
textfile-collector scrape path is wired.
- Companion promtool unit test backup-restic-alerts.test.yaml with
five scenarios (fresh/stale, check pass/fail, oversize) -- verified
locally with promtool 2.53.1: SUCCESS.
- README.md: document the three restic CronJobs, the shared
SealedSecret keys (access-key/secret-key/endpoint/bucket/
restic-password), the emitted textfile metrics, and the alert list.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
216 lines
8.3 KiB
YAML
216 lines
8.3 KiB
YAML
---
|
|
# promtool test file for backup-restic-alerts.yaml (DEV-490 DoD).
|
|
#
|
|
# Extract the rules from the PrometheusRule wrapper first:
|
|
# awk '/^spec:/{f=1;next} f{sub(/^ /,"");print}' \
|
|
# apps/monitoring/backup-restic-alerts.yaml > /tmp/rules.yaml
|
|
# Then run:
|
|
# promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
|
|
# (this file uses `rule_files: [/tmp/rules.yaml]` — pass the extracted
|
|
# path explicitly).
|
|
rule_files:
|
|
- /tmp/backup-restic-rules.yaml
|
|
|
|
# Evaluation cadence for the test scenarios. The alerts' `for:` uses 5m/15m/30m,
|
|
# so we run 60m of series and evaluate at 60m so all alerts have had time to
|
|
# settle into the firing state.
|
|
evaluation_interval: 1m
|
|
|
|
tests:
|
|
# ---------------------------------------------------------------------------
|
|
# 1. Freshness alerts fire when timestamp is stale (>28h old).
|
|
# ---------------------------------------------------------------------------
|
|
- interval: 1m
|
|
input_series:
|
|
# Timestamp value = "current time - 29h" in Unix seconds. During the
|
|
# test, promtool's clock starts at the Unix epoch (0). We fix the
|
|
# value to a constant far in the past so `time() - value > 28*3600`
|
|
# for the entire test run.
|
|
#
|
|
# Concretely: time() returns the sample timestamp in seconds. Over a
|
|
# 60-minute test starting at t=0, time() runs 0 .. 3600. A constant
|
|
# value of -104400 (=-29h) keeps `time() - value` >= 29h across the
|
|
# whole window, safely above the 28h threshold.
|
|
- series: 'backup_loki_timestamp_seconds'
|
|
values: '-104400x60'
|
|
- series: 'backup_grafana_timestamp_seconds'
|
|
values: '-104400x60'
|
|
- series: 'backup_k8s_resources_timestamp_seconds'
|
|
values: '-104400x60'
|
|
alert_rule_test:
|
|
- eval_time: 30m
|
|
alertname: BackupLokiStale
|
|
exp_alerts:
|
|
- exp_labels:
|
|
severity: warning
|
|
service: monitoring
|
|
component: backup
|
|
repo: loki
|
|
exp_annotations:
|
|
summary: "Loki restic backup is stale (>28h)"
|
|
description: |
|
|
`backup-loki-restic` has not written a fresh
|
|
`backup_loki_timestamp_seconds` sample in more than
|
|
28 hours. Expected daily at 03:00 UTC.
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
- eval_time: 30m
|
|
alertname: BackupGrafanaStale
|
|
exp_alerts:
|
|
- exp_labels:
|
|
severity: warning
|
|
service: monitoring
|
|
component: backup
|
|
repo: grafana
|
|
exp_annotations:
|
|
summary: "Grafana restic backup is stale (>28h)"
|
|
description: |
|
|
`backup-grafana-restic` has not written a fresh
|
|
`backup_grafana_timestamp_seconds` sample in more
|
|
than 28 hours. Expected daily at 03:15 UTC.
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
- eval_time: 30m
|
|
alertname: BackupK8sResourcesStale
|
|
exp_alerts:
|
|
- exp_labels:
|
|
severity: warning
|
|
service: monitoring
|
|
component: backup
|
|
repo: k8s-resources
|
|
exp_annotations:
|
|
summary: "K8s-resources restic backup is stale (>28h)"
|
|
description: |
|
|
`backup-k8s-resources` has not written a fresh
|
|
`backup_k8s_resources_timestamp_seconds` sample in
|
|
more than 28 hours. Expected daily at 02:00 UTC.
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. Freshness alerts stay silent when timestamp is fresh (<28h).
|
|
# ---------------------------------------------------------------------------
|
|
- interval: 1m
|
|
input_series:
|
|
# Value = -3600 (=-1h) → time() - value stays around 1-2h, well below
|
|
# the 28h threshold across the run.
|
|
- series: 'backup_loki_timestamp_seconds'
|
|
values: '-3600x60'
|
|
- series: 'backup_grafana_timestamp_seconds'
|
|
values: '-3600x60'
|
|
- series: 'backup_k8s_resources_timestamp_seconds'
|
|
values: '-3600x60'
|
|
alert_rule_test:
|
|
- eval_time: 30m
|
|
alertname: BackupLokiStale
|
|
exp_alerts: []
|
|
- eval_time: 30m
|
|
alertname: BackupGrafanaStale
|
|
exp_alerts: []
|
|
- eval_time: 30m
|
|
alertname: BackupK8sResourcesStale
|
|
exp_alerts: []
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. Check-status alerts fire when the metric is non-zero.
|
|
# ---------------------------------------------------------------------------
|
|
- interval: 1m
|
|
input_series:
|
|
- series: 'backup_loki_check_status'
|
|
values: '1x30'
|
|
- series: 'backup_grafana_check_status'
|
|
values: '2x30'
|
|
- series: 'backup_k8s_resources_check_status'
|
|
values: '1x30'
|
|
alert_rule_test:
|
|
- eval_time: 15m
|
|
alertname: BackupLokiCheckFailed
|
|
exp_alerts:
|
|
- exp_labels:
|
|
severity: warning
|
|
service: monitoring
|
|
component: backup
|
|
repo: loki
|
|
exp_annotations:
|
|
summary: "restic check failed on loki repo"
|
|
description: |
|
|
`restic check --read-data-subset=5%` returned
|
|
exit code 1 on the loki repository.
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
- eval_time: 15m
|
|
alertname: BackupGrafanaCheckFailed
|
|
exp_alerts:
|
|
- exp_labels:
|
|
severity: warning
|
|
service: monitoring
|
|
component: backup
|
|
repo: grafana
|
|
exp_annotations:
|
|
summary: "restic check failed on grafana repo"
|
|
description: |
|
|
`restic check --read-data-subset=5%` returned
|
|
exit code 2 on the grafana repository.
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
- eval_time: 15m
|
|
alertname: BackupK8sResourcesCheckFailed
|
|
exp_alerts:
|
|
- exp_labels:
|
|
severity: warning
|
|
service: monitoring
|
|
component: backup
|
|
repo: k8s-resources
|
|
exp_annotations:
|
|
summary: "restic check failed on k8s-resources repo"
|
|
description: |
|
|
`restic check --read-data-subset=5%` returned
|
|
exit code 1 on the k8s-resources
|
|
repository.
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 4. Check-status alerts stay silent on 0.
|
|
# ---------------------------------------------------------------------------
|
|
- interval: 1m
|
|
input_series:
|
|
- series: 'backup_loki_check_status'
|
|
values: '0x30'
|
|
- series: 'backup_grafana_check_status'
|
|
values: '0x30'
|
|
- series: 'backup_k8s_resources_check_status'
|
|
values: '0x30'
|
|
alert_rule_test:
|
|
- eval_time: 15m
|
|
alertname: BackupLokiCheckFailed
|
|
exp_alerts: []
|
|
- eval_time: 15m
|
|
alertname: BackupGrafanaCheckFailed
|
|
exp_alerts: []
|
|
- eval_time: 15m
|
|
alertname: BackupK8sResourcesCheckFailed
|
|
exp_alerts: []
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 5. Repo-size alert fires when >20 GiB (21474836480 bytes).
|
|
# ---------------------------------------------------------------------------
|
|
- interval: 1m
|
|
input_series:
|
|
# 25 GiB = 26843545600 bytes.
|
|
- series: 'restic_repo_size_bytes{repo="loki"}'
|
|
values: '26843545600x45'
|
|
# 15 GiB = 16106127360 bytes (below threshold).
|
|
- series: 'restic_repo_size_bytes{repo="grafana"}'
|
|
values: '16106127360x45'
|
|
alert_rule_test:
|
|
- eval_time: 40m
|
|
alertname: ResticRepoOversize
|
|
exp_alerts:
|
|
- exp_labels:
|
|
severity: warning
|
|
service: monitoring
|
|
component: backup
|
|
repo: loki
|
|
exp_annotations:
|
|
summary: "restic repository loki exceeds 20 GiB"
|
|
description: |
|
|
Repository loki is
|
|
25GiB, above the 20 GiB
|
|
guard. Baseline is <5 GiB per repo. Investigate
|
|
retention/prune (see forget flags in the CronJob)
|
|
and dedup effectiveness.
|