2026-08-16 16:07:55 +00:00
|
|
|
---
|
|
|
|
|
# promtool test file for backup-restic-alerts.yaml (DEV-490 DoD).
|
|
|
|
|
#
|
|
|
|
|
# Extract the rules from the PrometheusRule wrapper first:
|
|
|
|
|
# awk '/^spec:/{f=1;next} f{sub(/^ /,"");print}' \
|
|
|
|
|
# apps/monitoring/backup-restic-alerts.yaml > /tmp/rules.yaml
|
|
|
|
|
# Then run:
|
|
|
|
|
# promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
|
|
|
|
|
# (this file uses `rule_files: [/tmp/rules.yaml]` — pass the extracted
|
|
|
|
|
# path explicitly).
|
|
|
|
|
rule_files:
|
|
|
|
|
- /tmp/backup-restic-rules.yaml
|
|
|
|
|
|
|
|
|
|
# Evaluation cadence for the test scenarios. The alerts' `for:` uses 5m/15m/30m,
|
|
|
|
|
# so we run 60m of series and evaluate at 60m so all alerts have had time to
|
|
|
|
|
# settle into the firing state.
|
|
|
|
|
evaluation_interval: 1m
|
|
|
|
|
|
|
|
|
|
tests:
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
# 1. Freshness alerts fire when timestamp is stale (>28h old).
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
- interval: 1m
|
|
|
|
|
input_series:
|
|
|
|
|
# Timestamp value = "current time - 29h" in Unix seconds. During the
|
|
|
|
|
# test, promtool's clock starts at the Unix epoch (0). We fix the
|
|
|
|
|
# value to a constant far in the past so `time() - value > 28*3600`
|
|
|
|
|
# for the entire test run.
|
|
|
|
|
#
|
|
|
|
|
# Concretely: time() returns the sample timestamp in seconds. Over a
|
|
|
|
|
# 60-minute test starting at t=0, time() runs 0 .. 3600. A constant
|
|
|
|
|
# value of -104400 (=-29h) keeps `time() - value` >= 29h across the
|
|
|
|
|
# whole window, safely above the 28h threshold.
|
|
|
|
|
- series: 'backup_loki_timestamp_seconds'
|
|
|
|
|
values: '-104400x60'
|
|
|
|
|
- series: 'backup_grafana_timestamp_seconds'
|
|
|
|
|
values: '-104400x60'
|
|
|
|
|
- series: 'backup_k8s_resources_timestamp_seconds'
|
|
|
|
|
values: '-104400x60'
|
feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
|
|
|
- series: 'backup_prometheus_timestamp_seconds'
|
|
|
|
|
values: '-104400x60'
|
2026-08-16 16:07:55 +00:00
|
|
|
alert_rule_test:
|
|
|
|
|
- eval_time: 30m
|
|
|
|
|
alertname: BackupLokiStale
|
|
|
|
|
exp_alerts:
|
|
|
|
|
- exp_labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
service: monitoring
|
|
|
|
|
component: backup
|
|
|
|
|
repo: loki
|
|
|
|
|
exp_annotations:
|
|
|
|
|
summary: "Loki restic backup is stale (>28h)"
|
|
|
|
|
description: |
|
|
|
|
|
`backup-loki-restic` has not written a fresh
|
|
|
|
|
`backup_loki_timestamp_seconds` sample in more than
|
|
|
|
|
28 hours. Expected daily at 03:00 UTC.
|
|
|
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
|
|
|
- eval_time: 30m
|
|
|
|
|
alertname: BackupGrafanaStale
|
|
|
|
|
exp_alerts:
|
|
|
|
|
- exp_labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
service: monitoring
|
|
|
|
|
component: backup
|
|
|
|
|
repo: grafana
|
|
|
|
|
exp_annotations:
|
|
|
|
|
summary: "Grafana restic backup is stale (>28h)"
|
|
|
|
|
description: |
|
|
|
|
|
`backup-grafana-restic` has not written a fresh
|
|
|
|
|
`backup_grafana_timestamp_seconds` sample in more
|
|
|
|
|
than 28 hours. Expected daily at 03:15 UTC.
|
|
|
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
|
|
|
- eval_time: 30m
|
|
|
|
|
alertname: BackupK8sResourcesStale
|
|
|
|
|
exp_alerts:
|
|
|
|
|
- exp_labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
service: monitoring
|
|
|
|
|
component: backup
|
|
|
|
|
repo: k8s-resources
|
|
|
|
|
exp_annotations:
|
|
|
|
|
summary: "K8s-resources restic backup is stale (>28h)"
|
|
|
|
|
description: |
|
|
|
|
|
`backup-k8s-resources` has not written a fresh
|
|
|
|
|
`backup_k8s_resources_timestamp_seconds` sample in
|
|
|
|
|
more than 28 hours. Expected daily at 02:00 UTC.
|
|
|
|
|
Runbook: docs/monitoring/restic-restore.md
|
feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
|
|
|
- eval_time: 30m
|
|
|
|
|
alertname: BackupPrometheusStale
|
|
|
|
|
exp_alerts:
|
|
|
|
|
- exp_labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
service: monitoring
|
|
|
|
|
component: backup
|
|
|
|
|
repo: prometheus
|
|
|
|
|
exp_annotations:
|
|
|
|
|
summary: "Prometheus restic backup is stale (>28h)"
|
|
|
|
|
description: |
|
|
|
|
|
`prometheus-backup` has not written a fresh
|
|
|
|
|
`backup_prometheus_timestamp_seconds` sample in more
|
|
|
|
|
than 28 hours. Expected daily at 03:30 UTC.
|
|
|
|
|
Runbook: docs/monitoring/restic-restore.md
|
2026-08-16 16:07:55 +00:00
|
|
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
# 2. Freshness alerts stay silent when timestamp is fresh (<28h).
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
- interval: 1m
|
|
|
|
|
input_series:
|
|
|
|
|
# Value = -3600 (=-1h) → time() - value stays around 1-2h, well below
|
|
|
|
|
# the 28h threshold across the run.
|
|
|
|
|
- series: 'backup_loki_timestamp_seconds'
|
|
|
|
|
values: '-3600x60'
|
|
|
|
|
- series: 'backup_grafana_timestamp_seconds'
|
|
|
|
|
values: '-3600x60'
|
|
|
|
|
- series: 'backup_k8s_resources_timestamp_seconds'
|
|
|
|
|
values: '-3600x60'
|
feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
|
|
|
- series: 'backup_prometheus_timestamp_seconds'
|
|
|
|
|
values: '-3600x60'
|
2026-08-16 16:07:55 +00:00
|
|
|
alert_rule_test:
|
|
|
|
|
- eval_time: 30m
|
|
|
|
|
alertname: BackupLokiStale
|
|
|
|
|
exp_alerts: []
|
|
|
|
|
- eval_time: 30m
|
|
|
|
|
alertname: BackupGrafanaStale
|
|
|
|
|
exp_alerts: []
|
|
|
|
|
- eval_time: 30m
|
|
|
|
|
alertname: BackupK8sResourcesStale
|
|
|
|
|
exp_alerts: []
|
feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
|
|
|
- eval_time: 30m
|
|
|
|
|
alertname: BackupPrometheusStale
|
|
|
|
|
exp_alerts: []
|
2026-08-16 16:07:55 +00:00
|
|
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
# 3. Check-status alerts fire when the metric is non-zero.
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
- interval: 1m
|
|
|
|
|
input_series:
|
|
|
|
|
- series: 'backup_loki_check_status'
|
|
|
|
|
values: '1x30'
|
|
|
|
|
- series: 'backup_grafana_check_status'
|
|
|
|
|
values: '2x30'
|
|
|
|
|
- series: 'backup_k8s_resources_check_status'
|
|
|
|
|
values: '1x30'
|
feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
|
|
|
- series: 'backup_prometheus_check_status'
|
|
|
|
|
values: '1x30'
|
2026-08-16 16:07:55 +00:00
|
|
|
alert_rule_test:
|
|
|
|
|
- eval_time: 15m
|
|
|
|
|
alertname: BackupLokiCheckFailed
|
|
|
|
|
exp_alerts:
|
|
|
|
|
- exp_labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
service: monitoring
|
|
|
|
|
component: backup
|
|
|
|
|
repo: loki
|
|
|
|
|
exp_annotations:
|
|
|
|
|
summary: "restic check failed on loki repo"
|
|
|
|
|
description: |
|
|
|
|
|
`restic check --read-data-subset=5%` returned
|
|
|
|
|
exit code 1 on the loki repository.
|
|
|
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
|
|
|
- eval_time: 15m
|
|
|
|
|
alertname: BackupGrafanaCheckFailed
|
|
|
|
|
exp_alerts:
|
|
|
|
|
- exp_labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
service: monitoring
|
|
|
|
|
component: backup
|
|
|
|
|
repo: grafana
|
|
|
|
|
exp_annotations:
|
|
|
|
|
summary: "restic check failed on grafana repo"
|
|
|
|
|
description: |
|
|
|
|
|
`restic check --read-data-subset=5%` returned
|
|
|
|
|
exit code 2 on the grafana repository.
|
|
|
|
|
Runbook: docs/monitoring/restic-restore.md
|
|
|
|
|
- eval_time: 15m
|
|
|
|
|
alertname: BackupK8sResourcesCheckFailed
|
|
|
|
|
exp_alerts:
|
|
|
|
|
- exp_labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
service: monitoring
|
|
|
|
|
component: backup
|
|
|
|
|
repo: k8s-resources
|
|
|
|
|
exp_annotations:
|
|
|
|
|
summary: "restic check failed on k8s-resources repo"
|
|
|
|
|
description: |
|
|
|
|
|
`restic check --read-data-subset=5%` returned
|
|
|
|
|
exit code 1 on the k8s-resources
|
|
|
|
|
repository.
|
|
|
|
|
Runbook: docs/monitoring/restic-restore.md
|
feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
|
|
|
- eval_time: 15m
|
|
|
|
|
alertname: BackupPrometheusCheckFailed
|
|
|
|
|
exp_alerts:
|
|
|
|
|
- exp_labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
service: monitoring
|
|
|
|
|
component: backup
|
|
|
|
|
repo: prometheus
|
|
|
|
|
exp_annotations:
|
|
|
|
|
summary: "restic check failed on prometheus repo"
|
|
|
|
|
description: |
|
|
|
|
|
`restic check --read-data-subset=5%` returned
|
|
|
|
|
exit code 1 on the prometheus
|
|
|
|
|
repository.
|
|
|
|
|
Runbook: docs/monitoring/restic-restore.md
|
2026-08-16 16:07:55 +00:00
|
|
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
# 4. Check-status alerts stay silent on 0.
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
- interval: 1m
|
|
|
|
|
input_series:
|
|
|
|
|
- series: 'backup_loki_check_status'
|
|
|
|
|
values: '0x30'
|
|
|
|
|
- series: 'backup_grafana_check_status'
|
|
|
|
|
values: '0x30'
|
|
|
|
|
- series: 'backup_k8s_resources_check_status'
|
|
|
|
|
values: '0x30'
|
feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
|
|
|
- series: 'backup_prometheus_check_status'
|
|
|
|
|
values: '0x30'
|
2026-08-16 16:07:55 +00:00
|
|
|
alert_rule_test:
|
|
|
|
|
- eval_time: 15m
|
|
|
|
|
alertname: BackupLokiCheckFailed
|
|
|
|
|
exp_alerts: []
|
|
|
|
|
- eval_time: 15m
|
|
|
|
|
alertname: BackupGrafanaCheckFailed
|
|
|
|
|
exp_alerts: []
|
|
|
|
|
- eval_time: 15m
|
|
|
|
|
alertname: BackupK8sResourcesCheckFailed
|
|
|
|
|
exp_alerts: []
|
feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
|
|
|
- eval_time: 15m
|
|
|
|
|
alertname: BackupPrometheusCheckFailed
|
|
|
|
|
exp_alerts: []
|
2026-08-16 16:07:55 +00:00
|
|
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
# 5. Repo-size alert fires when >20 GiB (21474836480 bytes).
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
- interval: 1m
|
|
|
|
|
input_series:
|
|
|
|
|
# 25 GiB = 26843545600 bytes.
|
|
|
|
|
- series: 'restic_repo_size_bytes{repo="loki"}'
|
|
|
|
|
values: '26843545600x45'
|
|
|
|
|
# 15 GiB = 16106127360 bytes (below threshold).
|
|
|
|
|
- series: 'restic_repo_size_bytes{repo="grafana"}'
|
|
|
|
|
values: '16106127360x45'
|
|
|
|
|
alert_rule_test:
|
|
|
|
|
- eval_time: 40m
|
|
|
|
|
alertname: ResticRepoOversize
|
|
|
|
|
exp_alerts:
|
|
|
|
|
- exp_labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
service: monitoring
|
|
|
|
|
component: backup
|
|
|
|
|
repo: loki
|
|
|
|
|
exp_annotations:
|
|
|
|
|
summary: "restic repository loki exceeds 20 GiB"
|
|
|
|
|
description: |
|
|
|
|
|
Repository loki is
|
|
|
|
|
25GiB, above the 20 GiB
|
|
|
|
|
guard. Baseline is <5 GiB per repo. Investigate
|
|
|
|
|
retention/prune (see forget flags in the CronJob)
|
|
|
|
|
and dedup effectiveness.
|
