basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/apps/monitoring/backup-restic-alerts.yaml
CTO Agent ef62dde67c feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494) 
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.

- apps/observability/patches/node-exporter-textfile-collector.yaml:
  strategic-merge patch on the kube-prometheus-stack node-exporter DS
  that adds `--collector.textfile.directory=/host/textfile_collector`
  and mounts `/var/lib/node_exporter/textfile_collector` read-only.
  Chart isn't tracked in ArgoCD, so we keep the patch under version
  control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
  swap `emptyDir` /metrics for a `hostPath` on the same directory
  (`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
  never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
  for all four freshness alerts so a stale `.prom` left on a node the
  job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
  the on-node directory, atomic write, cross-node staleness rationale.

Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00

186 lines
7.3 KiB
YAML
Raw Blame History

---
# Prometheus alerting rules for the restic-based monitoring backups
# (DEV-490 / DEV-482 Option 4). Freshness alerts fire when a
# CronJob has not emitted its `backup_<kind>_timestamp_seconds`
# metric within 28h (schedules are daily; 28h gives one missed
# run + 4h grace before we page).
#
# Check-status alerts fire on the first non-zero exit from
# `restic check --read-data-subset=5%`.
#
# Repo-size alerts fire when a repository exceeds 20 GiB — the
# baseline is expected < 5 GiB per repo. The alert catches
# retention/prune bugs and runaway growth. The metric is emitted
# by the same CronJob step (see backup-*-restic-cronjob.yaml,
# `restic stats --json --mode raw-data`).
#
# The metrics are written to the node's textfile-collector directory
# (`/var/lib/node_exporter/textfile_collector`) and scraped by the
# kube-prometheus-stack node-exporter DaemonSet (DEV-494).
#
# Because a CronJob may run on a different worker across days (loki
# backup follows the loki pod; k8s-resources is unpinned), stale
# `.prom` files can linger on nodes the job has since left. That
# would leave a per-node series with an old timestamp/check-status
# indefinitely. The alerts below aggregate across instances so a
# single fresh sample from the node where the job currently runs is
# enough to keep the freshness alert quiet, and check/size alerts
# fire when *any* node reports a bad value (which is the correct
# behaviour — a recent failure is still a signal).
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: backup-restic-alerts
namespace: monitoring
labels:
app: backup
backend: restic
release: kube-prometheus-stack
spec:
groups:
- name: backup-restic.freshness
interval: 60s
rules:
- alert: BackupLokiStale
# `max()` collapses per-node samples so a stale `.prom` file
# left behind on a node the loki backup no longer runs on
# does not fire this alert; the freshest sample wins.
expr: time() - max(backup_loki_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: loki
annotations:
summary: "Loki restic backup is stale (>28h)"
description: |
`backup-loki-restic` has not written a fresh
`backup_loki_timestamp_seconds` sample in more than
28 hours. Expected daily at 03:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupGrafanaStale
expr: time() - max(backup_grafana_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: grafana
annotations:
summary: "Grafana restic backup is stale (>28h)"
description: |
`backup-grafana-restic` has not written a fresh
`backup_grafana_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:15 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupK8sResourcesStale
expr: time() - max(backup_k8s_resources_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: k8s-resources
annotations:
summary: "K8s-resources restic backup is stale (>28h)"
description: |
`backup-k8s-resources` has not written a fresh
`backup_k8s_resources_timestamp_seconds` sample in
more than 28 hours. Expected daily at 02:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupPrometheusStale
expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
annotations:
summary: "Prometheus restic backup is stale (>28h)"
description: |
`prometheus-backup` has not written a fresh
`backup_prometheus_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:30 UTC.
Runbook: docs/monitoring/restic-restore.md
- name: backup-restic.integrity
interval: 60s
rules:
- alert: BackupLokiCheckFailed
expr: backup_loki_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: loki
annotations:
summary: "restic check failed on loki repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the loki repository.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupGrafanaCheckFailed
expr: backup_grafana_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: grafana
annotations:
summary: "restic check failed on grafana repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the grafana repository.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupK8sResourcesCheckFailed
expr: backup_k8s_resources_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: k8s-resources
annotations:
summary: "restic check failed on k8s-resources repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the k8s-resources
repository.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupPrometheusCheckFailed
expr: backup_prometheus_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
annotations:
summary: "restic check failed on prometheus repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the prometheus
repository.
Runbook: docs/monitoring/restic-restore.md
- name: backup-restic.size
interval: 60s
rules:
- alert: ResticRepoOversize
# 20 GiB = 20 * 1024^3 = 21474836480 bytes.
expr: restic_repo_size_bytes > 21474836480
for: 30m
labels:
severity: warning
service: monitoring
component: backup
annotations:
summary: "restic repository {{ $labels.repo }} exceeds 20 GiB"
description: |
Repository {{ $labels.repo }} is
{{ $value | humanize1024 }}B, above the 20 GiB
guard. Baseline is <5 GiB per repo. Investigate
retention/prune (see forget flags in the CronJob)
and dedup effectiveness.
Reference in a new issue View git blame Copy permalink