New freshness (BackupForgejoStale, 28h) and integrity
(BackupForgejoCheckFailed) rules matching the sibling
loki/grafana/k8s-resources/prometheus alerts.
Metrics come from backup_forgejo_{timestamp_seconds,check_status,success}
emitted by apps/forgejo/forgejo-backup-cronjob.yaml.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- New PrometheusRule apps/monitoring/backup-restic-alerts.yaml with
seven warning-level rules:
* BackupLokiStale / BackupGrafanaStale / BackupK8sResourcesStale
-> time() - backup_<kind>_timestamp_seconds > 28h
* BackupLokiCheckFailed / BackupGrafanaCheckFailed /
BackupK8sResourcesCheckFailed -> backup_<kind>_check_status != 0
* ResticRepoOversize -> restic_repo_size_bytes > 20 GiB (per-repo)
- Emit restic_repo_size_bytes{repo="<kind>"} from all three restic
CronJobs (loki, grafana, k8s-resources) via `restic stats --json
--mode raw-data` so ResticRepoOversize has data to match once the
textfile-collector scrape path is wired.
- Companion promtool unit test backup-restic-alerts.test.yaml with
five scenarios (fresh/stale, check pass/fail, oversize) -- verified
locally with promtool 2.53.1: SUCCESS.
- README.md: document the three restic CronJobs, the shared
SealedSecret keys (access-key/secret-key/endpoint/bucket/
restic-password), the emitted textfile metrics, and the alert list.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
