chore(monitoring): add restic alerts for forgejo repo (DEV-514)
New freshness (BackupForgejoStale, 28h) and integrity
(BackupForgejoCheckFailed) rules matching the sibling
loki/grafana/k8s-resources/prometheus alerts.
Metrics come from backup_forgejo_{timestamp_seconds,check_status,success}
emitted by apps/forgejo/forgejo-backup-cronjob.yaml.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
parent
585d6ee114
commit
89cf3094d0
1 changed files with 30 additions and 0 deletions
|
|
@ -104,6 +104,21 @@ spec:
|
||||||
`backup_prometheus_timestamp_seconds` sample in more
|
`backup_prometheus_timestamp_seconds` sample in more
|
||||||
than 28 hours. Expected daily at 03:30 UTC.
|
than 28 hours. Expected daily at 03:30 UTC.
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
Runbook: docs/monitoring/restic-restore.md
|
||||||
|
- alert: BackupForgejoStale
|
||||||
|
expr: time() - max(backup_forgejo_timestamp_seconds) > 28 * 3600
|
||||||
|
for: 15m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
service: monitoring
|
||||||
|
component: backup
|
||||||
|
repo: forgejo
|
||||||
|
annotations:
|
||||||
|
summary: "Forgejo restic backup is stale (>28h)"
|
||||||
|
description: |
|
||||||
|
`forgejo-backup` has not written a fresh
|
||||||
|
`backup_forgejo_timestamp_seconds` sample in more
|
||||||
|
than 28 hours. Expected daily at 03:00 UTC.
|
||||||
|
Runbook: docs/monitoring/restic-restore.md
|
||||||
- name: backup-restic.integrity
|
- name: backup-restic.integrity
|
||||||
interval: 60s
|
interval: 60s
|
||||||
rules:
|
rules:
|
||||||
|
|
@ -165,6 +180,21 @@ spec:
|
||||||
exit code {{ $value }} on the prometheus
|
exit code {{ $value }} on the prometheus
|
||||||
repository.
|
repository.
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
Runbook: docs/monitoring/restic-restore.md
|
||||||
|
- alert: BackupForgejoCheckFailed
|
||||||
|
expr: backup_forgejo_check_status != 0
|
||||||
|
for: 5m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
service: monitoring
|
||||||
|
component: backup
|
||||||
|
repo: forgejo
|
||||||
|
annotations:
|
||||||
|
summary: "restic check failed on forgejo repo"
|
||||||
|
description: |
|
||||||
|
`restic check --read-data-subset=5%` returned
|
||||||
|
exit code {{ $value }} on the forgejo
|
||||||
|
repository.
|
||||||
|
Runbook: docs/monitoring/restic-restore.md
|
||||||
- name: backup-restic.size
|
- name: backup-restic.size
|
||||||
interval: 60s
|
interval: 60s
|
||||||
rules:
|
rules:
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue