chore(monitoring): add restic alerts for forgejo repo (DEV-514)
New freshness (BackupForgejoStale, 28h) and integrity
(BackupForgejoCheckFailed) rules matching the sibling
loki/grafana/k8s-resources/prometheus alerts.
Metrics come from backup_forgejo_{timestamp_seconds,check_status,success}
emitted by apps/forgejo/forgejo-backup-cronjob.yaml.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
parent
585d6ee114
commit
89cf3094d0
1 changed files with 30 additions and 0 deletions
|
|
@ -104,6 +104,21 @@ spec:
|
|||
`backup_prometheus_timestamp_seconds` sample in more
|
||||
than 28 hours. Expected daily at 03:30 UTC.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- alert: BackupForgejoStale
|
||||
expr: time() - max(backup_forgejo_timestamp_seconds) > 28 * 3600
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
service: monitoring
|
||||
component: backup
|
||||
repo: forgejo
|
||||
annotations:
|
||||
summary: "Forgejo restic backup is stale (>28h)"
|
||||
description: |
|
||||
`forgejo-backup` has not written a fresh
|
||||
`backup_forgejo_timestamp_seconds` sample in more
|
||||
than 28 hours. Expected daily at 03:00 UTC.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- name: backup-restic.integrity
|
||||
interval: 60s
|
||||
rules:
|
||||
|
|
@ -165,6 +180,21 @@ spec:
|
|||
exit code {{ $value }} on the prometheus
|
||||
repository.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- alert: BackupForgejoCheckFailed
|
||||
expr: backup_forgejo_check_status != 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
service: monitoring
|
||||
component: backup
|
||||
repo: forgejo
|
||||
annotations:
|
||||
summary: "restic check failed on forgejo repo"
|
||||
description: |
|
||||
`restic check --read-data-subset=5%` returned
|
||||
exit code {{ $value }} on the forgejo
|
||||
repository.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- name: backup-restic.size
|
||||
interval: 60s
|
||||
rules:
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue