retire(monitoring): remove backup-k8s-resources CronJob + restic alerts (DEV-576)

Velero cluster-all now covers cluster-manifest backup with two successive
Completed runs verified and one restore drill (openwebui) green (DEV-575).

- Delete apps/monitoring/backup-k8s-resources-cronjob.yaml
- Drop BackupK8sResourcesStale + BackupK8sResourcesCheckFailed from
  backup-restic-alerts.yaml and matching series/cases from the promtool
  test file (all 9 remaining rules pass `promtool test rules`)
- Update apps/monitoring/README.md + prometheus-backup-cronjob comments
- Runbook lives in BookStack: EN
  https://books.basicstack.de/books/it-operations-manual/page/velero-backup-restore-runbook
  DE
  https://books.basicstack.de/books/it-betriebshandbuch/page/velero-backup-und-restore-runbook

Rollback: revert this commit — Argo restores the CronJob + alerts and
restic k8s-resources runs resume at the next 02:00 UTC schedule.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO (Paperclip Agent) 2026-09-12 18:24:00 +00:00
parent 765962c936
commit 7368283d76
5 changed files with 11 additions and 291 deletions

View file

@ -2,11 +2,10 @@
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context. Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.19.1` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)). - `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)). - `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block. - `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block.
- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the four restic repos (loki/grafana/k8s-resources/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)). - `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos (loki/grafana/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)). The `k8s-resources` restic repo was superseded by Velero cluster-reconstruction backups and retired in [DEV-576](/DEV/issues/DEV-576).
## Shared SealedSecret ## Shared SealedSecret
@ -18,7 +17,7 @@ All four restic CronJobs read Hetzner S3 credentials + the restic repository pas
| `secret-key` | Hetzner Object Storage secret access key | | `secret-key` | Hetzner Object Storage secret access key |
| `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) | | `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) |
| `bucket` | Bucket name (single bucket, per-prefix repos) | | `bucket` | Bucket name (single bucket, per-prefix repos) |
| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all four repos (loki/grafana/k8s-resources/prometheus) — rotating rewrites the key file on every repo. | | `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all three remaining repos (loki/grafana/prometheus) — rotating rewrites the key file on every repo. |
## Restore ## Restore
@ -30,20 +29,20 @@ Each restic CronJob writes `/metrics/backup_<kind>.prom` (atomic — `.prom.tmp`
| Metric | Emitted by | | Metric | Emitted by |
|--------------------------------------------|----------------------------------------------------------------------------------| |--------------------------------------------|----------------------------------------------------------------------------------|
| `backup_<kind>_success` (0/1) | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` | | `backup_<kind>_success` (0/1) | `backup-{loki,grafana}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` |
| `backup_<kind>_timestamp_seconds` | same | | `backup_<kind>_timestamp_seconds` | same |
| `backup_<kind>_check_status` (exit code) | same — from `restic check --read-data-subset=5%` | | `backup_<kind>_check_status` (exit code) | same — from `restic check --read-data-subset=5%` |
| `backup_prometheus_backup_status` | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) | | `backup_prometheus_backup_status` | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) |
| `restic_repo_size_bytes{repo="<kind>"}` | same — from `restic stats --json --mode raw-data` (added in DEV-490) | | `restic_repo_size_bytes{repo="<kind>"}` | same — from `restic stats --json --mode raw-data` (added in DEV-490) |
**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods, `k8s-resources` is unpinned), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node. **Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.
## Alerts ## Alerts
`backup-restic-alerts.yaml` defines nine alerts (all `severity: warning`): `backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`):
- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire. - `BackupLokiStale` / `BackupGrafanaStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`. - `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
- `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions. - `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions.
All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally: All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally:
@ -55,4 +54,4 @@ promtool check rules /tmp/backup-restic-rules.yaml
promtool test rules apps/monitoring/backup-restic-alerts.test.yaml promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
``` ```
The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` has no PVC dep and stays unpinned. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers. The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers. The `backup-k8s-resources` CronJob and its restic repo were retired in [DEV-576](/DEV/issues/DEV-576) after Velero `cluster-all` took over cluster-manifest coverage.

View file

@ -1,205 +0,0 @@
---
# Kubernetes-resource backup via restic to Hetzner Object Storage
# (DEV-487, DEV-482 Option 4). Step 4 of the Option 4 rollout.
#
# Streams a concatenated YAML dump of cluster-scoped and per-namespace
# resources through `restic backup --stdin` into
# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources`. No PVC mount
# (drops the local-path `backup-storage` dependency), no node pin.
#
# Two-container pattern:
# 1. `kubectl-dump` init container (`alpine/k8s:1.29.4`) writes
# /dump/cluster.yaml into an emptyDir. Uses `serviceAccountName:
# backup-sa` (unchanged from the legacy job).
# 2. `restic` main container (`restic/restic:0.19.1`, matches the
# loki/grafana siblings) reads that file on stdin and streams it
# into the restic repo with `--stdin-filename cluster.yaml`.
apiVersion: batch/v1
kind: CronJob
metadata:
name: backup-k8s-resources
namespace: monitoring
labels:
app: backup
type: k8s-resources
backend: restic
spec:
schedule: "0 2 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
metadata:
labels:
app: backup
type: k8s-resources
backend: restic
spec:
backoffLimit: 2
activeDeadlineSeconds: 3600
template:
metadata:
labels:
app: backup
type: k8s-resources
backend: restic
spec:
restartPolicy: OnFailure
serviceAccountName: backup-sa
initContainers:
- name: kubectl-dump
image: alpine/k8s:1.29.4
command:
- /bin/sh
- -c
- |
set -eu
echo "=== kubectl-dump started at $(date -u +%FT%TZ) ==="
DUMP=/dump/cluster.yaml
: > "${DUMP}"
echo "--- namespaces ---"
kubectl get namespaces -o yaml >> "${DUMP}"
echo "---" >> "${DUMP}"
echo "--- cluster-scoped resources ---"
kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \
-o yaml >> "${DUMP}"
echo "---" >> "${DUMP}"
echo "--- namespaced resources ---"
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
echo " ns=${ns}"
kubectl get \
configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \
-n "${ns}" -o yaml >> "${DUMP}" 2>/dev/null || true
echo "---" >> "${DUMP}"
done
echo "dump size: $(wc -c < ${DUMP}) bytes"
echo "=== kubectl-dump finished at $(date -u +%FT%TZ) ==="
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
volumeMounts:
- name: dump
mountPath: /dump
containers:
- name: restic
image: harbor.basicstack.de/library/restic:0.19.1
# Mirrored from docker.io/restic/restic:0.19.1 (DEV-493) —
# deterministic ingress via Harbor. Retag procedure in
# docs/monitoring/restic-restore.md § "Tag-bump procedure".
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: access-key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: secret-key
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: restic-password
- name: S3_ENDPOINT
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: endpoint
- name: S3_BUCKET
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: bucket
- name: RESTIC_REPOSITORY
value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/k8s-resources"
command:
- /bin/sh
- -c
- |
set -eu
echo "=== backup-k8s-resources-restic started at $(date -u +%FT%TZ) ==="
echo "Repository: ${RESTIC_REPOSITORY}"
# First-run tolerance: init if the repo isn't there yet.
if restic snapshots >/dev/null 2>&1; then
echo "Repo exists, skipping init."
else
echo "Repo missing, initialising..."
restic init
fi
echo "--- restic backup --stdin cluster.yaml ---"
restic backup --stdin \
--stdin-filename cluster.yaml \
--tag k8s-resources \
--host k3s < /dump/cluster.yaml
echo "--- restic forget/prune ---"
restic forget --tag k8s-resources \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
--prune
echo "--- restic check --read-data-subset=5% ---"
CHECK_STATUS=0
restic check --read-data-subset=5% || CHECK_STATUS=$?
echo "restic check exit: ${CHECK_STATUS}"
echo "--- restic stats (repo size) ---"
REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
| grep -oE '"total_size":[0-9]+' \
| head -1 \
| cut -d: -f2)
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics; identical wiring to the
# loki/grafana siblings. See loki cronjob for the
# atomic-write rationale (DEV-494).
{
echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
echo "backup_k8s_resources_timestamp_seconds $(date +%s)"
echo "backup_k8s_resources_check_status ${CHECK_STATUS}"
echo "restic_repo_size_bytes{repo=\"k8s-resources\"} ${REPO_SIZE_BYTES}"
} > /metrics/backup_k8s_resources.prom.tmp
mv /metrics/backup_k8s_resources.prom.tmp /metrics/backup_k8s_resources.prom
echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ==="
exit ${CHECK_STATUS}
volumeMounts:
- name: dump
mountPath: /dump
readOnly: true
- name: metrics
mountPath: /metrics
- name: cache
mountPath: /root/.cache/restic
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 1500m
memory: 1Gi
volumes:
- name: dump
emptyDir: {}
- name: metrics
hostPath:
# node-exporter's textfile-collector directory
# (DEV-494). See sibling loki cronjob for detail.
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
- name: cache
emptyDir: {}

View file

@ -35,8 +35,6 @@ tests:
values: '-104400x60' values: '-104400x60'
- series: 'backup_grafana_timestamp_seconds' - series: 'backup_grafana_timestamp_seconds'
values: '-104400x60' values: '-104400x60'
- series: 'backup_k8s_resources_timestamp_seconds'
values: '-104400x60'
- series: 'backup_prometheus_timestamp_seconds' - series: 'backup_prometheus_timestamp_seconds'
values: '-104400x60' values: '-104400x60'
alert_rule_test: alert_rule_test:
@ -70,21 +68,6 @@ tests:
`backup_grafana_timestamp_seconds` sample in more `backup_grafana_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:15 UTC. than 28 hours. Expected daily at 03:15 UTC.
Runbook: docs/monitoring/restic-restore.md Runbook: docs/monitoring/restic-restore.md
- eval_time: 30m
alertname: BackupK8sResourcesStale
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: k8s-resources
exp_annotations:
summary: "K8s-resources restic backup is stale (>28h)"
description: |
`backup-k8s-resources` has not written a fresh
`backup_k8s_resources_timestamp_seconds` sample in
more than 28 hours. Expected daily at 02:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- eval_time: 30m - eval_time: 30m
alertname: BackupPrometheusStale alertname: BackupPrometheusStale
exp_alerts: exp_alerts:
@ -112,8 +95,6 @@ tests:
values: '-3600x60' values: '-3600x60'
- series: 'backup_grafana_timestamp_seconds' - series: 'backup_grafana_timestamp_seconds'
values: '-3600x60' values: '-3600x60'
- series: 'backup_k8s_resources_timestamp_seconds'
values: '-3600x60'
- series: 'backup_prometheus_timestamp_seconds' - series: 'backup_prometheus_timestamp_seconds'
values: '-3600x60' values: '-3600x60'
alert_rule_test: alert_rule_test:
@ -123,9 +104,6 @@ tests:
- eval_time: 30m - eval_time: 30m
alertname: BackupGrafanaStale alertname: BackupGrafanaStale
exp_alerts: [] exp_alerts: []
- eval_time: 30m
alertname: BackupK8sResourcesStale
exp_alerts: []
- eval_time: 30m - eval_time: 30m
alertname: BackupPrometheusStale alertname: BackupPrometheusStale
exp_alerts: [] exp_alerts: []
@ -139,8 +117,6 @@ tests:
values: '1x30' values: '1x30'
- series: 'backup_grafana_check_status' - series: 'backup_grafana_check_status'
values: '2x30' values: '2x30'
- series: 'backup_k8s_resources_check_status'
values: '1x30'
- series: 'backup_prometheus_check_status' - series: 'backup_prometheus_check_status'
values: '1x30' values: '1x30'
alert_rule_test: alert_rule_test:
@ -172,21 +148,6 @@ tests:
`restic check --read-data-subset=5%` returned `restic check --read-data-subset=5%` returned
exit code 2 on the grafana repository. exit code 2 on the grafana repository.
Runbook: docs/monitoring/restic-restore.md Runbook: docs/monitoring/restic-restore.md
- eval_time: 15m
alertname: BackupK8sResourcesCheckFailed
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: k8s-resources
exp_annotations:
summary: "restic check failed on k8s-resources repo"
description: |
`restic check --read-data-subset=5%` returned
exit code 1 on the k8s-resources
repository.
Runbook: docs/monitoring/restic-restore.md
- eval_time: 15m - eval_time: 15m
alertname: BackupPrometheusCheckFailed alertname: BackupPrometheusCheckFailed
exp_alerts: exp_alerts:
@ -212,8 +173,6 @@ tests:
values: '0x30' values: '0x30'
- series: 'backup_grafana_check_status' - series: 'backup_grafana_check_status'
values: '0x30' values: '0x30'
- series: 'backup_k8s_resources_check_status'
values: '0x30'
- series: 'backup_prometheus_check_status' - series: 'backup_prometheus_check_status'
values: '0x30' values: '0x30'
alert_rule_test: alert_rule_test:
@ -223,9 +182,6 @@ tests:
- eval_time: 15m - eval_time: 15m
alertname: BackupGrafanaCheckFailed alertname: BackupGrafanaCheckFailed
exp_alerts: [] exp_alerts: []
- eval_time: 15m
alertname: BackupK8sResourcesCheckFailed
exp_alerts: []
- eval_time: 15m - eval_time: 15m
alertname: BackupPrometheusCheckFailed alertname: BackupPrometheusCheckFailed
exp_alerts: [] exp_alerts: []

View file

@ -19,7 +19,7 @@
# kube-prometheus-stack node-exporter DaemonSet (DEV-494). # kube-prometheus-stack node-exporter DaemonSet (DEV-494).
# #
# Because a CronJob may run on a different worker across days (loki # Because a CronJob may run on a different worker across days (loki
# backup follows the loki pod; k8s-resources is unpinned), stale # backup follows the loki pod), stale
# `.prom` files can linger on nodes the job has since left. That # `.prom` files can linger on nodes the job has since left. That
# would leave a per-node series with an old timestamp/check-status # would leave a per-node series with an old timestamp/check-status
# indefinitely. The alerts below aggregate across instances so a # indefinitely. The alerts below aggregate across instances so a
@ -74,21 +74,6 @@ spec:
`backup_grafana_timestamp_seconds` sample in more `backup_grafana_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:15 UTC. than 28 hours. Expected daily at 03:15 UTC.
Runbook: docs/monitoring/restic-restore.md Runbook: docs/monitoring/restic-restore.md
- alert: BackupK8sResourcesStale
expr: time() - max(backup_k8s_resources_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: k8s-resources
annotations:
summary: "K8s-resources restic backup is stale (>28h)"
description: |
`backup-k8s-resources` has not written a fresh
`backup_k8s_resources_timestamp_seconds` sample in
more than 28 hours. Expected daily at 02:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupPrometheusStale - alert: BackupPrometheusStale
expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600 expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600
for: 15m for: 15m
@ -150,21 +135,6 @@ spec:
`restic check --read-data-subset=5%` returned `restic check --read-data-subset=5%` returned
exit code {{ $value }} on the grafana repository. exit code {{ $value }} on the grafana repository.
Runbook: docs/monitoring/restic-restore.md Runbook: docs/monitoring/restic-restore.md
- alert: BackupK8sResourcesCheckFailed
expr: backup_k8s_resources_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: k8s-resources
annotations:
summary: "restic check failed on k8s-resources repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the k8s-resources
repository.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupPrometheusCheckFailed - alert: BackupPrometheusCheckFailed
expr: backup_prometheus_check_status != 0 expr: backup_prometheus_check_status != 0
for: 5m for: 5m

View file

@ -48,7 +48,7 @@ metadata:
type: prometheus type: prometheus
backend: restic backend: restic
spec: spec:
schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana/k8s-resources schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana
concurrencyPolicy: Forbid concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3 successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3 failedJobsHistoryLimit: 3
@ -173,7 +173,7 @@ spec:
echo "restic repo size: ${REPO_SIZE_BYTES} bytes" echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics. Same hostPath pattern as # Textfile-collector metrics. Same hostPath pattern as
# the loki/grafana/k8s-resources siblings (DEV-494) — # the loki/grafana siblings (DEV-494) —
# written atomically via `.tmp` + rename so a mid-write # written atomically via `.tmp` + rename so a mid-write
# read never surfaces a truncated sample. # read never surfaces a truncated sample.
# backup_prometheus_success rolls in both stages: the # backup_prometheus_success rolls in both stages: the