retire(monitoring): remove backup-k8s-resources CronJob + restic alerts (DEV-576)
Velero cluster-all now covers cluster-manifest backup with two successive Completed runs verified and one restore drill (openwebui) green (DEV-575). - Delete apps/monitoring/backup-k8s-resources-cronjob.yaml - Drop BackupK8sResourcesStale + BackupK8sResourcesCheckFailed from backup-restic-alerts.yaml and matching series/cases from the promtool test file (all 9 remaining rules pass `promtool test rules`) - Update apps/monitoring/README.md + prometheus-backup-cronjob comments - Runbook lives in BookStack: EN https://books.basicstack.de/books/it-operations-manual/page/velero-backup-restore-runbook DE https://books.basicstack.de/books/it-betriebshandbuch/page/velero-backup-und-restore-runbook Rollback: revert this commit — Argo restores the CronJob + alerts and restic k8s-resources runs resume at the next 02:00 UTC schedule. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO (Paperclip Agent)
parent
765962c936
commit
7368283d76
5 changed files with 11 additions and 291 deletions
|
|
@ -2,11 +2,10 @@
|
|||
|
||||
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
|
||||
|
||||
- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.19.1` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
|
||||
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
|
||||
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
|
||||
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block.
|
||||
- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the four restic repos (loki/grafana/k8s-resources/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)).
|
||||
- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos (loki/grafana/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)). The `k8s-resources` restic repo was superseded by Velero cluster-reconstruction backups and retired in [DEV-576](/DEV/issues/DEV-576).
|
||||
|
||||
## Shared SealedSecret
|
||||
|
||||
|
|
@ -18,7 +17,7 @@ All four restic CronJobs read Hetzner S3 credentials + the restic repository pas
|
|||
| `secret-key` | Hetzner Object Storage secret access key |
|
||||
| `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) |
|
||||
| `bucket` | Bucket name (single bucket, per-prefix repos) |
|
||||
| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all four repos (loki/grafana/k8s-resources/prometheus) — rotating rewrites the key file on every repo. |
|
||||
| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all three remaining repos (loki/grafana/prometheus) — rotating rewrites the key file on every repo. |
|
||||
|
||||
## Restore
|
||||
|
||||
|
|
@ -30,20 +29,20 @@ Each restic CronJob writes `/metrics/backup_<kind>.prom` (atomic — `.prom.tmp`
|
|||
|
||||
| Metric | Emitted by |
|
||||
|--------------------------------------------|----------------------------------------------------------------------------------|
|
||||
| `backup_<kind>_success` (0/1) | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` |
|
||||
| `backup_<kind>_success` (0/1) | `backup-{loki,grafana}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` |
|
||||
| `backup_<kind>_timestamp_seconds` | same |
|
||||
| `backup_<kind>_check_status` (exit code) | same — from `restic check --read-data-subset=5%` |
|
||||
| `backup_prometheus_backup_status` | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) |
|
||||
| `restic_repo_size_bytes{repo="<kind>"}` | same — from `restic stats --json --mode raw-data` (added in DEV-490) |
|
||||
|
||||
**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods, `k8s-resources` is unpinned), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.
|
||||
**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.
|
||||
|
||||
## Alerts
|
||||
|
||||
`backup-restic-alerts.yaml` defines nine alerts (all `severity: warning`):
|
||||
`backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`):
|
||||
|
||||
- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
|
||||
- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
|
||||
- `BackupLokiStale` / `BackupGrafanaStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
|
||||
- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
|
||||
- `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions.
|
||||
|
||||
All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally:
|
||||
|
|
@ -55,4 +54,4 @@ promtool check rules /tmp/backup-restic-rules.yaml
|
|||
promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
|
||||
```
|
||||
|
||||
The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` has no PVC dep and stays unpinned. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers.
|
||||
The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers. The `backup-k8s-resources` CronJob and its restic repo were retired in [DEV-576](/DEV/issues/DEV-576) after Velero `cluster-all` took over cluster-manifest coverage.
|
||||
|
|
|
|||
|
|
@ -1,205 +0,0 @@
|
|||
---
|
||||
# Kubernetes-resource backup via restic to Hetzner Object Storage
|
||||
# (DEV-487, DEV-482 Option 4). Step 4 of the Option 4 rollout.
|
||||
#
|
||||
# Streams a concatenated YAML dump of cluster-scoped and per-namespace
|
||||
# resources through `restic backup --stdin` into
|
||||
# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources`. No PVC mount
|
||||
# (drops the local-path `backup-storage` dependency), no node pin.
|
||||
#
|
||||
# Two-container pattern:
|
||||
# 1. `kubectl-dump` init container (`alpine/k8s:1.29.4`) writes
|
||||
# /dump/cluster.yaml into an emptyDir. Uses `serviceAccountName:
|
||||
# backup-sa` (unchanged from the legacy job).
|
||||
# 2. `restic` main container (`restic/restic:0.19.1`, matches the
|
||||
# loki/grafana siblings) reads that file on stdin and streams it
|
||||
# into the restic repo with `--stdin-filename cluster.yaml`.
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: backup-k8s-resources
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app: backup
|
||||
type: k8s-resources
|
||||
backend: restic
|
||||
spec:
|
||||
schedule: "0 2 * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
metadata:
|
||||
labels:
|
||||
app: backup
|
||||
type: k8s-resources
|
||||
backend: restic
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 3600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: backup
|
||||
type: k8s-resources
|
||||
backend: restic
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
serviceAccountName: backup-sa
|
||||
initContainers:
|
||||
- name: kubectl-dump
|
||||
image: alpine/k8s:1.29.4
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
echo "=== kubectl-dump started at $(date -u +%FT%TZ) ==="
|
||||
DUMP=/dump/cluster.yaml
|
||||
: > "${DUMP}"
|
||||
|
||||
echo "--- namespaces ---"
|
||||
kubectl get namespaces -o yaml >> "${DUMP}"
|
||||
echo "---" >> "${DUMP}"
|
||||
|
||||
echo "--- cluster-scoped resources ---"
|
||||
kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \
|
||||
-o yaml >> "${DUMP}"
|
||||
echo "---" >> "${DUMP}"
|
||||
|
||||
echo "--- namespaced resources ---"
|
||||
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
|
||||
echo " ns=${ns}"
|
||||
kubectl get \
|
||||
configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \
|
||||
-n "${ns}" -o yaml >> "${DUMP}" 2>/dev/null || true
|
||||
echo "---" >> "${DUMP}"
|
||||
done
|
||||
|
||||
echo "dump size: $(wc -c < ${DUMP}) bytes"
|
||||
echo "=== kubectl-dump finished at $(date -u +%FT%TZ) ==="
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
volumeMounts:
|
||||
- name: dump
|
||||
mountPath: /dump
|
||||
containers:
|
||||
- name: restic
|
||||
image: harbor.basicstack.de/library/restic:0.19.1
|
||||
# Mirrored from docker.io/restic/restic:0.19.1 (DEV-493) —
|
||||
# deterministic ingress via Harbor. Retag procedure in
|
||||
# docs/monitoring/restic-restore.md § "Tag-bump procedure".
|
||||
env:
|
||||
- name: AWS_ACCESS_KEY_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: access-key
|
||||
- name: AWS_SECRET_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: secret-key
|
||||
- name: RESTIC_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: restic-password
|
||||
- name: S3_ENDPOINT
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: endpoint
|
||||
- name: S3_BUCKET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: bucket
|
||||
- name: RESTIC_REPOSITORY
|
||||
value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/k8s-resources"
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
echo "=== backup-k8s-resources-restic started at $(date -u +%FT%TZ) ==="
|
||||
echo "Repository: ${RESTIC_REPOSITORY}"
|
||||
|
||||
# First-run tolerance: init if the repo isn't there yet.
|
||||
if restic snapshots >/dev/null 2>&1; then
|
||||
echo "Repo exists, skipping init."
|
||||
else
|
||||
echo "Repo missing, initialising..."
|
||||
restic init
|
||||
fi
|
||||
|
||||
echo "--- restic backup --stdin cluster.yaml ---"
|
||||
restic backup --stdin \
|
||||
--stdin-filename cluster.yaml \
|
||||
--tag k8s-resources \
|
||||
--host k3s < /dump/cluster.yaml
|
||||
|
||||
echo "--- restic forget/prune ---"
|
||||
restic forget --tag k8s-resources \
|
||||
--keep-daily 7 \
|
||||
--keep-weekly 4 \
|
||||
--keep-monthly 6 \
|
||||
--prune
|
||||
|
||||
echo "--- restic check --read-data-subset=5% ---"
|
||||
CHECK_STATUS=0
|
||||
restic check --read-data-subset=5% || CHECK_STATUS=$?
|
||||
echo "restic check exit: ${CHECK_STATUS}"
|
||||
|
||||
echo "--- restic stats (repo size) ---"
|
||||
REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
|
||||
| grep -oE '"total_size":[0-9]+' \
|
||||
| head -1 \
|
||||
| cut -d: -f2)
|
||||
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
|
||||
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
|
||||
|
||||
# Textfile-collector metrics; identical wiring to the
|
||||
# loki/grafana siblings. See loki cronjob for the
|
||||
# atomic-write rationale (DEV-494).
|
||||
{
|
||||
echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
|
||||
echo "backup_k8s_resources_timestamp_seconds $(date +%s)"
|
||||
echo "backup_k8s_resources_check_status ${CHECK_STATUS}"
|
||||
echo "restic_repo_size_bytes{repo=\"k8s-resources\"} ${REPO_SIZE_BYTES}"
|
||||
} > /metrics/backup_k8s_resources.prom.tmp
|
||||
mv /metrics/backup_k8s_resources.prom.tmp /metrics/backup_k8s_resources.prom
|
||||
|
||||
echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ==="
|
||||
exit ${CHECK_STATUS}
|
||||
volumeMounts:
|
||||
- name: dump
|
||||
mountPath: /dump
|
||||
readOnly: true
|
||||
- name: metrics
|
||||
mountPath: /metrics
|
||||
- name: cache
|
||||
mountPath: /root/.cache/restic
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 1500m
|
||||
memory: 1Gi
|
||||
volumes:
|
||||
- name: dump
|
||||
emptyDir: {}
|
||||
- name: metrics
|
||||
hostPath:
|
||||
# node-exporter's textfile-collector directory
|
||||
# (DEV-494). See sibling loki cronjob for detail.
|
||||
path: /var/lib/node_exporter/textfile_collector
|
||||
type: DirectoryOrCreate
|
||||
- name: cache
|
||||
emptyDir: {}
|
||||
|
|
@ -35,8 +35,6 @@ tests:
|
|||
values: '-104400x60'
|
||||
- series: 'backup_grafana_timestamp_seconds'
|
||||
values: '-104400x60'
|
||||
- series: 'backup_k8s_resources_timestamp_seconds'
|
||||
values: '-104400x60'
|
||||
- series: 'backup_prometheus_timestamp_seconds'
|
||||
values: '-104400x60'
|
||||
alert_rule_test:
|
||||
|
|
@ -70,21 +68,6 @@ tests:
|
|||
`backup_grafana_timestamp_seconds` sample in more
|
||||
than 28 hours. Expected daily at 03:15 UTC.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- eval_time: 30m
|
||||
alertname: BackupK8sResourcesStale
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
service: monitoring
|
||||
component: backup
|
||||
repo: k8s-resources
|
||||
exp_annotations:
|
||||
summary: "K8s-resources restic backup is stale (>28h)"
|
||||
description: |
|
||||
`backup-k8s-resources` has not written a fresh
|
||||
`backup_k8s_resources_timestamp_seconds` sample in
|
||||
more than 28 hours. Expected daily at 02:00 UTC.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- eval_time: 30m
|
||||
alertname: BackupPrometheusStale
|
||||
exp_alerts:
|
||||
|
|
@ -112,8 +95,6 @@ tests:
|
|||
values: '-3600x60'
|
||||
- series: 'backup_grafana_timestamp_seconds'
|
||||
values: '-3600x60'
|
||||
- series: 'backup_k8s_resources_timestamp_seconds'
|
||||
values: '-3600x60'
|
||||
- series: 'backup_prometheus_timestamp_seconds'
|
||||
values: '-3600x60'
|
||||
alert_rule_test:
|
||||
|
|
@ -123,9 +104,6 @@ tests:
|
|||
- eval_time: 30m
|
||||
alertname: BackupGrafanaStale
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: BackupK8sResourcesStale
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: BackupPrometheusStale
|
||||
exp_alerts: []
|
||||
|
|
@ -139,8 +117,6 @@ tests:
|
|||
values: '1x30'
|
||||
- series: 'backup_grafana_check_status'
|
||||
values: '2x30'
|
||||
- series: 'backup_k8s_resources_check_status'
|
||||
values: '1x30'
|
||||
- series: 'backup_prometheus_check_status'
|
||||
values: '1x30'
|
||||
alert_rule_test:
|
||||
|
|
@ -172,21 +148,6 @@ tests:
|
|||
`restic check --read-data-subset=5%` returned
|
||||
exit code 2 on the grafana repository.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- eval_time: 15m
|
||||
alertname: BackupK8sResourcesCheckFailed
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
service: monitoring
|
||||
component: backup
|
||||
repo: k8s-resources
|
||||
exp_annotations:
|
||||
summary: "restic check failed on k8s-resources repo"
|
||||
description: |
|
||||
`restic check --read-data-subset=5%` returned
|
||||
exit code 1 on the k8s-resources
|
||||
repository.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- eval_time: 15m
|
||||
alertname: BackupPrometheusCheckFailed
|
||||
exp_alerts:
|
||||
|
|
@ -212,8 +173,6 @@ tests:
|
|||
values: '0x30'
|
||||
- series: 'backup_grafana_check_status'
|
||||
values: '0x30'
|
||||
- series: 'backup_k8s_resources_check_status'
|
||||
values: '0x30'
|
||||
- series: 'backup_prometheus_check_status'
|
||||
values: '0x30'
|
||||
alert_rule_test:
|
||||
|
|
@ -223,9 +182,6 @@ tests:
|
|||
- eval_time: 15m
|
||||
alertname: BackupGrafanaCheckFailed
|
||||
exp_alerts: []
|
||||
- eval_time: 15m
|
||||
alertname: BackupK8sResourcesCheckFailed
|
||||
exp_alerts: []
|
||||
- eval_time: 15m
|
||||
alertname: BackupPrometheusCheckFailed
|
||||
exp_alerts: []
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@
|
|||
# kube-prometheus-stack node-exporter DaemonSet (DEV-494).
|
||||
#
|
||||
# Because a CronJob may run on a different worker across days (loki
|
||||
# backup follows the loki pod; k8s-resources is unpinned), stale
|
||||
# backup follows the loki pod), stale
|
||||
# `.prom` files can linger on nodes the job has since left. That
|
||||
# would leave a per-node series with an old timestamp/check-status
|
||||
# indefinitely. The alerts below aggregate across instances so a
|
||||
|
|
@ -74,21 +74,6 @@ spec:
|
|||
`backup_grafana_timestamp_seconds` sample in more
|
||||
than 28 hours. Expected daily at 03:15 UTC.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- alert: BackupK8sResourcesStale
|
||||
expr: time() - max(backup_k8s_resources_timestamp_seconds) > 28 * 3600
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
service: monitoring
|
||||
component: backup
|
||||
repo: k8s-resources
|
||||
annotations:
|
||||
summary: "K8s-resources restic backup is stale (>28h)"
|
||||
description: |
|
||||
`backup-k8s-resources` has not written a fresh
|
||||
`backup_k8s_resources_timestamp_seconds` sample in
|
||||
more than 28 hours. Expected daily at 02:00 UTC.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- alert: BackupPrometheusStale
|
||||
expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600
|
||||
for: 15m
|
||||
|
|
@ -150,21 +135,6 @@ spec:
|
|||
`restic check --read-data-subset=5%` returned
|
||||
exit code {{ $value }} on the grafana repository.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- alert: BackupK8sResourcesCheckFailed
|
||||
expr: backup_k8s_resources_check_status != 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
service: monitoring
|
||||
component: backup
|
||||
repo: k8s-resources
|
||||
annotations:
|
||||
summary: "restic check failed on k8s-resources repo"
|
||||
description: |
|
||||
`restic check --read-data-subset=5%` returned
|
||||
exit code {{ $value }} on the k8s-resources
|
||||
repository.
|
||||
Runbook: docs/monitoring/restic-restore.md
|
||||
- alert: BackupPrometheusCheckFailed
|
||||
expr: backup_prometheus_check_status != 0
|
||||
for: 5m
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ metadata:
|
|||
type: prometheus
|
||||
backend: restic
|
||||
spec:
|
||||
schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana/k8s-resources
|
||||
schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 3
|
||||
|
|
@ -173,7 +173,7 @@ spec:
|
|||
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
|
||||
|
||||
# Textfile-collector metrics. Same hostPath pattern as
|
||||
# the loki/grafana/k8s-resources siblings (DEV-494) —
|
||||
# the loki/grafana siblings (DEV-494) —
|
||||
# written atomically via `.tmp` + rename so a mid-write
|
||||
# read never surfaces a truncated sample.
|
||||
# backup_prometheus_success rolls in both stages: the
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue