retire(monitoring): remove backup-k8s-resources CronJob + restic alerts (DEV-576)
Velero cluster-all now covers cluster-manifest backup with two successive Completed runs verified and one restore drill (openwebui) green (DEV-575). - Delete apps/monitoring/backup-k8s-resources-cronjob.yaml - Drop BackupK8sResourcesStale + BackupK8sResourcesCheckFailed from backup-restic-alerts.yaml and matching series/cases from the promtool test file (all 9 remaining rules pass `promtool test rules`) - Update apps/monitoring/README.md + prometheus-backup-cronjob comments - Runbook lives in BookStack: EN https://books.basicstack.de/books/it-operations-manual/page/velero-backup-restore-runbook DE https://books.basicstack.de/books/it-betriebshandbuch/page/velero-backup-und-restore-runbook Rollback: revert this commit — Argo restores the CronJob + alerts and restic k8s-resources runs resume at the next 02:00 UTC schedule. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
parent
765962c936
commit
7368283d76
5 changed files with 11 additions and 291 deletions
|
|
@ -2,11 +2,10 @@
|
||||||
|
|
||||||
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
|
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
|
||||||
|
|
||||||
- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.19.1` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
|
|
||||||
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
|
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
|
||||||
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
|
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
|
||||||
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block.
|
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block.
|
||||||
- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the four restic repos (loki/grafana/k8s-resources/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)).
|
- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos (loki/grafana/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)). The `k8s-resources` restic repo was superseded by Velero cluster-reconstruction backups and retired in [DEV-576](/DEV/issues/DEV-576).
|
||||||
|
|
||||||
## Shared SealedSecret
|
## Shared SealedSecret
|
||||||
|
|
||||||
|
|
@ -18,7 +17,7 @@ All four restic CronJobs read Hetzner S3 credentials + the restic repository pas
|
||||||
| `secret-key` | Hetzner Object Storage secret access key |
|
| `secret-key` | Hetzner Object Storage secret access key |
|
||||||
| `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) |
|
| `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) |
|
||||||
| `bucket` | Bucket name (single bucket, per-prefix repos) |
|
| `bucket` | Bucket name (single bucket, per-prefix repos) |
|
||||||
| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all four repos (loki/grafana/k8s-resources/prometheus) — rotating rewrites the key file on every repo. |
|
| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all three remaining repos (loki/grafana/prometheus) — rotating rewrites the key file on every repo. |
|
||||||
|
|
||||||
## Restore
|
## Restore
|
||||||
|
|
||||||
|
|
@ -30,20 +29,20 @@ Each restic CronJob writes `/metrics/backup_<kind>.prom` (atomic — `.prom.tmp`
|
||||||
|
|
||||||
| Metric | Emitted by |
|
| Metric | Emitted by |
|
||||||
|--------------------------------------------|----------------------------------------------------------------------------------|
|
|--------------------------------------------|----------------------------------------------------------------------------------|
|
||||||
| `backup_<kind>_success` (0/1) | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` |
|
| `backup_<kind>_success` (0/1) | `backup-{loki,grafana}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` |
|
||||||
| `backup_<kind>_timestamp_seconds` | same |
|
| `backup_<kind>_timestamp_seconds` | same |
|
||||||
| `backup_<kind>_check_status` (exit code) | same — from `restic check --read-data-subset=5%` |
|
| `backup_<kind>_check_status` (exit code) | same — from `restic check --read-data-subset=5%` |
|
||||||
| `backup_prometheus_backup_status` | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) |
|
| `backup_prometheus_backup_status` | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) |
|
||||||
| `restic_repo_size_bytes{repo="<kind>"}` | same — from `restic stats --json --mode raw-data` (added in DEV-490) |
|
| `restic_repo_size_bytes{repo="<kind>"}` | same — from `restic stats --json --mode raw-data` (added in DEV-490) |
|
||||||
|
|
||||||
**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods, `k8s-resources` is unpinned), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.
|
**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.
|
||||||
|
|
||||||
## Alerts
|
## Alerts
|
||||||
|
|
||||||
`backup-restic-alerts.yaml` defines nine alerts (all `severity: warning`):
|
`backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`):
|
||||||
|
|
||||||
- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
|
- `BackupLokiStale` / `BackupGrafanaStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
|
||||||
- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
|
- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
|
||||||
- `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions.
|
- `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions.
|
||||||
|
|
||||||
All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally:
|
All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally:
|
||||||
|
|
@ -55,4 +54,4 @@ promtool check rules /tmp/backup-restic-rules.yaml
|
||||||
promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
|
promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` has no PVC dep and stays unpinned. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers.
|
The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers. The `backup-k8s-resources` CronJob and its restic repo were retired in [DEV-576](/DEV/issues/DEV-576) after Velero `cluster-all` took over cluster-manifest coverage.
|
||||||
|
|
|
||||||
|
|
@ -1,205 +0,0 @@
|
||||||
---
|
|
||||||
# Kubernetes-resource backup via restic to Hetzner Object Storage
|
|
||||||
# (DEV-487, DEV-482 Option 4). Step 4 of the Option 4 rollout.
|
|
||||||
#
|
|
||||||
# Streams a concatenated YAML dump of cluster-scoped and per-namespace
|
|
||||||
# resources through `restic backup --stdin` into
|
|
||||||
# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources`. No PVC mount
|
|
||||||
# (drops the local-path `backup-storage` dependency), no node pin.
|
|
||||||
#
|
|
||||||
# Two-container pattern:
|
|
||||||
# 1. `kubectl-dump` init container (`alpine/k8s:1.29.4`) writes
|
|
||||||
# /dump/cluster.yaml into an emptyDir. Uses `serviceAccountName:
|
|
||||||
# backup-sa` (unchanged from the legacy job).
|
|
||||||
# 2. `restic` main container (`restic/restic:0.19.1`, matches the
|
|
||||||
# loki/grafana siblings) reads that file on stdin and streams it
|
|
||||||
# into the restic repo with `--stdin-filename cluster.yaml`.
|
|
||||||
apiVersion: batch/v1
|
|
||||||
kind: CronJob
|
|
||||||
metadata:
|
|
||||||
name: backup-k8s-resources
|
|
||||||
namespace: monitoring
|
|
||||||
labels:
|
|
||||||
app: backup
|
|
||||||
type: k8s-resources
|
|
||||||
backend: restic
|
|
||||||
spec:
|
|
||||||
schedule: "0 2 * * *"
|
|
||||||
concurrencyPolicy: Forbid
|
|
||||||
successfulJobsHistoryLimit: 3
|
|
||||||
failedJobsHistoryLimit: 3
|
|
||||||
jobTemplate:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app: backup
|
|
||||||
type: k8s-resources
|
|
||||||
backend: restic
|
|
||||||
spec:
|
|
||||||
backoffLimit: 2
|
|
||||||
activeDeadlineSeconds: 3600
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app: backup
|
|
||||||
type: k8s-resources
|
|
||||||
backend: restic
|
|
||||||
spec:
|
|
||||||
restartPolicy: OnFailure
|
|
||||||
serviceAccountName: backup-sa
|
|
||||||
initContainers:
|
|
||||||
- name: kubectl-dump
|
|
||||||
image: alpine/k8s:1.29.4
|
|
||||||
command:
|
|
||||||
- /bin/sh
|
|
||||||
- -c
|
|
||||||
- |
|
|
||||||
set -eu
|
|
||||||
echo "=== kubectl-dump started at $(date -u +%FT%TZ) ==="
|
|
||||||
DUMP=/dump/cluster.yaml
|
|
||||||
: > "${DUMP}"
|
|
||||||
|
|
||||||
echo "--- namespaces ---"
|
|
||||||
kubectl get namespaces -o yaml >> "${DUMP}"
|
|
||||||
echo "---" >> "${DUMP}"
|
|
||||||
|
|
||||||
echo "--- cluster-scoped resources ---"
|
|
||||||
kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \
|
|
||||||
-o yaml >> "${DUMP}"
|
|
||||||
echo "---" >> "${DUMP}"
|
|
||||||
|
|
||||||
echo "--- namespaced resources ---"
|
|
||||||
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
|
|
||||||
echo " ns=${ns}"
|
|
||||||
kubectl get \
|
|
||||||
configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \
|
|
||||||
-n "${ns}" -o yaml >> "${DUMP}" 2>/dev/null || true
|
|
||||||
echo "---" >> "${DUMP}"
|
|
||||||
done
|
|
||||||
|
|
||||||
echo "dump size: $(wc -c < ${DUMP}) bytes"
|
|
||||||
echo "=== kubectl-dump finished at $(date -u +%FT%TZ) ==="
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 50m
|
|
||||||
memory: 128Mi
|
|
||||||
limits:
|
|
||||||
cpu: 500m
|
|
||||||
memory: 512Mi
|
|
||||||
volumeMounts:
|
|
||||||
- name: dump
|
|
||||||
mountPath: /dump
|
|
||||||
containers:
|
|
||||||
- name: restic
|
|
||||||
image: harbor.basicstack.de/library/restic:0.19.1
|
|
||||||
# Mirrored from docker.io/restic/restic:0.19.1 (DEV-493) —
|
|
||||||
# deterministic ingress via Harbor. Retag procedure in
|
|
||||||
# docs/monitoring/restic-restore.md § "Tag-bump procedure".
|
|
||||||
env:
|
|
||||||
- name: AWS_ACCESS_KEY_ID
|
|
||||||
valueFrom:
|
|
||||||
secretKeyRef:
|
|
||||||
name: monitoring-s3-backup
|
|
||||||
key: access-key
|
|
||||||
- name: AWS_SECRET_ACCESS_KEY
|
|
||||||
valueFrom:
|
|
||||||
secretKeyRef:
|
|
||||||
name: monitoring-s3-backup
|
|
||||||
key: secret-key
|
|
||||||
- name: RESTIC_PASSWORD
|
|
||||||
valueFrom:
|
|
||||||
secretKeyRef:
|
|
||||||
name: monitoring-s3-backup
|
|
||||||
key: restic-password
|
|
||||||
- name: S3_ENDPOINT
|
|
||||||
valueFrom:
|
|
||||||
secretKeyRef:
|
|
||||||
name: monitoring-s3-backup
|
|
||||||
key: endpoint
|
|
||||||
- name: S3_BUCKET
|
|
||||||
valueFrom:
|
|
||||||
secretKeyRef:
|
|
||||||
name: monitoring-s3-backup
|
|
||||||
key: bucket
|
|
||||||
- name: RESTIC_REPOSITORY
|
|
||||||
value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/k8s-resources"
|
|
||||||
command:
|
|
||||||
- /bin/sh
|
|
||||||
- -c
|
|
||||||
- |
|
|
||||||
set -eu
|
|
||||||
echo "=== backup-k8s-resources-restic started at $(date -u +%FT%TZ) ==="
|
|
||||||
echo "Repository: ${RESTIC_REPOSITORY}"
|
|
||||||
|
|
||||||
# First-run tolerance: init if the repo isn't there yet.
|
|
||||||
if restic snapshots >/dev/null 2>&1; then
|
|
||||||
echo "Repo exists, skipping init."
|
|
||||||
else
|
|
||||||
echo "Repo missing, initialising..."
|
|
||||||
restic init
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "--- restic backup --stdin cluster.yaml ---"
|
|
||||||
restic backup --stdin \
|
|
||||||
--stdin-filename cluster.yaml \
|
|
||||||
--tag k8s-resources \
|
|
||||||
--host k3s < /dump/cluster.yaml
|
|
||||||
|
|
||||||
echo "--- restic forget/prune ---"
|
|
||||||
restic forget --tag k8s-resources \
|
|
||||||
--keep-daily 7 \
|
|
||||||
--keep-weekly 4 \
|
|
||||||
--keep-monthly 6 \
|
|
||||||
--prune
|
|
||||||
|
|
||||||
echo "--- restic check --read-data-subset=5% ---"
|
|
||||||
CHECK_STATUS=0
|
|
||||||
restic check --read-data-subset=5% || CHECK_STATUS=$?
|
|
||||||
echo "restic check exit: ${CHECK_STATUS}"
|
|
||||||
|
|
||||||
echo "--- restic stats (repo size) ---"
|
|
||||||
REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
|
|
||||||
| grep -oE '"total_size":[0-9]+' \
|
|
||||||
| head -1 \
|
|
||||||
| cut -d: -f2)
|
|
||||||
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
|
|
||||||
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
|
|
||||||
|
|
||||||
# Textfile-collector metrics; identical wiring to the
|
|
||||||
# loki/grafana siblings. See loki cronjob for the
|
|
||||||
# atomic-write rationale (DEV-494).
|
|
||||||
{
|
|
||||||
echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
|
|
||||||
echo "backup_k8s_resources_timestamp_seconds $(date +%s)"
|
|
||||||
echo "backup_k8s_resources_check_status ${CHECK_STATUS}"
|
|
||||||
echo "restic_repo_size_bytes{repo=\"k8s-resources\"} ${REPO_SIZE_BYTES}"
|
|
||||||
} > /metrics/backup_k8s_resources.prom.tmp
|
|
||||||
mv /metrics/backup_k8s_resources.prom.tmp /metrics/backup_k8s_resources.prom
|
|
||||||
|
|
||||||
echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ==="
|
|
||||||
exit ${CHECK_STATUS}
|
|
||||||
volumeMounts:
|
|
||||||
- name: dump
|
|
||||||
mountPath: /dump
|
|
||||||
readOnly: true
|
|
||||||
- name: metrics
|
|
||||||
mountPath: /metrics
|
|
||||||
- name: cache
|
|
||||||
mountPath: /root/.cache/restic
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 100m
|
|
||||||
memory: 128Mi
|
|
||||||
limits:
|
|
||||||
cpu: 1500m
|
|
||||||
memory: 1Gi
|
|
||||||
volumes:
|
|
||||||
- name: dump
|
|
||||||
emptyDir: {}
|
|
||||||
- name: metrics
|
|
||||||
hostPath:
|
|
||||||
# node-exporter's textfile-collector directory
|
|
||||||
# (DEV-494). See sibling loki cronjob for detail.
|
|
||||||
path: /var/lib/node_exporter/textfile_collector
|
|
||||||
type: DirectoryOrCreate
|
|
||||||
- name: cache
|
|
||||||
emptyDir: {}
|
|
||||||
|
|
@ -35,8 +35,6 @@ tests:
|
||||||
values: '-104400x60'
|
values: '-104400x60'
|
||||||
- series: 'backup_grafana_timestamp_seconds'
|
- series: 'backup_grafana_timestamp_seconds'
|
||||||
values: '-104400x60'
|
values: '-104400x60'
|
||||||
- series: 'backup_k8s_resources_timestamp_seconds'
|
|
||||||
values: '-104400x60'
|
|
||||||
- series: 'backup_prometheus_timestamp_seconds'
|
- series: 'backup_prometheus_timestamp_seconds'
|
||||||
values: '-104400x60'
|
values: '-104400x60'
|
||||||
alert_rule_test:
|
alert_rule_test:
|
||||||
|
|
@ -70,21 +68,6 @@ tests:
|
||||||
`backup_grafana_timestamp_seconds` sample in more
|
`backup_grafana_timestamp_seconds` sample in more
|
||||||
than 28 hours. Expected daily at 03:15 UTC.
|
than 28 hours. Expected daily at 03:15 UTC.
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
Runbook: docs/monitoring/restic-restore.md
|
||||||
- eval_time: 30m
|
|
||||||
alertname: BackupK8sResourcesStale
|
|
||||||
exp_alerts:
|
|
||||||
- exp_labels:
|
|
||||||
severity: warning
|
|
||||||
service: monitoring
|
|
||||||
component: backup
|
|
||||||
repo: k8s-resources
|
|
||||||
exp_annotations:
|
|
||||||
summary: "K8s-resources restic backup is stale (>28h)"
|
|
||||||
description: |
|
|
||||||
`backup-k8s-resources` has not written a fresh
|
|
||||||
`backup_k8s_resources_timestamp_seconds` sample in
|
|
||||||
more than 28 hours. Expected daily at 02:00 UTC.
|
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
|
||||||
- eval_time: 30m
|
- eval_time: 30m
|
||||||
alertname: BackupPrometheusStale
|
alertname: BackupPrometheusStale
|
||||||
exp_alerts:
|
exp_alerts:
|
||||||
|
|
@ -112,8 +95,6 @@ tests:
|
||||||
values: '-3600x60'
|
values: '-3600x60'
|
||||||
- series: 'backup_grafana_timestamp_seconds'
|
- series: 'backup_grafana_timestamp_seconds'
|
||||||
values: '-3600x60'
|
values: '-3600x60'
|
||||||
- series: 'backup_k8s_resources_timestamp_seconds'
|
|
||||||
values: '-3600x60'
|
|
||||||
- series: 'backup_prometheus_timestamp_seconds'
|
- series: 'backup_prometheus_timestamp_seconds'
|
||||||
values: '-3600x60'
|
values: '-3600x60'
|
||||||
alert_rule_test:
|
alert_rule_test:
|
||||||
|
|
@ -123,9 +104,6 @@ tests:
|
||||||
- eval_time: 30m
|
- eval_time: 30m
|
||||||
alertname: BackupGrafanaStale
|
alertname: BackupGrafanaStale
|
||||||
exp_alerts: []
|
exp_alerts: []
|
||||||
- eval_time: 30m
|
|
||||||
alertname: BackupK8sResourcesStale
|
|
||||||
exp_alerts: []
|
|
||||||
- eval_time: 30m
|
- eval_time: 30m
|
||||||
alertname: BackupPrometheusStale
|
alertname: BackupPrometheusStale
|
||||||
exp_alerts: []
|
exp_alerts: []
|
||||||
|
|
@ -139,8 +117,6 @@ tests:
|
||||||
values: '1x30'
|
values: '1x30'
|
||||||
- series: 'backup_grafana_check_status'
|
- series: 'backup_grafana_check_status'
|
||||||
values: '2x30'
|
values: '2x30'
|
||||||
- series: 'backup_k8s_resources_check_status'
|
|
||||||
values: '1x30'
|
|
||||||
- series: 'backup_prometheus_check_status'
|
- series: 'backup_prometheus_check_status'
|
||||||
values: '1x30'
|
values: '1x30'
|
||||||
alert_rule_test:
|
alert_rule_test:
|
||||||
|
|
@ -172,21 +148,6 @@ tests:
|
||||||
`restic check --read-data-subset=5%` returned
|
`restic check --read-data-subset=5%` returned
|
||||||
exit code 2 on the grafana repository.
|
exit code 2 on the grafana repository.
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
Runbook: docs/monitoring/restic-restore.md
|
||||||
- eval_time: 15m
|
|
||||||
alertname: BackupK8sResourcesCheckFailed
|
|
||||||
exp_alerts:
|
|
||||||
- exp_labels:
|
|
||||||
severity: warning
|
|
||||||
service: monitoring
|
|
||||||
component: backup
|
|
||||||
repo: k8s-resources
|
|
||||||
exp_annotations:
|
|
||||||
summary: "restic check failed on k8s-resources repo"
|
|
||||||
description: |
|
|
||||||
`restic check --read-data-subset=5%` returned
|
|
||||||
exit code 1 on the k8s-resources
|
|
||||||
repository.
|
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
|
||||||
- eval_time: 15m
|
- eval_time: 15m
|
||||||
alertname: BackupPrometheusCheckFailed
|
alertname: BackupPrometheusCheckFailed
|
||||||
exp_alerts:
|
exp_alerts:
|
||||||
|
|
@ -212,8 +173,6 @@ tests:
|
||||||
values: '0x30'
|
values: '0x30'
|
||||||
- series: 'backup_grafana_check_status'
|
- series: 'backup_grafana_check_status'
|
||||||
values: '0x30'
|
values: '0x30'
|
||||||
- series: 'backup_k8s_resources_check_status'
|
|
||||||
values: '0x30'
|
|
||||||
- series: 'backup_prometheus_check_status'
|
- series: 'backup_prometheus_check_status'
|
||||||
values: '0x30'
|
values: '0x30'
|
||||||
alert_rule_test:
|
alert_rule_test:
|
||||||
|
|
@ -223,9 +182,6 @@ tests:
|
||||||
- eval_time: 15m
|
- eval_time: 15m
|
||||||
alertname: BackupGrafanaCheckFailed
|
alertname: BackupGrafanaCheckFailed
|
||||||
exp_alerts: []
|
exp_alerts: []
|
||||||
- eval_time: 15m
|
|
||||||
alertname: BackupK8sResourcesCheckFailed
|
|
||||||
exp_alerts: []
|
|
||||||
- eval_time: 15m
|
- eval_time: 15m
|
||||||
alertname: BackupPrometheusCheckFailed
|
alertname: BackupPrometheusCheckFailed
|
||||||
exp_alerts: []
|
exp_alerts: []
|
||||||
|
|
|
||||||
|
|
@ -19,7 +19,7 @@
|
||||||
# kube-prometheus-stack node-exporter DaemonSet (DEV-494).
|
# kube-prometheus-stack node-exporter DaemonSet (DEV-494).
|
||||||
#
|
#
|
||||||
# Because a CronJob may run on a different worker across days (loki
|
# Because a CronJob may run on a different worker across days (loki
|
||||||
# backup follows the loki pod; k8s-resources is unpinned), stale
|
# backup follows the loki pod), stale
|
||||||
# `.prom` files can linger on nodes the job has since left. That
|
# `.prom` files can linger on nodes the job has since left. That
|
||||||
# would leave a per-node series with an old timestamp/check-status
|
# would leave a per-node series with an old timestamp/check-status
|
||||||
# indefinitely. The alerts below aggregate across instances so a
|
# indefinitely. The alerts below aggregate across instances so a
|
||||||
|
|
@ -74,21 +74,6 @@ spec:
|
||||||
`backup_grafana_timestamp_seconds` sample in more
|
`backup_grafana_timestamp_seconds` sample in more
|
||||||
than 28 hours. Expected daily at 03:15 UTC.
|
than 28 hours. Expected daily at 03:15 UTC.
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
Runbook: docs/monitoring/restic-restore.md
|
||||||
- alert: BackupK8sResourcesStale
|
|
||||||
expr: time() - max(backup_k8s_resources_timestamp_seconds) > 28 * 3600
|
|
||||||
for: 15m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
service: monitoring
|
|
||||||
component: backup
|
|
||||||
repo: k8s-resources
|
|
||||||
annotations:
|
|
||||||
summary: "K8s-resources restic backup is stale (>28h)"
|
|
||||||
description: |
|
|
||||||
`backup-k8s-resources` has not written a fresh
|
|
||||||
`backup_k8s_resources_timestamp_seconds` sample in
|
|
||||||
more than 28 hours. Expected daily at 02:00 UTC.
|
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
|
||||||
- alert: BackupPrometheusStale
|
- alert: BackupPrometheusStale
|
||||||
expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600
|
expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600
|
||||||
for: 15m
|
for: 15m
|
||||||
|
|
@ -150,21 +135,6 @@ spec:
|
||||||
`restic check --read-data-subset=5%` returned
|
`restic check --read-data-subset=5%` returned
|
||||||
exit code {{ $value }} on the grafana repository.
|
exit code {{ $value }} on the grafana repository.
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
Runbook: docs/monitoring/restic-restore.md
|
||||||
- alert: BackupK8sResourcesCheckFailed
|
|
||||||
expr: backup_k8s_resources_check_status != 0
|
|
||||||
for: 5m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
service: monitoring
|
|
||||||
component: backup
|
|
||||||
repo: k8s-resources
|
|
||||||
annotations:
|
|
||||||
summary: "restic check failed on k8s-resources repo"
|
|
||||||
description: |
|
|
||||||
`restic check --read-data-subset=5%` returned
|
|
||||||
exit code {{ $value }} on the k8s-resources
|
|
||||||
repository.
|
|
||||||
Runbook: docs/monitoring/restic-restore.md
|
|
||||||
- alert: BackupPrometheusCheckFailed
|
- alert: BackupPrometheusCheckFailed
|
||||||
expr: backup_prometheus_check_status != 0
|
expr: backup_prometheus_check_status != 0
|
||||||
for: 5m
|
for: 5m
|
||||||
|
|
|
||||||
|
|
@ -48,7 +48,7 @@ metadata:
|
||||||
type: prometheus
|
type: prometheus
|
||||||
backend: restic
|
backend: restic
|
||||||
spec:
|
spec:
|
||||||
schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana/k8s-resources
|
schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana
|
||||||
concurrencyPolicy: Forbid
|
concurrencyPolicy: Forbid
|
||||||
successfulJobsHistoryLimit: 3
|
successfulJobsHistoryLimit: 3
|
||||||
failedJobsHistoryLimit: 3
|
failedJobsHistoryLimit: 3
|
||||||
|
|
@ -173,7 +173,7 @@ spec:
|
||||||
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
|
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
|
||||||
|
|
||||||
# Textfile-collector metrics. Same hostPath pattern as
|
# Textfile-collector metrics. Same hostPath pattern as
|
||||||
# the loki/grafana/k8s-resources siblings (DEV-494) —
|
# the loki/grafana siblings (DEV-494) —
|
||||||
# written atomically via `.tmp` + rename so a mid-write
|
# written atomically via `.tmp` + rename so a mid-write
|
||||||
# read never surfaces a truncated sample.
|
# read never surfaces a truncated sample.
|
||||||
# backup_prometheus_success rolls in both stages: the
|
# backup_prometheus_success rolls in both stages: the
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue