basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)

Browse source 
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.

- apps/observability/patches/node-exporter-textfile-collector.yaml:
  strategic-merge patch on the kube-prometheus-stack node-exporter DS
  that adds `--collector.textfile.directory=/host/textfile_collector`
  and mounts `/var/lib/node_exporter/textfile_collector` read-only.
  Chart isn't tracked in ArgoCD, so we keep the patch under version
  control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
  swap `emptyDir` /metrics for a `hostPath` on the same directory
  (`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
  never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
  for all four freshness alerts so a stale `.prom` left on a node the
  job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
  the on-node directory, atomic write, cross-node staleness rationale.

Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-08-16 16:21:56 +00:00
parent b1680431c6
commit ef62dde67c
9 changed files with 411 additions and 176 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

33
apps/monitoring/README.md
View file

@ -5,12 +5,12 @@ Manifests recording the cluster-side monitoring backup CronJobs that were previo
- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)). Note: still uses `rclone sync` (plaintext at rest on Hetzner). Migration to `restic` or `rclone crypt` is filed as a follow-up.
- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos, plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490)).
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block.
- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the four restic repos (loki/grafana/k8s-resources/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)).
## Shared SealedSecret
All three restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys:
All four restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys:
| Key | Purpose |
|------------------|------------------------------------------------------------------------------------------------------------------|
@ -18,7 +18,7 @@ All three restic CronJobs read Hetzner S3 credentials + the restic repository pa
| `secret-key` | Hetzner Object Storage secret access key |
| `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) |
| `bucket` | Bucket name (single bucket, per-prefix repos) |
| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. |
| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all four repos (loki/grafana/k8s-resources/prometheus) — rotating rewrites the key file on every repo. |
## Restore
@ -26,21 +26,24 @@ Restore procedure for all three restic repos: [`docs/monitoring/restic-restore.m
## Emitted metrics (textfile-collector format)
Each restic CronJob writes to `/metrics/backup_<kind>.prom` inside the pod's emptyDir. Once node-exporter's textfile-collector path is wired up (follow-up), these become scrapeable and the alerts in `backup-restic-alerts.yaml` evaluate against live data.
Each restic CronJob writes `/metrics/backup_<kind>.prom` (atomic — `.prom.tmp` + `mv`) into a `hostPath` volume mounted at the node-exporter textfile-collector directory (`/var/lib/node_exporter/textfile_collector`). The kube-prometheus-stack node-exporter DaemonSet has `--collector.textfile.directory=/host/textfile_collector` enabled ([DEV-494](/DEV/issues/DEV-494), applied via [`apps/observability/patches/node-exporter-textfile-collector.yaml`](../observability/patches/node-exporter-textfile-collector.yaml)) and surfaces those samples in Prometheus.
| Metric | Emitted by |
|--------------------------------------------|---------------------------------------------------------------------|
| `backup_<kind>_success` (0/1) | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml` |
| `backup_<kind>_timestamp_seconds` | same |
| `backup_<kind>_check_status` (exit code) | same — from `restic check --read-data-subset=5%` |
| `restic_repo_size_bytes{repo="<kind>"}` | same — from `restic stats --json --mode raw-data` (added in DEV-490)|
| Metric | Emitted by |
|--------------------------------------------|----------------------------------------------------------------------------------|
| `backup_<kind>_success` (0/1) | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` |
| `backup_<kind>_timestamp_seconds` | same |
| `backup_<kind>_check_status` (exit code) | same — from `restic check --read-data-subset=5%` |
| `backup_prometheus_backup_status` | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) |
| `restic_repo_size_bytes{repo="<kind>"}` | same — from `restic stats --json --mode raw-data` (added in DEV-490) |
**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods, `k8s-resources` is unpinned), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.
## Alerts
`backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`):
`backup-restic-alerts.yaml` defines nine alerts (all `severity: warning`):
- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` — `time() - backup_<kind>_timestamp_seconds > 28h`. Daily schedule + 4 h grace.
- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` — `backup_<kind>_check_status != 0`.
- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
- `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions.
All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally:
@ -52,4 +55,4 @@ promtool check rules /tmp/backup-restic-rules.yaml
promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
```
The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` and `prometheus-backup` remain unpinned.
The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` has no PVC dep and stays unpinned. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers.

15
apps/monitoring/backup-grafana-restic-cronjob.yaml
View file

@ -118,15 +118,16 @@ spec:
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics; identical wiring to the loki
# sibling. Scrapeable once node-exporter's textfile
# collector path is enabled — tracked in DEV-482.
# Textfile-collector metrics; identical wiring to the
# loki sibling. See that file for the atomic-write
# rationale (DEV-494).
{
echo "backup_grafana_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
echo "backup_grafana_timestamp_seconds $(date +%s)"
echo "backup_grafana_check_status ${CHECK_STATUS}"
echo "restic_repo_size_bytes{repo=\"grafana\"} ${REPO_SIZE_BYTES}"
} > /metrics/backup_grafana.prom
} > /metrics/backup_grafana.prom.tmp
mv /metrics/backup_grafana.prom.tmp /metrics/backup_grafana.prom
echo "=== backup-grafana-restic finished at $(date -u +%FT%TZ) ==="
exit ${CHECK_STATUS}
@ -150,6 +151,10 @@ spec:
persistentVolumeClaim:
claimName: grafana-storage
- name: metrics
emptyDir: {}
hostPath:
# node-exporter's textfile-collector directory
# (DEV-494). See sibling loki cronjob for detail.
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
- name: cache
emptyDir: {}

13
apps/monitoring/backup-k8s-resources-cronjob.yaml
View file

@ -165,14 +165,15 @@ spec:
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics; identical wiring to the
# loki/grafana siblings. Scrapeable once node-exporter's
# textfile collector path is enabled — tracked in DEV-482.
# loki/grafana siblings. See loki cronjob for the
# atomic-write rationale (DEV-494).
{
echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
echo "backup_k8s_resources_timestamp_seconds $(date +%s)"
echo "backup_k8s_resources_check_status ${CHECK_STATUS}"
echo "restic_repo_size_bytes{repo=\"k8s-resources\"} ${REPO_SIZE_BYTES}"
} > /metrics/backup_k8s_resources.prom
} > /metrics/backup_k8s_resources.prom.tmp
mv /metrics/backup_k8s_resources.prom.tmp /metrics/backup_k8s_resources.prom
echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ==="
exit ${CHECK_STATUS}
@ -195,6 +196,10 @@ spec:
- name: dump
emptyDir: {}
- name: metrics
emptyDir: {}
hostPath:
# node-exporter's textfile-collector directory
# (DEV-494). See sibling loki cronjob for detail.
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
- name: cache
emptyDir: {}

23
apps/monitoring/backup-loki-restic-cronjob.yaml
View file

@ -134,17 +134,19 @@ spec:
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics. Written into an emptyDir per
# the current pattern used by the other backup CronJobs.
# Once a node-exporter textfile collector path is wired
# up, these become scrapeable — see the follow-up notes
# in DEV-482.
# Textfile-collector metrics. Written to the shared
# host directory that node-exporter's textfile collector
# scrapes (DEV-494). Atomic write: build the file with a
# `.tmp` extension (ignored by node-exporter) and rename
# into place, so a mid-write read never surfaces a
# truncated sample.
{
echo "backup_loki_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
echo "backup_loki_timestamp_seconds $(date +%s)"
echo "backup_loki_check_status ${CHECK_STATUS}"
echo "restic_repo_size_bytes{repo=\"loki\"} ${REPO_SIZE_BYTES}"
} > /metrics/backup_loki.prom
} > /metrics/backup_loki.prom.tmp
mv /metrics/backup_loki.prom.tmp /metrics/backup_loki.prom
echo "=== backup-loki-restic finished at $(date -u +%FT%TZ) ==="
exit ${CHECK_STATUS}
@ -173,6 +175,13 @@ spec:
persistentVolumeClaim:
claimName: loki-storage-encrypted
- name: metrics
emptyDir: {}
hostPath:
# node-exporter's textfile-collector directory
# (DEV-494). `DirectoryOrCreate` lets kubelet create
# the dir on the current node if it does not yet
# exist — nodes were pre-created out-of-band, this
# is a safety net.
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
- name: cache
emptyDir: {}

44
apps/monitoring/backup-restic-alerts.test.yaml
View file

@ -37,6 +37,8 @@ tests:
values: '-104400x60'
- series: 'backup_k8s_resources_timestamp_seconds'
values: '-104400x60'
- series: 'backup_prometheus_timestamp_seconds'
values: '-104400x60'
alert_rule_test:
- eval_time: 30m
alertname: BackupLokiStale
@ -83,6 +85,21 @@ tests:
`backup_k8s_resources_timestamp_seconds` sample in
more than 28 hours. Expected daily at 02:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- eval_time: 30m
alertname: BackupPrometheusStale
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
exp_annotations:
summary: "Prometheus restic backup is stale (>28h)"
description: |
`prometheus-backup` has not written a fresh
`backup_prometheus_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:30 UTC.
Runbook: docs/monitoring/restic-restore.md
# ---------------------------------------------------------------------------
# 2. Freshness alerts stay silent when timestamp is fresh (<28h).
@ -97,6 +114,8 @@ tests:
values: '-3600x60'
- series: 'backup_k8s_resources_timestamp_seconds'
values: '-3600x60'
- series: 'backup_prometheus_timestamp_seconds'
values: '-3600x60'
alert_rule_test:
- eval_time: 30m
alertname: BackupLokiStale
@ -107,6 +126,9 @@ tests:
- eval_time: 30m
alertname: BackupK8sResourcesStale
exp_alerts: []
- eval_time: 30m
alertname: BackupPrometheusStale
exp_alerts: []
# ---------------------------------------------------------------------------
# 3. Check-status alerts fire when the metric is non-zero.
@ -119,6 +141,8 @@ tests:
values: '2x30'
- series: 'backup_k8s_resources_check_status'
values: '1x30'
- series: 'backup_prometheus_check_status'
values: '1x30'
alert_rule_test:
- eval_time: 15m
alertname: BackupLokiCheckFailed
@ -163,6 +187,21 @@ tests:
exit code 1 on the k8s-resources
repository.
Runbook: docs/monitoring/restic-restore.md
- eval_time: 15m
alertname: BackupPrometheusCheckFailed
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
exp_annotations:
summary: "restic check failed on prometheus repo"
description: |
`restic check --read-data-subset=5%` returned
exit code 1 on the prometheus
repository.
Runbook: docs/monitoring/restic-restore.md
# ---------------------------------------------------------------------------
# 4. Check-status alerts stay silent on 0.
@ -175,6 +214,8 @@ tests:
values: '0x30'
- series: 'backup_k8s_resources_check_status'
values: '0x30'
- series: 'backup_prometheus_check_status'
values: '0x30'
alert_rule_test:
- eval_time: 15m
alertname: BackupLokiCheckFailed
@ -185,6 +226,9 @@ tests:
- eval_time: 15m
alertname: BackupK8sResourcesCheckFailed
exp_alerts: []
- eval_time: 15m
alertname: BackupPrometheusCheckFailed
exp_alerts: []
# ---------------------------------------------------------------------------
# 5. Repo-size alert fires when >20 GiB (21474836480 bytes).

57
apps/monitoring/backup-restic-alerts.yaml
View file

@ -14,11 +14,19 @@
# by the same CronJob step (see backup-*-restic-cronjob.yaml,
# `restic stats --json --mode raw-data`).
#
# The metrics are written to an emptyDir at /metrics per job.
# They become scrapeable once node-exporter's textfile collector
# path is wired up — tracked as a follow-up. `promtool test rules`
# against apps/monitoring/backup-restic-alerts.test.yaml verifies
# the rules with synthetic samples in the meantime.
# The metrics are written to the node's textfile-collector directory
# (`/var/lib/node_exporter/textfile_collector`) and scraped by the
# kube-prometheus-stack node-exporter DaemonSet (DEV-494).
#
# Because a CronJob may run on a different worker across days (loki
# backup follows the loki pod; k8s-resources is unpinned), stale
# `.prom` files can linger on nodes the job has since left. That
# would leave a per-node series with an old timestamp/check-status
# indefinitely. The alerts below aggregate across instances so a
# single fresh sample from the node where the job currently runs is
# enough to keep the freshness alert quiet, and check/size alerts
# fire when *any* node reports a bad value (which is the correct
# behaviour — a recent failure is still a signal).
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
@ -34,7 +42,10 @@ spec:
interval: 60s
rules:
- alert: BackupLokiStale
expr: time() - backup_loki_timestamp_seconds > 28 * 3600
# `max()` collapses per-node samples so a stale `.prom` file
# left behind on a node the loki backup no longer runs on
# does not fire this alert; the freshest sample wins.
expr: time() - max(backup_loki_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
@ -49,7 +60,7 @@ spec:
28 hours. Expected daily at 03:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupGrafanaStale
expr: time() - backup_grafana_timestamp_seconds > 28 * 3600
expr: time() - max(backup_grafana_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
@ -64,7 +75,7 @@ spec:
than 28 hours. Expected daily at 03:15 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupK8sResourcesStale
expr: time() - backup_k8s_resources_timestamp_seconds > 28 * 3600
expr: time() - max(backup_k8s_resources_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
@ -78,6 +89,21 @@ spec:
`backup_k8s_resources_timestamp_seconds` sample in
more than 28 hours. Expected daily at 02:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupPrometheusStale
expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
annotations:
summary: "Prometheus restic backup is stale (>28h)"
description: |
`prometheus-backup` has not written a fresh
`backup_prometheus_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:30 UTC.
Runbook: docs/monitoring/restic-restore.md
- name: backup-restic.integrity
interval: 60s
rules:
@ -124,6 +150,21 @@ spec:
exit code {{ $value }} on the k8s-resources
repository.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupPrometheusCheckFailed
expr: backup_prometheus_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
annotations:
summary: "restic check failed on prometheus repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the prometheus
repository.
Runbook: docs/monitoring/restic-restore.md
- name: backup-restic.size
interval: 60s
rules:

294
apps/monitoring/prometheus-backup-cronjob.yaml
View file

@ -1,16 +1,43 @@
---
# Prometheus data backup (DEV-465).
# Prometheus data backup via restic to Hetzner Object Storage
# (DEV-492, DEV-482 Option 4). Replaces the DEV-465 rclone-sync job so
# every monitoring backup ships client-side-encrypted; Hetzner Object
# Storage has no SSE-S3/SSE-KMS, so the previous plaintext-at-rest
# object layout was the only remaining gap.
#
# Prometheus data lives on the RWO PVC `prometheus-data-encrypted` in
# namespace `monitoring`. That PVC is mounted by the Prometheus pod, and
# the Hetzner CSI volume can only be attached to one node at a time.
# Streams the RWO PVC `prometheus-data-encrypted` (mounted read-only)
# into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, a client-side
# encrypted restic repository (tag=`prometheus`, host=`k3s`).
#
# This CronJob co-schedules with the Prometheus pod via podAffinity, so it
# lands on whichever node currently holds `prometheus-data-encrypted`. The
# PVC is mounted read-only alongside the running Prometheus pod (RWO permits
# additional read-only mounts on the same node) and streamed to Hetzner S3
# via rclone under `basicstack-backup/prometheus/prometheus-<DATE>/`. Old
# snapshots are pruned after 7 days.
# Node scheduling matches the previous job: podAffinity co-schedules
# with the Prometheus pod (app=prometheus, topology
# kubernetes.io/hostname). Hetzner CSI RWO permits additional read-only
# mounts on the node that holds the PVC's VolumeAttachment, so this
# survives Prometheus being rescheduled to a different worker.
#
# Prometheus TSDB compaction race
# --------------------------------
# Prometheus rewrites the on-disk store roughly every 2 h: it creates a
# new block dir, then deletes the source dirs. restic walks the source
# tree once and may catch a file that disappeared mid-walk; restic
# 0.17.3 exits 3 ("at least one source file could not be read") in
# that case, and the snapshot excludes only the missing file. The
# next daily run picks up the successor block, so the race is not a
# data-loss risk — but we must not treat exit 3 as a hard failure, or
# the daily job will alert-flap.
#
# Mitigation:
# - exclude `wal/*` (WAL is replayed from a fresh instance on
# restart; we accept losing the last ~15 s of ingested samples
# rather than snapshotting a moving segment)
# - exclude `chunks_head/*` (in-memory head block; ephemeral, would
# be rebuilt from WAL which we do not keep)
# - exclude Prometheus lock/scratch files (`lock`, `queries.active`,
# `*.tmp`, `lost+found/*`)
# - treat restic exit code 3 as a soft warning (log, continue);
# any other non-zero exit is still fatal
# - restore drill re-runs `promtool tsdb analyze` against every
# block so a corrupted snapshot is caught end-to-end
apiVersion: batch/v1
kind: CronJob
metadata:
@ -19,8 +46,9 @@ metadata:
labels:
app: backup
type: prometheus
backend: restic
spec:
schedule: "30 3 * * *" # daily 03:30, offset from the loki/grafana restic runs
schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana/k8s-resources
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
@ -29,19 +57,18 @@ spec:
labels:
app: backup
type: prometheus
backend: restic
spec:
backoffLimit: 2
activeDeadlineSeconds: 3600 # 60 min hard cap; Prometheus data is ~9GB
activeDeadlineSeconds: 3600
template:
metadata:
labels:
app: backup
type: prometheus
backend: restic
spec:
restartPolicy: OnFailure
# Co-schedule with the Prometheus pod so the RWO PVC attaches on
# the same node. This survives Prometheus being rescheduled to a
# different worker (the backup follows).
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
@ -53,134 +80,24 @@ spec:
- prometheus
topologyKey: kubernetes.io/hostname
containers:
- name: prometheus-backup
image: rclone/rclone:1.68
command:
- /bin/sh
- -c
- |
set -eu
DATE=$(date +%Y%m%d-%H%M%S)
echo "=== Prometheus backup started at $(date) (target prefix: prometheus-${DATE}) ==="
mkdir -p /root/.config/rclone
cat > /root/.config/rclone/rclone.conf <<EOC
[hetzner-s3]
type = s3
provider = Other
access_key_id = ${S3_ACCESS_KEY}
secret_access_key = ${S3_SECRET_KEY}
endpoint = ${S3_ENDPOINT}
acl = private
EOC
SOURCE_BYTES=$(du -sb /source 2>/dev/null | cut -f1 || echo 0)
SOURCE_HUMAN=$(du -sh /source 2>/dev/null | cut -f1 || echo unknown)
echo "Source /source size: ${SOURCE_HUMAN} (${SOURCE_BYTES} bytes)"
# Prometheus TSDB is largely immutable chunk files plus an
# append-only WAL. rclone sync is safe with the database
# live: on restore the WAL is replayed. HOWEVER Prometheus
# compacts blocks every ~2h and deletes their source dirs,
# which races with the copy and produces "no such file or
# directory" errors mid-run. Those are expected and do
# not indicate data loss — the compacted successor blocks
# are picked up on the same or the next daily run. We
# therefore do not fail the job on rclone's non-zero exit
# from those transient errors; instead we validate the
# backup by comparing destination size to source (must be
# >= 80% of source bytes and > 100 MiB).
DEST="hetzner-s3:${S3_BUCKET}/prometheus/prometheus-${DATE}/"
echo "Streaming Prometheus data to ${DEST} ..."
RCLONE_EXIT=0
rclone sync /source "${DEST}" \
--transfers 4 \
--checkers 4 \
--stats 30s \
--stats-log-level NOTICE \
--s3-chunk-size 32M \
--s3-upload-concurrency 4 \
--retries 3 \
--retries-sleep 30s || RCLONE_EXIT=$?
echo "rclone sync exit code: ${RCLONE_EXIT}"
echo "Measuring destination size..."
DEST_BYTES=$(rclone size "${DEST}" --json 2>/dev/null | \
sed -n 's/.*"bytes":\s*\([0-9]\+\).*/\1/p' | head -1)
DEST_BYTES=${DEST_BYTES:-0}
DEST_HUMAN=$(rclone size "${DEST}" 2>/dev/null | \
grep -oE 'Total size:.*' || echo "Total size: unknown")
echo "Destination bytes: ${DEST_BYTES}"
echo "Destination summary: ${DEST_HUMAN}"
MIN_ACCEPTABLE=$(( SOURCE_BYTES * 80 / 100 ))
FLOOR=104857600 # 100 MiB absolute floor
echo "Acceptance threshold: dest >= ${MIN_ACCEPTABLE} bytes and > ${FLOOR} bytes"
if [ "${DEST_BYTES}" -lt "${FLOOR}" ]; then
echo "ERROR: destination is below hard floor (100 MiB) — backup failed."
exit 2
fi
if [ "${DEST_BYTES}" -lt "${MIN_ACCEPTABLE}" ]; then
echo "ERROR: destination is < 80% of source (${DEST_BYTES} < ${MIN_ACCEPTABLE}) — backup incomplete."
exit 3
fi
echo "OK: destination size acceptable."
# Prune by prefix name, NOT by file mtime. rclone preserves
# each source file's mtime on upload, and Prometheus TSDB
# chunk files retain very old mtimes (weeks-old blocks),
# so `rclone delete --min-age 7d` would eat the just-
# uploaded backup. Every top-level prefix is named
# `prometheus-YYYYMMDD-HHMMSS`, so we compare the encoded
# date to a 7-day threshold and purge whole prefixes.
echo "Pruning prometheus backup prefixes older than 7 days..."
# BusyBox date lacks GNU's `-d "7 days ago"` and BSD's `-v -7d`,
# so compute the cutoff via @epoch which BusyBox does support.
CUTOFF_EPOCH=$(( $(date +%s) - 7 * 86400 ))
CUTOFF=$(date -d "@${CUTOFF_EPOCH}" +%Y%m%d)
echo "Cutoff (delete prefixes with date < ${CUTOFF}):"
KEEP=0
PRUNE=0
for PREFIX in $(rclone lsf --dirs-only "hetzner-s3:${S3_BUCKET}/prometheus/" 2>/dev/null | sed 's:/$::'); do
case "${PREFIX}" in
prometheus-*)
PDATE=$(echo "${PREFIX}" | sed -n 's/^prometheus-\([0-9]\{8\}\)-.*/\1/p')
if [ -z "${PDATE}" ]; then
echo " SKIP ${PREFIX} (unparseable name)"
continue
fi
if [ "${PDATE}" -lt "${CUTOFF}" ]; then
echo " PURGE ${PREFIX} (date ${PDATE} < ${CUTOFF})"
rclone purge "hetzner-s3:${S3_BUCKET}/prometheus/${PREFIX}" \
|| echo " WARN purge failed for ${PREFIX} (continuing)"
PRUNE=$((PRUNE + 1))
else
echo " KEEP ${PREFIX} (date ${PDATE})"
KEEP=$((KEEP + 1))
fi
;;
*)
echo " SKIP ${PREFIX} (not a prometheus- prefix)"
;;
esac
done
echo "Prune summary: kept ${KEEP}, purged ${PRUNE}."
echo "Post-run inventory (prometheus/ prefixes):"
rclone lsd "hetzner-s3:${S3_BUCKET}/prometheus/" || true
echo "=== Prometheus backup completed at $(date) ==="
- name: restic
image: restic/restic:0.17.3
env:
- name: S3_ACCESS_KEY
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: access-key
- name: S3_SECRET_KEY
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: secret-key
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: restic-password
- name: S3_ENDPOINT
valueFrom:
secretKeyRef:
@ -191,18 +108,121 @@ spec:
secretKeyRef:
name: monitoring-s3-backup
key: bucket
- name: RESTIC_REPOSITORY
value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/prometheus"
command:
- /bin/sh
- -c
- |
set -eu
echo "=== backup-prometheus-restic started at $(date -u +%FT%TZ) ==="
echo "Repository: ${RESTIC_REPOSITORY}"
# First-run tolerance: init if the repo isn't there yet.
if restic snapshots >/dev/null 2>&1; then
echo "Repo exists, skipping init."
else
echo "Repo missing, initialising..."
restic init
fi
# See "Prometheus TSDB compaction race" in the file
# header. WAL + chunks_head are excluded on purpose;
# exit code 3 (source file vanished mid-walk during a
# compaction) is accepted and reported, any other
# non-zero exit is fatal.
echo "--- restic backup /source (exclude wal/chunks_head + lock files) ---"
BACKUP_STATUS=0
restic backup /source \
--tag prometheus \
--host k3s \
--exclude 'wal/*' \
--exclude 'chunks_head/*' \
--exclude 'lock' \
--exclude 'queries.active' \
--exclude 'lost+found/*' \
--exclude '*.tmp' || BACKUP_STATUS=$?
echo "restic backup exit: ${BACKUP_STATUS}"
if [ "${BACKUP_STATUS}" -eq 0 ]; then
echo "backup: all files captured cleanly"
elif [ "${BACKUP_STATUS}" -eq 3 ]; then
echo "backup: exit 3 (source files vanished mid-walk) — expected under Prometheus compaction, continuing"
else
echo "backup: FATAL exit ${BACKUP_STATUS} (not compaction-race)"
exit ${BACKUP_STATUS}
fi
echo "--- restic forget/prune ---"
restic forget --tag prometheus \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
--prune
echo "--- restic check --read-data-subset=5% ---"
CHECK_STATUS=0
restic check --read-data-subset=5% || CHECK_STATUS=$?
echo "restic check exit: ${CHECK_STATUS}"
echo "--- restic stats (repo size) ---"
REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
| grep -oE '"total_size":[0-9]+' \
| head -1 \
| cut -d: -f2)
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics. Same hostPath pattern as
# the loki/grafana/k8s-resources siblings (DEV-494) —
# written atomically via `.tmp` + rename so a mid-write
# read never surfaces a truncated sample.
# backup_prometheus_success rolls in both stages: the
# backup step (accepting exit 3) and restic check.
BACKUP_OK=0
if [ "${BACKUP_STATUS}" -eq 0 ] || [ "${BACKUP_STATUS}" -eq 3 ]; then
BACKUP_OK=1
fi
SUCCESS=0
if [ "${BACKUP_OK}" -eq 1 ] && [ "${CHECK_STATUS}" -eq 0 ]; then
SUCCESS=1
fi
{
echo "backup_prometheus_success ${SUCCESS}"
echo "backup_prometheus_timestamp_seconds $(date +%s)"
echo "backup_prometheus_check_status ${CHECK_STATUS}"
echo "backup_prometheus_backup_status ${BACKUP_STATUS}"
echo "restic_repo_size_bytes{repo=\"prometheus\"} ${REPO_SIZE_BYTES}"
} > /metrics/backup_prometheus.prom.tmp
mv /metrics/backup_prometheus.prom.tmp /metrics/backup_prometheus.prom
echo "=== backup-prometheus-restic finished at $(date -u +%FT%TZ) ==="
exit ${CHECK_STATUS}
volumeMounts:
- name: prometheus-data
mountPath: /source
readOnly: true
- name: metrics
mountPath: /metrics
- name: cache
mountPath: /root/.cache/restic
resources:
# Prometheus TSDB is ~8-10 GiB; give restic room to
# burst during pack/check but keep steady-state small.
requests:
cpu: 100m
memory: 128Mi
memory: 256Mi
limits:
cpu: 1000m
memory: 512Mi
cpu: 1500m
memory: 1Gi
volumes:
- name: prometheus-data
persistentVolumeClaim:
claimName: prometheus-data-encrypted
- name: metrics
hostPath:
# node-exporter's textfile-collector directory
# (DEV-494). See sibling loki cronjob for detail.
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
- name: cache
emptyDir: {}

38
apps/observability/patches/README.md Normal file
View file

@ -0,0 +1,38 @@
# observability patches
Strategic-merge patches applied on top of Helm-managed observability
resources. Each file is idempotent (re-applying is a no-op) and is
reasserted by hand rather than by a controller, so re-run after any
`helm upgrade` of the affected release.
## `node-exporter-textfile-collector.yaml` (DEV-494)
Enables the node-exporter textfile collector on the
`kube-prometheus-stack-prometheus-node-exporter` DaemonSet by:
1. adding `--collector.textfile.directory=/host/textfile_collector`
to the container args, and
2. mounting the host directory `/var/lib/node_exporter/textfile_collector`
read-only at `/host/textfile_collector` (`hostPath` type
`DirectoryOrCreate`, kubelet creates it on nodes where the
directory does not exist yet).
The four monitoring backup CronJobs in `apps/monitoring/` write their
textfile-collector `.prom` files into that same host directory, so the
metrics surface in Prometheus via node-exporter's normal scrape.
Apply / re-apply:
```bash
kubectl -n observability patch daemonset \
kube-prometheus-stack-prometheus-node-exporter \
--type=strategic \
--patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
kubectl -n observability rollout status daemonset \
kube-prometheus-stack-prometheus-node-exporter
```
The kube-prometheus-stack chart is not currently tracked in ArgoCD;
if it moves under GitOps, fold these values into the chart values as
`prometheus-node-exporter.extraArgs` + `.extraHostVolumeMounts`
instead of maintaining this patch.

70
apps/observability/patches/node-exporter-textfile-collector.yaml Normal file
View file

@ -0,0 +1,70 @@
---
# Strategic-merge patch enabling node-exporter's textfile collector on
# the kube-prometheus-stack node-exporter DaemonSet (DEV-494).
#
# The chart is Helm-managed (release `kube-prometheus-stack` in
# namespace `observability`, chart kube-prometheus-stack-86.2.2 /
# prometheus-node-exporter-4.55.0) and is NOT currently tracked in
# ArgoCD, so a direct DaemonSet patch is the pragmatic wiring path.
# The change:
#
# 1. adds the `--collector.textfile.directory=/host/textfile_collector`
# arg to the node-exporter container, and
# 2. mounts the host directory `/var/lib/node_exporter/textfile_collector`
# read-only at `/host/textfile_collector` (type DirectoryOrCreate so
# kubelet creates the dir on nodes where it does not yet exist).
#
# Apply with:
# kubectl -n observability patch daemonset \
# kube-prometheus-stack-prometheus-node-exporter \
# --type=strategic \
# --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
#
# If the Helm release is ever `helm upgrade`d without folding these
# values into the chart values, this patch will be reverted — re-apply
# it after the upgrade (or move it into a repo-owned values file).
spec:
template:
spec:
containers:
- name: node-exporter
args:
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/host/root
- --path.udev.data=/host/root/run/udev/data
- --web.listen-address=[$(HOST_IP)]:9100
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
- --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$
- --collector.textfile.directory=/host/textfile_collector
volumeMounts:
- mountPath: /host/proc
name: proc
readOnly: true
- mountPath: /host/sys
name: sys
readOnly: true
- mountPath: /host/root
mountPropagation: HostToContainer
name: root
readOnly: true
- mountPath: /host/textfile_collector
name: textfile-collector
readOnly: true
volumes:
- hostPath:
path: /proc
type: ""
name: proc
- hostPath:
path: /sys
type: ""
name: sys
- hostPath:
path: /
type: ""
name: root
- hostPath:
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
name: textfile-collector