diff --git a/apps/monitoring/README.md b/apps/monitoring/README.md index 1875988..1b8bd14 100644 --- a/apps/monitoring/README.md +++ b/apps/monitoring/README.md @@ -5,12 +5,12 @@ Manifests recording the cluster-side monitoring backup CronJobs that were previo - `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)). - `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)). - `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)). -- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)). Note: still uses `rclone sync` (plaintext at rest on Hetzner). Migration to `restic` or `rclone crypt` is filed as a follow-up. -- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos, plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490)). +- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block. +- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the four restic repos (loki/grafana/k8s-resources/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)). ## Shared SealedSecret -All three restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys: +All four restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys: | Key | Purpose | |------------------|------------------------------------------------------------------------------------------------------------------| @@ -18,7 +18,7 @@ All three restic CronJobs read Hetzner S3 credentials + the restic repository pa | `secret-key` | Hetzner Object Storage secret access key | | `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) | | `bucket` | Bucket name (single bucket, per-prefix repos) | -| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. | +| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all four repos (loki/grafana/k8s-resources/prometheus) — rotating rewrites the key file on every repo. | ## Restore @@ -26,21 +26,24 @@ Restore procedure for all three restic repos: [`docs/monitoring/restic-restore.m ## Emitted metrics (textfile-collector format) -Each restic CronJob writes to `/metrics/backup_.prom` inside the pod's emptyDir. Once node-exporter's textfile-collector path is wired up (follow-up), these become scrapeable and the alerts in `backup-restic-alerts.yaml` evaluate against live data. +Each restic CronJob writes `/metrics/backup_.prom` (atomic — `.prom.tmp` + `mv`) into a `hostPath` volume mounted at the node-exporter textfile-collector directory (`/var/lib/node_exporter/textfile_collector`). The kube-prometheus-stack node-exporter DaemonSet has `--collector.textfile.directory=/host/textfile_collector` enabled ([DEV-494](/DEV/issues/DEV-494), applied via [`apps/observability/patches/node-exporter-textfile-collector.yaml`](../observability/patches/node-exporter-textfile-collector.yaml)) and surfaces those samples in Prometheus. -| Metric | Emitted by | -|--------------------------------------------|---------------------------------------------------------------------| -| `backup__success` (0/1) | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml` | -| `backup__timestamp_seconds` | same | -| `backup__check_status` (exit code) | same — from `restic check --read-data-subset=5%` | -| `restic_repo_size_bytes{repo=""}` | same — from `restic stats --json --mode raw-data` (added in DEV-490)| +| Metric | Emitted by | +|--------------------------------------------|----------------------------------------------------------------------------------| +| `backup__success` (0/1) | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` | +| `backup__timestamp_seconds` | same | +| `backup__check_status` (exit code) | same — from `restic check --read-data-subset=5%` | +| `backup_prometheus_backup_status` | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) | +| `restic_repo_size_bytes{repo=""}` | same — from `restic stats --json --mode raw-data` (added in DEV-490) | + +**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods, `k8s-resources` is unpinned), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node. ## Alerts -`backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`): +`backup-restic-alerts.yaml` defines nine alerts (all `severity: warning`): -- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` — `time() - backup__timestamp_seconds > 28h`. Daily schedule + 4 h grace. -- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` — `backup__check_status != 0`. +- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` / `BackupPrometheusStale` — `time() - max(backup__timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire. +- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` / `BackupPrometheusCheckFailed` — `backup__check_status != 0`. - `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions. All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally: @@ -52,4 +55,4 @@ promtool check rules /tmp/backup-restic-rules.yaml promtool test rules apps/monitoring/backup-restic-alerts.test.yaml ``` -The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` and `prometheus-backup` remain unpinned. +The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` has no PVC dep and stays unpinned. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers. diff --git a/apps/monitoring/backup-grafana-restic-cronjob.yaml b/apps/monitoring/backup-grafana-restic-cronjob.yaml index 0968b75..242ed62 100644 --- a/apps/monitoring/backup-grafana-restic-cronjob.yaml +++ b/apps/monitoring/backup-grafana-restic-cronjob.yaml @@ -118,15 +118,16 @@ spec: REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0} echo "restic repo size: ${REPO_SIZE_BYTES} bytes" - # Textfile-collector metrics; identical wiring to the loki - # sibling. Scrapeable once node-exporter's textfile - # collector path is enabled — tracked in DEV-482. + # Textfile-collector metrics; identical wiring to the + # loki sibling. See that file for the atomic-write + # rationale (DEV-494). { echo "backup_grafana_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)" echo "backup_grafana_timestamp_seconds $(date +%s)" echo "backup_grafana_check_status ${CHECK_STATUS}" echo "restic_repo_size_bytes{repo=\"grafana\"} ${REPO_SIZE_BYTES}" - } > /metrics/backup_grafana.prom + } > /metrics/backup_grafana.prom.tmp + mv /metrics/backup_grafana.prom.tmp /metrics/backup_grafana.prom echo "=== backup-grafana-restic finished at $(date -u +%FT%TZ) ===" exit ${CHECK_STATUS} @@ -150,6 +151,10 @@ spec: persistentVolumeClaim: claimName: grafana-storage - name: metrics - emptyDir: {} + hostPath: + # node-exporter's textfile-collector directory + # (DEV-494). See sibling loki cronjob for detail. + path: /var/lib/node_exporter/textfile_collector + type: DirectoryOrCreate - name: cache emptyDir: {} diff --git a/apps/monitoring/backup-k8s-resources-cronjob.yaml b/apps/monitoring/backup-k8s-resources-cronjob.yaml index 751135a..af89fd0 100644 --- a/apps/monitoring/backup-k8s-resources-cronjob.yaml +++ b/apps/monitoring/backup-k8s-resources-cronjob.yaml @@ -165,14 +165,15 @@ spec: echo "restic repo size: ${REPO_SIZE_BYTES} bytes" # Textfile-collector metrics; identical wiring to the - # loki/grafana siblings. Scrapeable once node-exporter's - # textfile collector path is enabled — tracked in DEV-482. + # loki/grafana siblings. See loki cronjob for the + # atomic-write rationale (DEV-494). { echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)" echo "backup_k8s_resources_timestamp_seconds $(date +%s)" echo "backup_k8s_resources_check_status ${CHECK_STATUS}" echo "restic_repo_size_bytes{repo=\"k8s-resources\"} ${REPO_SIZE_BYTES}" - } > /metrics/backup_k8s_resources.prom + } > /metrics/backup_k8s_resources.prom.tmp + mv /metrics/backup_k8s_resources.prom.tmp /metrics/backup_k8s_resources.prom echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ===" exit ${CHECK_STATUS} @@ -195,6 +196,10 @@ spec: - name: dump emptyDir: {} - name: metrics - emptyDir: {} + hostPath: + # node-exporter's textfile-collector directory + # (DEV-494). See sibling loki cronjob for detail. + path: /var/lib/node_exporter/textfile_collector + type: DirectoryOrCreate - name: cache emptyDir: {} diff --git a/apps/monitoring/backup-loki-restic-cronjob.yaml b/apps/monitoring/backup-loki-restic-cronjob.yaml index fc66734..a0b6fdf 100644 --- a/apps/monitoring/backup-loki-restic-cronjob.yaml +++ b/apps/monitoring/backup-loki-restic-cronjob.yaml @@ -134,17 +134,19 @@ spec: REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0} echo "restic repo size: ${REPO_SIZE_BYTES} bytes" - # Textfile-collector metrics. Written into an emptyDir per - # the current pattern used by the other backup CronJobs. - # Once a node-exporter textfile collector path is wired - # up, these become scrapeable — see the follow-up notes - # in DEV-482. + # Textfile-collector metrics. Written to the shared + # host directory that node-exporter's textfile collector + # scrapes (DEV-494). Atomic write: build the file with a + # `.tmp` extension (ignored by node-exporter) and rename + # into place, so a mid-write read never surfaces a + # truncated sample. { echo "backup_loki_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)" echo "backup_loki_timestamp_seconds $(date +%s)" echo "backup_loki_check_status ${CHECK_STATUS}" echo "restic_repo_size_bytes{repo=\"loki\"} ${REPO_SIZE_BYTES}" - } > /metrics/backup_loki.prom + } > /metrics/backup_loki.prom.tmp + mv /metrics/backup_loki.prom.tmp /metrics/backup_loki.prom echo "=== backup-loki-restic finished at $(date -u +%FT%TZ) ===" exit ${CHECK_STATUS} @@ -173,6 +175,13 @@ spec: persistentVolumeClaim: claimName: loki-storage-encrypted - name: metrics - emptyDir: {} + hostPath: + # node-exporter's textfile-collector directory + # (DEV-494). `DirectoryOrCreate` lets kubelet create + # the dir on the current node if it does not yet + # exist — nodes were pre-created out-of-band, this + # is a safety net. + path: /var/lib/node_exporter/textfile_collector + type: DirectoryOrCreate - name: cache emptyDir: {} diff --git a/apps/monitoring/backup-restic-alerts.test.yaml b/apps/monitoring/backup-restic-alerts.test.yaml index 095fdbf..a961aa8 100644 --- a/apps/monitoring/backup-restic-alerts.test.yaml +++ b/apps/monitoring/backup-restic-alerts.test.yaml @@ -37,6 +37,8 @@ tests: values: '-104400x60' - series: 'backup_k8s_resources_timestamp_seconds' values: '-104400x60' + - series: 'backup_prometheus_timestamp_seconds' + values: '-104400x60' alert_rule_test: - eval_time: 30m alertname: BackupLokiStale @@ -83,6 +85,21 @@ tests: `backup_k8s_resources_timestamp_seconds` sample in more than 28 hours. Expected daily at 02:00 UTC. Runbook: docs/monitoring/restic-restore.md + - eval_time: 30m + alertname: BackupPrometheusStale + exp_alerts: + - exp_labels: + severity: warning + service: monitoring + component: backup + repo: prometheus + exp_annotations: + summary: "Prometheus restic backup is stale (>28h)" + description: | + `prometheus-backup` has not written a fresh + `backup_prometheus_timestamp_seconds` sample in more + than 28 hours. Expected daily at 03:30 UTC. + Runbook: docs/monitoring/restic-restore.md # --------------------------------------------------------------------------- # 2. Freshness alerts stay silent when timestamp is fresh (<28h). @@ -97,6 +114,8 @@ tests: values: '-3600x60' - series: 'backup_k8s_resources_timestamp_seconds' values: '-3600x60' + - series: 'backup_prometheus_timestamp_seconds' + values: '-3600x60' alert_rule_test: - eval_time: 30m alertname: BackupLokiStale @@ -107,6 +126,9 @@ tests: - eval_time: 30m alertname: BackupK8sResourcesStale exp_alerts: [] + - eval_time: 30m + alertname: BackupPrometheusStale + exp_alerts: [] # --------------------------------------------------------------------------- # 3. Check-status alerts fire when the metric is non-zero. @@ -119,6 +141,8 @@ tests: values: '2x30' - series: 'backup_k8s_resources_check_status' values: '1x30' + - series: 'backup_prometheus_check_status' + values: '1x30' alert_rule_test: - eval_time: 15m alertname: BackupLokiCheckFailed @@ -163,6 +187,21 @@ tests: exit code 1 on the k8s-resources repository. Runbook: docs/monitoring/restic-restore.md + - eval_time: 15m + alertname: BackupPrometheusCheckFailed + exp_alerts: + - exp_labels: + severity: warning + service: monitoring + component: backup + repo: prometheus + exp_annotations: + summary: "restic check failed on prometheus repo" + description: | + `restic check --read-data-subset=5%` returned + exit code 1 on the prometheus + repository. + Runbook: docs/monitoring/restic-restore.md # --------------------------------------------------------------------------- # 4. Check-status alerts stay silent on 0. @@ -175,6 +214,8 @@ tests: values: '0x30' - series: 'backup_k8s_resources_check_status' values: '0x30' + - series: 'backup_prometheus_check_status' + values: '0x30' alert_rule_test: - eval_time: 15m alertname: BackupLokiCheckFailed @@ -185,6 +226,9 @@ tests: - eval_time: 15m alertname: BackupK8sResourcesCheckFailed exp_alerts: [] + - eval_time: 15m + alertname: BackupPrometheusCheckFailed + exp_alerts: [] # --------------------------------------------------------------------------- # 5. Repo-size alert fires when >20 GiB (21474836480 bytes). diff --git a/apps/monitoring/backup-restic-alerts.yaml b/apps/monitoring/backup-restic-alerts.yaml index d3fc386..83615e5 100644 --- a/apps/monitoring/backup-restic-alerts.yaml +++ b/apps/monitoring/backup-restic-alerts.yaml @@ -14,11 +14,19 @@ # by the same CronJob step (see backup-*-restic-cronjob.yaml, # `restic stats --json --mode raw-data`). # -# The metrics are written to an emptyDir at /metrics per job. -# They become scrapeable once node-exporter's textfile collector -# path is wired up — tracked as a follow-up. `promtool test rules` -# against apps/monitoring/backup-restic-alerts.test.yaml verifies -# the rules with synthetic samples in the meantime. +# The metrics are written to the node's textfile-collector directory +# (`/var/lib/node_exporter/textfile_collector`) and scraped by the +# kube-prometheus-stack node-exporter DaemonSet (DEV-494). +# +# Because a CronJob may run on a different worker across days (loki +# backup follows the loki pod; k8s-resources is unpinned), stale +# `.prom` files can linger on nodes the job has since left. That +# would leave a per-node series with an old timestamp/check-status +# indefinitely. The alerts below aggregate across instances so a +# single fresh sample from the node where the job currently runs is +# enough to keep the freshness alert quiet, and check/size alerts +# fire when *any* node reports a bad value (which is the correct +# behaviour — a recent failure is still a signal). apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: @@ -34,7 +42,10 @@ spec: interval: 60s rules: - alert: BackupLokiStale - expr: time() - backup_loki_timestamp_seconds > 28 * 3600 + # `max()` collapses per-node samples so a stale `.prom` file + # left behind on a node the loki backup no longer runs on + # does not fire this alert; the freshest sample wins. + expr: time() - max(backup_loki_timestamp_seconds) > 28 * 3600 for: 15m labels: severity: warning @@ -49,7 +60,7 @@ spec: 28 hours. Expected daily at 03:00 UTC. Runbook: docs/monitoring/restic-restore.md - alert: BackupGrafanaStale - expr: time() - backup_grafana_timestamp_seconds > 28 * 3600 + expr: time() - max(backup_grafana_timestamp_seconds) > 28 * 3600 for: 15m labels: severity: warning @@ -64,7 +75,7 @@ spec: than 28 hours. Expected daily at 03:15 UTC. Runbook: docs/monitoring/restic-restore.md - alert: BackupK8sResourcesStale - expr: time() - backup_k8s_resources_timestamp_seconds > 28 * 3600 + expr: time() - max(backup_k8s_resources_timestamp_seconds) > 28 * 3600 for: 15m labels: severity: warning @@ -78,6 +89,21 @@ spec: `backup_k8s_resources_timestamp_seconds` sample in more than 28 hours. Expected daily at 02:00 UTC. Runbook: docs/monitoring/restic-restore.md + - alert: BackupPrometheusStale + expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600 + for: 15m + labels: + severity: warning + service: monitoring + component: backup + repo: prometheus + annotations: + summary: "Prometheus restic backup is stale (>28h)" + description: | + `prometheus-backup` has not written a fresh + `backup_prometheus_timestamp_seconds` sample in more + than 28 hours. Expected daily at 03:30 UTC. + Runbook: docs/monitoring/restic-restore.md - name: backup-restic.integrity interval: 60s rules: @@ -124,6 +150,21 @@ spec: exit code {{ $value }} on the k8s-resources repository. Runbook: docs/monitoring/restic-restore.md + - alert: BackupPrometheusCheckFailed + expr: backup_prometheus_check_status != 0 + for: 5m + labels: + severity: warning + service: monitoring + component: backup + repo: prometheus + annotations: + summary: "restic check failed on prometheus repo" + description: | + `restic check --read-data-subset=5%` returned + exit code {{ $value }} on the prometheus + repository. + Runbook: docs/monitoring/restic-restore.md - name: backup-restic.size interval: 60s rules: diff --git a/apps/monitoring/prometheus-backup-cronjob.yaml b/apps/monitoring/prometheus-backup-cronjob.yaml index a8428f3..705fab2 100644 --- a/apps/monitoring/prometheus-backup-cronjob.yaml +++ b/apps/monitoring/prometheus-backup-cronjob.yaml @@ -1,16 +1,43 @@ --- -# Prometheus data backup (DEV-465). +# Prometheus data backup via restic to Hetzner Object Storage +# (DEV-492, DEV-482 Option 4). Replaces the DEV-465 rclone-sync job so +# every monitoring backup ships client-side-encrypted; Hetzner Object +# Storage has no SSE-S3/SSE-KMS, so the previous plaintext-at-rest +# object layout was the only remaining gap. # -# Prometheus data lives on the RWO PVC `prometheus-data-encrypted` in -# namespace `monitoring`. That PVC is mounted by the Prometheus pod, and -# the Hetzner CSI volume can only be attached to one node at a time. +# Streams the RWO PVC `prometheus-data-encrypted` (mounted read-only) +# into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, a client-side +# encrypted restic repository (tag=`prometheus`, host=`k3s`). # -# This CronJob co-schedules with the Prometheus pod via podAffinity, so it -# lands on whichever node currently holds `prometheus-data-encrypted`. The -# PVC is mounted read-only alongside the running Prometheus pod (RWO permits -# additional read-only mounts on the same node) and streamed to Hetzner S3 -# via rclone under `basicstack-backup/prometheus/prometheus-/`. Old -# snapshots are pruned after 7 days. +# Node scheduling matches the previous job: podAffinity co-schedules +# with the Prometheus pod (app=prometheus, topology +# kubernetes.io/hostname). Hetzner CSI RWO permits additional read-only +# mounts on the node that holds the PVC's VolumeAttachment, so this +# survives Prometheus being rescheduled to a different worker. +# +# Prometheus TSDB compaction race +# -------------------------------- +# Prometheus rewrites the on-disk store roughly every 2 h: it creates a +# new block dir, then deletes the source dirs. restic walks the source +# tree once and may catch a file that disappeared mid-walk; restic +# 0.17.3 exits 3 ("at least one source file could not be read") in +# that case, and the snapshot excludes only the missing file. The +# next daily run picks up the successor block, so the race is not a +# data-loss risk — but we must not treat exit 3 as a hard failure, or +# the daily job will alert-flap. +# +# Mitigation: +# - exclude `wal/*` (WAL is replayed from a fresh instance on +# restart; we accept losing the last ~15 s of ingested samples +# rather than snapshotting a moving segment) +# - exclude `chunks_head/*` (in-memory head block; ephemeral, would +# be rebuilt from WAL which we do not keep) +# - exclude Prometheus lock/scratch files (`lock`, `queries.active`, +# `*.tmp`, `lost+found/*`) +# - treat restic exit code 3 as a soft warning (log, continue); +# any other non-zero exit is still fatal +# - restore drill re-runs `promtool tsdb analyze` against every +# block so a corrupted snapshot is caught end-to-end apiVersion: batch/v1 kind: CronJob metadata: @@ -19,8 +46,9 @@ metadata: labels: app: backup type: prometheus + backend: restic spec: - schedule: "30 3 * * *" # daily 03:30, offset from the loki/grafana restic runs + schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana/k8s-resources concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 @@ -29,19 +57,18 @@ spec: labels: app: backup type: prometheus + backend: restic spec: backoffLimit: 2 - activeDeadlineSeconds: 3600 # 60 min hard cap; Prometheus data is ~9GB + activeDeadlineSeconds: 3600 template: metadata: labels: app: backup type: prometheus + backend: restic spec: restartPolicy: OnFailure - # Co-schedule with the Prometheus pod so the RWO PVC attaches on - # the same node. This survives Prometheus being rescheduled to a - # different worker (the backup follows). affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: @@ -53,134 +80,24 @@ spec: - prometheus topologyKey: kubernetes.io/hostname containers: - - name: prometheus-backup - image: rclone/rclone:1.68 - command: - - /bin/sh - - -c - - | - set -eu - DATE=$(date +%Y%m%d-%H%M%S) - echo "=== Prometheus backup started at $(date) (target prefix: prometheus-${DATE}) ===" - - mkdir -p /root/.config/rclone - cat > /root/.config/rclone/rclone.conf </dev/null | cut -f1 || echo 0) - SOURCE_HUMAN=$(du -sh /source 2>/dev/null | cut -f1 || echo unknown) - echo "Source /source size: ${SOURCE_HUMAN} (${SOURCE_BYTES} bytes)" - - # Prometheus TSDB is largely immutable chunk files plus an - # append-only WAL. rclone sync is safe with the database - # live: on restore the WAL is replayed. HOWEVER Prometheus - # compacts blocks every ~2h and deletes their source dirs, - # which races with the copy and produces "no such file or - # directory" errors mid-run. Those are expected and do - # not indicate data loss — the compacted successor blocks - # are picked up on the same or the next daily run. We - # therefore do not fail the job on rclone's non-zero exit - # from those transient errors; instead we validate the - # backup by comparing destination size to source (must be - # >= 80% of source bytes and > 100 MiB). - DEST="hetzner-s3:${S3_BUCKET}/prometheus/prometheus-${DATE}/" - echo "Streaming Prometheus data to ${DEST} ..." - RCLONE_EXIT=0 - rclone sync /source "${DEST}" \ - --transfers 4 \ - --checkers 4 \ - --stats 30s \ - --stats-log-level NOTICE \ - --s3-chunk-size 32M \ - --s3-upload-concurrency 4 \ - --retries 3 \ - --retries-sleep 30s || RCLONE_EXIT=$? - echo "rclone sync exit code: ${RCLONE_EXIT}" - - echo "Measuring destination size..." - DEST_BYTES=$(rclone size "${DEST}" --json 2>/dev/null | \ - sed -n 's/.*"bytes":\s*\([0-9]\+\).*/\1/p' | head -1) - DEST_BYTES=${DEST_BYTES:-0} - DEST_HUMAN=$(rclone size "${DEST}" 2>/dev/null | \ - grep -oE 'Total size:.*' || echo "Total size: unknown") - echo "Destination bytes: ${DEST_BYTES}" - echo "Destination summary: ${DEST_HUMAN}" - - MIN_ACCEPTABLE=$(( SOURCE_BYTES * 80 / 100 )) - FLOOR=104857600 # 100 MiB absolute floor - echo "Acceptance threshold: dest >= ${MIN_ACCEPTABLE} bytes and > ${FLOOR} bytes" - if [ "${DEST_BYTES}" -lt "${FLOOR}" ]; then - echo "ERROR: destination is below hard floor (100 MiB) — backup failed." - exit 2 - fi - if [ "${DEST_BYTES}" -lt "${MIN_ACCEPTABLE}" ]; then - echo "ERROR: destination is < 80% of source (${DEST_BYTES} < ${MIN_ACCEPTABLE}) — backup incomplete." - exit 3 - fi - echo "OK: destination size acceptable." - - # Prune by prefix name, NOT by file mtime. rclone preserves - # each source file's mtime on upload, and Prometheus TSDB - # chunk files retain very old mtimes (weeks-old blocks), - # so `rclone delete --min-age 7d` would eat the just- - # uploaded backup. Every top-level prefix is named - # `prometheus-YYYYMMDD-HHMMSS`, so we compare the encoded - # date to a 7-day threshold and purge whole prefixes. - echo "Pruning prometheus backup prefixes older than 7 days..." - # BusyBox date lacks GNU's `-d "7 days ago"` and BSD's `-v -7d`, - # so compute the cutoff via @epoch which BusyBox does support. - CUTOFF_EPOCH=$(( $(date +%s) - 7 * 86400 )) - CUTOFF=$(date -d "@${CUTOFF_EPOCH}" +%Y%m%d) - echo "Cutoff (delete prefixes with date < ${CUTOFF}):" - KEEP=0 - PRUNE=0 - for PREFIX in $(rclone lsf --dirs-only "hetzner-s3:${S3_BUCKET}/prometheus/" 2>/dev/null | sed 's:/$::'); do - case "${PREFIX}" in - prometheus-*) - PDATE=$(echo "${PREFIX}" | sed -n 's/^prometheus-\([0-9]\{8\}\)-.*/\1/p') - if [ -z "${PDATE}" ]; then - echo " SKIP ${PREFIX} (unparseable name)" - continue - fi - if [ "${PDATE}" -lt "${CUTOFF}" ]; then - echo " PURGE ${PREFIX} (date ${PDATE} < ${CUTOFF})" - rclone purge "hetzner-s3:${S3_BUCKET}/prometheus/${PREFIX}" \ - || echo " WARN purge failed for ${PREFIX} (continuing)" - PRUNE=$((PRUNE + 1)) - else - echo " KEEP ${PREFIX} (date ${PDATE})" - KEEP=$((KEEP + 1)) - fi - ;; - *) - echo " SKIP ${PREFIX} (not a prometheus- prefix)" - ;; - esac - done - echo "Prune summary: kept ${KEEP}, purged ${PRUNE}." - - echo "Post-run inventory (prometheus/ prefixes):" - rclone lsd "hetzner-s3:${S3_BUCKET}/prometheus/" || true - - echo "=== Prometheus backup completed at $(date) ===" + - name: restic + image: restic/restic:0.17.3 env: - - name: S3_ACCESS_KEY + - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: monitoring-s3-backup key: access-key - - name: S3_SECRET_KEY + - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: monitoring-s3-backup key: secret-key + - name: RESTIC_PASSWORD + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: restic-password - name: S3_ENDPOINT valueFrom: secretKeyRef: @@ -191,18 +108,121 @@ spec: secretKeyRef: name: monitoring-s3-backup key: bucket + - name: RESTIC_REPOSITORY + value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/prometheus" + command: + - /bin/sh + - -c + - | + set -eu + echo "=== backup-prometheus-restic started at $(date -u +%FT%TZ) ===" + echo "Repository: ${RESTIC_REPOSITORY}" + + # First-run tolerance: init if the repo isn't there yet. + if restic snapshots >/dev/null 2>&1; then + echo "Repo exists, skipping init." + else + echo "Repo missing, initialising..." + restic init + fi + + # See "Prometheus TSDB compaction race" in the file + # header. WAL + chunks_head are excluded on purpose; + # exit code 3 (source file vanished mid-walk during a + # compaction) is accepted and reported, any other + # non-zero exit is fatal. + echo "--- restic backup /source (exclude wal/chunks_head + lock files) ---" + BACKUP_STATUS=0 + restic backup /source \ + --tag prometheus \ + --host k3s \ + --exclude 'wal/*' \ + --exclude 'chunks_head/*' \ + --exclude 'lock' \ + --exclude 'queries.active' \ + --exclude 'lost+found/*' \ + --exclude '*.tmp' || BACKUP_STATUS=$? + echo "restic backup exit: ${BACKUP_STATUS}" + if [ "${BACKUP_STATUS}" -eq 0 ]; then + echo "backup: all files captured cleanly" + elif [ "${BACKUP_STATUS}" -eq 3 ]; then + echo "backup: exit 3 (source files vanished mid-walk) — expected under Prometheus compaction, continuing" + else + echo "backup: FATAL exit ${BACKUP_STATUS} (not compaction-race)" + exit ${BACKUP_STATUS} + fi + + echo "--- restic forget/prune ---" + restic forget --tag prometheus \ + --keep-daily 7 \ + --keep-weekly 4 \ + --keep-monthly 6 \ + --prune + + echo "--- restic check --read-data-subset=5% ---" + CHECK_STATUS=0 + restic check --read-data-subset=5% || CHECK_STATUS=$? + echo "restic check exit: ${CHECK_STATUS}" + + echo "--- restic stats (repo size) ---" + REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \ + | grep -oE '"total_size":[0-9]+' \ + | head -1 \ + | cut -d: -f2) + REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0} + echo "restic repo size: ${REPO_SIZE_BYTES} bytes" + + # Textfile-collector metrics. Same hostPath pattern as + # the loki/grafana/k8s-resources siblings (DEV-494) — + # written atomically via `.tmp` + rename so a mid-write + # read never surfaces a truncated sample. + # backup_prometheus_success rolls in both stages: the + # backup step (accepting exit 3) and restic check. + BACKUP_OK=0 + if [ "${BACKUP_STATUS}" -eq 0 ] || [ "${BACKUP_STATUS}" -eq 3 ]; then + BACKUP_OK=1 + fi + SUCCESS=0 + if [ "${BACKUP_OK}" -eq 1 ] && [ "${CHECK_STATUS}" -eq 0 ]; then + SUCCESS=1 + fi + { + echo "backup_prometheus_success ${SUCCESS}" + echo "backup_prometheus_timestamp_seconds $(date +%s)" + echo "backup_prometheus_check_status ${CHECK_STATUS}" + echo "backup_prometheus_backup_status ${BACKUP_STATUS}" + echo "restic_repo_size_bytes{repo=\"prometheus\"} ${REPO_SIZE_BYTES}" + } > /metrics/backup_prometheus.prom.tmp + mv /metrics/backup_prometheus.prom.tmp /metrics/backup_prometheus.prom + + echo "=== backup-prometheus-restic finished at $(date -u +%FT%TZ) ===" + exit ${CHECK_STATUS} volumeMounts: - name: prometheus-data mountPath: /source readOnly: true + - name: metrics + mountPath: /metrics + - name: cache + mountPath: /root/.cache/restic resources: + # Prometheus TSDB is ~8-10 GiB; give restic room to + # burst during pack/check but keep steady-state small. requests: cpu: 100m - memory: 128Mi + memory: 256Mi limits: - cpu: 1000m - memory: 512Mi + cpu: 1500m + memory: 1Gi volumes: - name: prometheus-data persistentVolumeClaim: claimName: prometheus-data-encrypted + - name: metrics + hostPath: + # node-exporter's textfile-collector directory + # (DEV-494). See sibling loki cronjob for detail. + path: /var/lib/node_exporter/textfile_collector + type: DirectoryOrCreate + - name: cache + emptyDir: {} diff --git a/apps/observability/patches/README.md b/apps/observability/patches/README.md new file mode 100644 index 0000000..89a3c20 --- /dev/null +++ b/apps/observability/patches/README.md @@ -0,0 +1,38 @@ +# observability patches + +Strategic-merge patches applied on top of Helm-managed observability +resources. Each file is idempotent (re-applying is a no-op) and is +reasserted by hand rather than by a controller, so re-run after any +`helm upgrade` of the affected release. + +## `node-exporter-textfile-collector.yaml` (DEV-494) + +Enables the node-exporter textfile collector on the +`kube-prometheus-stack-prometheus-node-exporter` DaemonSet by: + +1. adding `--collector.textfile.directory=/host/textfile_collector` + to the container args, and +2. mounting the host directory `/var/lib/node_exporter/textfile_collector` + read-only at `/host/textfile_collector` (`hostPath` type + `DirectoryOrCreate`, kubelet creates it on nodes where the + directory does not exist yet). + +The four monitoring backup CronJobs in `apps/monitoring/` write their +textfile-collector `.prom` files into that same host directory, so the +metrics surface in Prometheus via node-exporter's normal scrape. + +Apply / re-apply: + +```bash +kubectl -n observability patch daemonset \ + kube-prometheus-stack-prometheus-node-exporter \ + --type=strategic \ + --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml +kubectl -n observability rollout status daemonset \ + kube-prometheus-stack-prometheus-node-exporter +``` + +The kube-prometheus-stack chart is not currently tracked in ArgoCD; +if it moves under GitOps, fold these values into the chart values as +`prometheus-node-exporter.extraArgs` + `.extraHostVolumeMounts` +instead of maintaining this patch. diff --git a/apps/observability/patches/node-exporter-textfile-collector.yaml b/apps/observability/patches/node-exporter-textfile-collector.yaml new file mode 100644 index 0000000..1ec8b69 --- /dev/null +++ b/apps/observability/patches/node-exporter-textfile-collector.yaml @@ -0,0 +1,70 @@ +--- +# Strategic-merge patch enabling node-exporter's textfile collector on +# the kube-prometheus-stack node-exporter DaemonSet (DEV-494). +# +# The chart is Helm-managed (release `kube-prometheus-stack` in +# namespace `observability`, chart kube-prometheus-stack-86.2.2 / +# prometheus-node-exporter-4.55.0) and is NOT currently tracked in +# ArgoCD, so a direct DaemonSet patch is the pragmatic wiring path. +# The change: +# +# 1. adds the `--collector.textfile.directory=/host/textfile_collector` +# arg to the node-exporter container, and +# 2. mounts the host directory `/var/lib/node_exporter/textfile_collector` +# read-only at `/host/textfile_collector` (type DirectoryOrCreate so +# kubelet creates the dir on nodes where it does not yet exist). +# +# Apply with: +# kubectl -n observability patch daemonset \ +# kube-prometheus-stack-prometheus-node-exporter \ +# --type=strategic \ +# --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml +# +# If the Helm release is ever `helm upgrade`d without folding these +# values into the chart values, this patch will be reverted — re-apply +# it after the upgrade (or move it into a repo-owned values file). +spec: + template: + spec: + containers: + - name: node-exporter + args: + - --path.procfs=/host/proc + - --path.sysfs=/host/sys + - --path.rootfs=/host/root + - --path.udev.data=/host/root/run/udev/data + - --web.listen-address=[$(HOST_IP)]:9100 + - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/) + - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$ + - --collector.textfile.directory=/host/textfile_collector + volumeMounts: + - mountPath: /host/proc + name: proc + readOnly: true + - mountPath: /host/sys + name: sys + readOnly: true + - mountPath: /host/root + mountPropagation: HostToContainer + name: root + readOnly: true + - mountPath: /host/textfile_collector + name: textfile-collector + readOnly: true + volumes: + - hostPath: + path: /proc + type: "" + name: proc + - hostPath: + path: /sys + type: "" + name: sys + - hostPath: + path: / + type: "" + name: root + - hostPath: + path: /var/lib/node_exporter/textfile_collector + type: DirectoryOrCreate + name: textfile-collector