stack.basicstack.de/apps/monitoring/README.md

# monitoring — backup CronJobs

Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.

- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block.
- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos (loki/grafana/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)). The `k8s-resources` restic repo was superseded by Velero cluster-reconstruction backups and retired in [DEV-576](/DEV/issues/DEV-576).

## Shared SealedSecret

All four restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys:

| Key              | Purpose                                                                                                          |
|------------------|------------------------------------------------------------------------------------------------------------------|
| `access-key`     | Hetzner Object Storage access key ID                                                                             |
| `secret-key`     | Hetzner Object Storage secret access key                                                                         |
| `endpoint`       | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`)                                                        |
| `bucket`         | Bucket name (single bucket, per-prefix repos)                                                                    |
| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all three remaining repos (loki/grafana/prometheus) — rotating rewrites the key file on every repo. |

## Restore

Restore procedure for all four restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16, loki/k8s-resources) passed — see [DEV-488](/DEV/issues/DEV-488). Prometheus repo added and drilled same day — see [DEV-492](/DEV/issues/DEV-492).

## Emitted metrics (textfile-collector format)

Each restic CronJob writes `/metrics/backup_<kind>.prom` (atomic — `.prom.tmp` + `mv`) into a `hostPath` volume mounted at the node-exporter textfile-collector directory (`/var/lib/node_exporter/textfile_collector`). The kube-prometheus-stack node-exporter DaemonSet has `--collector.textfile.directory=/host/textfile_collector` enabled ([DEV-494](/DEV/issues/DEV-494), applied via [`apps/observability/patches/node-exporter-textfile-collector.yaml`](../observability/patches/node-exporter-textfile-collector.yaml)) and surfaces those samples in Prometheus.

| Metric                                     | Emitted by                                                                       |
|--------------------------------------------|----------------------------------------------------------------------------------|
| `backup_<kind>_success` (0/1)              | `backup-{loki,grafana}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml`         |
| `backup_<kind>_timestamp_seconds`          | same                                                                             |
| `backup_<kind>_check_status` (exit code)   | same — from `restic check --read-data-subset=5%`                                 |
| `backup_prometheus_backup_status`          | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) |
| `restic_repo_size_bytes{repo="<kind>"}`    | same — from `restic stats --json --mode raw-data` (added in DEV-490)             |

**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.

## Alerts

`backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`):

- `BackupLokiStale` / `BackupGrafanaStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
- `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions.

All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally:

```bash
awk '/^spec:/{f=1;next} f{sub(/^  /,"");print}' \
  apps/monitoring/backup-restic-alerts.yaml > /tmp/backup-restic-rules.yaml
promtool check rules /tmp/backup-restic-rules.yaml
promtool test rules  apps/monitoring/backup-restic-alerts.test.yaml
```

The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers. The `backup-k8s-resources` CronJob and its restic repo were retired in [DEV-576](/DEV/issues/DEV-576) after Velero `cluster-all` took over cluster-manifest coverage.