stack.basicstack.de/apps/observability/patches/README.md

39 lines
1.5 KiB
Markdown
Raw Permalink Normal View History

feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494) Turns the four monitoring backup CronJobs' .prom output into scrapeable Prometheus series so the DEV-490 alerts finally evaluate against live data. - apps/observability/patches/node-exporter-textfile-collector.yaml: strategic-merge patch on the kube-prometheus-stack node-exporter DS that adds `--collector.textfile.directory=/host/textfile_collector` and mounts `/var/lib/node_exporter/textfile_collector` read-only. Chart isn't tracked in ArgoCD, so we keep the patch under version control and re-apply after any helm upgrade (see patches/README.md). - apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml: swap `emptyDir` /metrics for a `hostPath` on the same directory (`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter never reads a truncated sample. - apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h` for all four freshness alerts so a stale `.prom` left on a node the job has since left does not fire the freshness pager. - apps/monitoring/README.md: drop the "once wired" caveat; document the on-node directory, atomic write, cross-node staleness rationale. Verified end-to-end with a synthetic `kubectl create job --from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4, .prom file materialized in /var/lib/node_exporter/textfile_collector, and Prometheus returned all four metric families (`backup_k8s_resources_success=1`, `..._timestamp_seconds`, `..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`). `time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s against a fresh run. `promtool check rules` + `promtool test rules` still pass (9 rules, 5 scenarios). Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
# observability patches
Strategic-merge patches applied on top of Helm-managed observability
resources. Each file is idempotent (re-applying is a no-op) and is
reasserted by hand rather than by a controller, so re-run after any
`helm upgrade` of the affected release.
## `node-exporter-textfile-collector.yaml` (DEV-494)
Enables the node-exporter textfile collector on the
`kube-prometheus-stack-prometheus-node-exporter` DaemonSet by:
1. adding `--collector.textfile.directory=/host/textfile_collector`
to the container args, and
2. mounting the host directory `/var/lib/node_exporter/textfile_collector`
read-only at `/host/textfile_collector` (`hostPath` type
`DirectoryOrCreate`, kubelet creates it on nodes where the
directory does not exist yet).
The four monitoring backup CronJobs in `apps/monitoring/` write their
textfile-collector `.prom` files into that same host directory, so the
metrics surface in Prometheus via node-exporter's normal scrape.
Apply / re-apply:
```bash
kubectl -n observability patch daemonset \
kube-prometheus-stack-prometheus-node-exporter \
--type=strategic \
--patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
kubectl -n observability rollout status daemonset \
kube-prometheus-stack-prometheus-node-exporter
```
The kube-prometheus-stack chart is not currently tracked in ArgoCD;
if it moves under GitOps, fold these values into the chart values as
`prometheus-node-exporter.extraArgs` + `.extraHostVolumeMounts`
instead of maintaining this patch.