basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/apps/observability/patches
History
CTO Agent ef62dde67c feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494) 
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.

- apps/observability/patches/node-exporter-textfile-collector.yaml:
  strategic-merge patch on the kube-prometheus-stack node-exporter DS
  that adds `--collector.textfile.directory=/host/textfile_collector`
  and mounts `/var/lib/node_exporter/textfile_collector` read-only.
  Chart isn't tracked in ArgoCD, so we keep the patch under version
  control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
  swap `emptyDir` /metrics for a `hostPath` on the same directory
  (`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
  never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
  for all four freshness alerts so a stale `.prom` left on a node the
  job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
  the on-node directory, atomic write, cross-node staleness rationale.

Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
..
node-exporter-textfile-collector.yaml feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494) 2026-08-16 16:21:56 +00:00
README.md feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494) 2026-08-16 16:21:56 +00:00

README.md

observability patches

Strategic-merge patches applied on top of Helm-managed observability resources. Each file is idempotent (re-applying is a no-op) and is reasserted by hand rather than by a controller, so re-run after any helm upgrade of the affected release.

node-exporter-textfile-collector.yaml (DEV-494)

Enables the node-exporter textfile collector on the kube-prometheus-stack-prometheus-node-exporter DaemonSet by:

  1. adding --collector.textfile.directory=/host/textfile_collector to the container args, and
  2. mounting the host directory /var/lib/node_exporter/textfile_collector read-only at /host/textfile_collector (hostPath type DirectoryOrCreate, kubelet creates it on nodes where the directory does not exist yet).

The four monitoring backup CronJobs in apps/monitoring/ write their textfile-collector .prom files into that same host directory, so the metrics surface in Prometheus via node-exporter's normal scrape.

Apply / re-apply:

kubectl -n observability patch daemonset \
  kube-prometheus-stack-prometheus-node-exporter \
  --type=strategic \
  --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
kubectl -n observability rollout status daemonset \
  kube-prometheus-stack-prometheus-node-exporter

The kube-prometheus-stack chart is not currently tracked in ArgoCD; if it moves under GitOps, fold these values into the chart values as prometheus-node-exporter.extraArgs + .extraHostVolumeMounts instead of maintaining this patch.