stack.basicstack.de/apps/observability/patches/node-exporter-textfile-collector.yaml

71 lines
2.8 KiB
YAML
Raw Normal View History

feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494) Turns the four monitoring backup CronJobs' .prom output into scrapeable Prometheus series so the DEV-490 alerts finally evaluate against live data. - apps/observability/patches/node-exporter-textfile-collector.yaml: strategic-merge patch on the kube-prometheus-stack node-exporter DS that adds `--collector.textfile.directory=/host/textfile_collector` and mounts `/var/lib/node_exporter/textfile_collector` read-only. Chart isn't tracked in ArgoCD, so we keep the patch under version control and re-apply after any helm upgrade (see patches/README.md). - apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml: swap `emptyDir` /metrics for a `hostPath` on the same directory (`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter never reads a truncated sample. - apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h` for all four freshness alerts so a stale `.prom` left on a node the job has since left does not fire the freshness pager. - apps/monitoring/README.md: drop the "once wired" caveat; document the on-node directory, atomic write, cross-node staleness rationale. Verified end-to-end with a synthetic `kubectl create job --from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4, .prom file materialized in /var/lib/node_exporter/textfile_collector, and Prometheus returned all four metric families (`backup_k8s_resources_success=1`, `..._timestamp_seconds`, `..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`). `time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s against a fresh run. `promtool check rules` + `promtool test rules` still pass (9 rules, 5 scenarios). Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
---
# Strategic-merge patch enabling node-exporter's textfile collector on
# the kube-prometheus-stack node-exporter DaemonSet (DEV-494).
#
# The chart is Helm-managed (release `kube-prometheus-stack` in
# namespace `observability`, chart kube-prometheus-stack-86.2.2 /
# prometheus-node-exporter-4.55.0) and is NOT currently tracked in
# ArgoCD, so a direct DaemonSet patch is the pragmatic wiring path.
# The change:
#
# 1. adds the `--collector.textfile.directory=/host/textfile_collector`
# arg to the node-exporter container, and
# 2. mounts the host directory `/var/lib/node_exporter/textfile_collector`
# read-only at `/host/textfile_collector` (type DirectoryOrCreate so
# kubelet creates the dir on nodes where it does not yet exist).
#
# Apply with:
# kubectl -n observability patch daemonset \
# kube-prometheus-stack-prometheus-node-exporter \
# --type=strategic \
# --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
#
# If the Helm release is ever `helm upgrade`d without folding these
# values into the chart values, this patch will be reverted — re-apply
# it after the upgrade (or move it into a repo-owned values file).
spec:
template:
spec:
containers:
- name: node-exporter
args:
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/host/root
- --path.udev.data=/host/root/run/udev/data
- --web.listen-address=[$(HOST_IP)]:9100
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
- --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$
- --collector.textfile.directory=/host/textfile_collector
volumeMounts:
- mountPath: /host/proc
name: proc
readOnly: true
- mountPath: /host/sys
name: sys
readOnly: true
- mountPath: /host/root
mountPropagation: HostToContainer
name: root
readOnly: true
- mountPath: /host/textfile_collector
name: textfile-collector
readOnly: true
volumes:
- hostPath:
path: /proc
type: ""
name: proc
- hostPath:
path: /sys
type: ""
name: sys
- hostPath:
path: /
type: ""
name: root
- hostPath:
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
name: textfile-collector