ef62dde67c
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
70 lines
2.8 KiB
YAML
70 lines
2.8 KiB
YAML
---
|
|
# Strategic-merge patch enabling node-exporter's textfile collector on
|
|
# the kube-prometheus-stack node-exporter DaemonSet (DEV-494).
|
|
#
|
|
# The chart is Helm-managed (release `kube-prometheus-stack` in
|
|
# namespace `observability`, chart kube-prometheus-stack-86.2.2 /
|
|
# prometheus-node-exporter-4.55.0) and is NOT currently tracked in
|
|
# ArgoCD, so a direct DaemonSet patch is the pragmatic wiring path.
|
|
# The change:
|
|
#
|
|
# 1. adds the `--collector.textfile.directory=/host/textfile_collector`
|
|
# arg to the node-exporter container, and
|
|
# 2. mounts the host directory `/var/lib/node_exporter/textfile_collector`
|
|
# read-only at `/host/textfile_collector` (type DirectoryOrCreate so
|
|
# kubelet creates the dir on nodes where it does not yet exist).
|
|
#
|
|
# Apply with:
|
|
# kubectl -n observability patch daemonset \
|
|
# kube-prometheus-stack-prometheus-node-exporter \
|
|
# --type=strategic \
|
|
# --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
|
|
#
|
|
# If the Helm release is ever `helm upgrade`d without folding these
|
|
# values into the chart values, this patch will be reverted — re-apply
|
|
# it after the upgrade (or move it into a repo-owned values file).
|
|
spec:
|
|
template:
|
|
spec:
|
|
containers:
|
|
- name: node-exporter
|
|
args:
|
|
- --path.procfs=/host/proc
|
|
- --path.sysfs=/host/sys
|
|
- --path.rootfs=/host/root
|
|
- --path.udev.data=/host/root/run/udev/data
|
|
- --web.listen-address=[$(HOST_IP)]:9100
|
|
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
|
|
- --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$
|
|
- --collector.textfile.directory=/host/textfile_collector
|
|
volumeMounts:
|
|
- mountPath: /host/proc
|
|
name: proc
|
|
readOnly: true
|
|
- mountPath: /host/sys
|
|
name: sys
|
|
readOnly: true
|
|
- mountPath: /host/root
|
|
mountPropagation: HostToContainer
|
|
name: root
|
|
readOnly: true
|
|
- mountPath: /host/textfile_collector
|
|
name: textfile-collector
|
|
readOnly: true
|
|
volumes:
|
|
- hostPath:
|
|
path: /proc
|
|
type: ""
|
|
name: proc
|
|
- hostPath:
|
|
path: /sys
|
|
type: ""
|
|
name: sys
|
|
- hostPath:
|
|
path: /
|
|
type: ""
|
|
name: root
|
|
- hostPath:
|
|
path: /var/lib/node_exporter/textfile_collector
|
|
type: DirectoryOrCreate
|
|
name: textfile-collector
|