feat(observability): wire node-exporter textfile collector for backup metrics (DEV-494)
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 16:21:56 +00:00
|
|
|
# observability patches
|
|
|
|
|
|
|
|
|
|
Strategic-merge patches applied on top of Helm-managed observability
|
|
|
|
|
resources. Each file is idempotent (re-applying is a no-op) and is
|
|
|
|
|
reasserted by hand rather than by a controller, so re-run after any
|
|
|
|
|
`helm upgrade` of the affected release.
|
|
|
|
|
|
|
|
|
|
## `node-exporter-textfile-collector.yaml` (DEV-494)
|
|
|
|
|
|
|
|
|
|
Enables the node-exporter textfile collector on the
|
|
|
|
|
`kube-prometheus-stack-prometheus-node-exporter` DaemonSet by:
|
|
|
|
|
|
|
|
|
|
1. adding `--collector.textfile.directory=/host/textfile_collector`
|
|
|
|
|
to the container args, and
|
|
|
|
|
2. mounting the host directory `/var/lib/node_exporter/textfile_collector`
|
|
|
|
|
read-only at `/host/textfile_collector` (`hostPath` type
|
|
|
|
|
`DirectoryOrCreate`, kubelet creates it on nodes where the
|
|
|
|
|
directory does not exist yet).
|
|
|
|
|
|
|
|
|
|
The four monitoring backup CronJobs in `apps/monitoring/` write their
|
|
|
|
|
textfile-collector `.prom` files into that same host directory, so the
|
|
|
|
|
metrics surface in Prometheus via node-exporter's normal scrape.
|
|
|
|
|
|
|
|
|
|
Apply / re-apply:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kubectl -n observability patch daemonset \
|
|
|
|
|
kube-prometheus-stack-prometheus-node-exporter \
|
|
|
|
|
--type=strategic \
|
|
|
|
|
--patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
|
|
|
|
|
kubectl -n observability rollout status daemonset \
|
|
|
|
|
kube-prometheus-stack-prometheus-node-exporter
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The kube-prometheus-stack chart is not currently tracked in ArgoCD;
|
|
|
|
|
if it moves under GitOps, fold these values into the chart values as
|
|
|
|
|
`prometheus-node-exporter.extraArgs` + `.extraHostVolumeMounts`
|
|
|
|
|
instead of maintaining this patch.
|
2026-08-23 12:05:12 +00:00
|
|
|
|
|
|
|
|
## `node-exporter-dns-config.yaml` (DEV-527)
|
|
|
|
|
|
|
|
|
|
Caps the `hostNetwork` node-exporter pod's DNS at three upstream
|
|
|
|
|
servers to silence Kubernetes' `DNSConfigForming` Warning event.
|
|
|
|
|
Hetzner's OS publishes four systemd-resolved upstreams
|
|
|
|
|
(`2a01:4ff:ff00::add:2`, `2a01:4ff:ff00::add:1`, `185.12.64.1`,
|
|
|
|
|
`185.12.64.2`), and kubelet drops the fourth because Kubernetes
|
|
|
|
|
pods are limited to three nameservers.
|
|
|
|
|
|
|
|
|
|
Sets:
|
|
|
|
|
|
|
|
|
|
1. `dnsPolicy: None` so `dnsConfig` is authoritative (with `Default`
|
|
|
|
|
or the auto-coerced `ClusterFirst→Default` on a hostNetwork pod,
|
|
|
|
|
kubelet still merges the node's resolv.conf on top and the fourth
|
|
|
|
|
nameserver keeps re-triggering the warning), and
|
|
|
|
|
2. an explicit `dnsConfig` with the two Hetzner IPv6 anycast entries
|
|
|
|
|
plus the first IPv4 entry, matching the three servers kubelet was
|
|
|
|
|
already picking, plus `edns0`/`trust-ad` resolv.conf options.
|
|
|
|
|
|
|
|
|
|
Apply / re-apply:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kubectl -n observability patch daemonset \
|
|
|
|
|
kube-prometheus-stack-prometheus-node-exporter \
|
|
|
|
|
--type=strategic \
|
|
|
|
|
--patch-file=apps/observability/patches/node-exporter-dns-config.yaml
|
|
|
|
|
kubectl -n observability rollout status daemonset \
|
|
|
|
|
kube-prometheus-stack-prometheus-node-exporter
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If the chart moves under GitOps, fold these values into the chart
|
|
|
|
|
values as `prometheus-node-exporter.dnsPolicy` +
|
|
|
|
|
`.dnsConfig` instead of maintaining this patch.
|
|
|
|
|
|
|
|
|
|
## `coredns-dns-config.yaml` (DEV-527)
|
|
|
|
|
|
|
|
|
|
Companion to the node-exporter patch for the k3s built-in CoreDNS
|
|
|
|
|
Deployment (`kube-system/coredns`, `dnsPolicy: Default`), which also
|
|
|
|
|
triggers the same `DNSConfigForming` warning on every pod restart.
|
|
|
|
|
|
|
|
|
|
Because CoreDNS is a **k3s Addon** whose source manifest lives at
|
|
|
|
|
`/var/lib/rancher/k3s/server/manifests/coredns.yaml` on each control
|
|
|
|
|
plane node, `kubectl patch` alone is NOT durable — the k3s addon
|
|
|
|
|
controller re-applies the source manifest and reverts `dnsPolicy`
|
|
|
|
|
back to `Default`. This patch file is kept in the repo as the
|
|
|
|
|
canonical description of the fix and can be used for a quick manual
|
|
|
|
|
re-apply (until the addon controller next reconciles), but the
|
|
|
|
|
authoritative fix is applied by editing the same `dnsPolicy` /
|
|
|
|
|
`dnsConfig` block into the CoreDNS Deployment stanza of the k3s
|
|
|
|
|
`coredns.yaml` on each CP node. See
|
|
|
|
|
`infrastructure/k3s-manifests/README-DEV-527.md` for the procedure.
|
|
|
|
|
|
|
|
|
|
Quick manual re-apply after a wrangler reconciliation reverted the
|
|
|
|
|
change (rare):
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kubectl -n kube-system patch deployment coredns \
|
|
|
|
|
--type=strategic \
|
|
|
|
|
--patch-file=apps/observability/patches/coredns-dns-config.yaml
|
|
|
|
|
kubectl -n kube-system rollout status deployment coredns
|
|
|
|
|
```
|