# observability patches

Strategic-merge patches applied on top of Helm-managed observability
resources. Each file is idempotent (re-applying is a no-op) and is
reasserted by hand rather than by a controller, so re-run after any
`helm upgrade` of the affected release.

## `node-exporter-textfile-collector.yaml` (DEV-494)

Enables the node-exporter textfile collector on the
`kube-prometheus-stack-prometheus-node-exporter` DaemonSet by:

1. adding `--collector.textfile.directory=/host/textfile_collector`
   to the container args, and
2. mounting the host directory `/var/lib/node_exporter/textfile_collector`
   read-only at `/host/textfile_collector` (`hostPath` type
   `DirectoryOrCreate`, kubelet creates it on nodes where the
   directory does not exist yet).

The four monitoring backup CronJobs in `apps/monitoring/` write their
textfile-collector `.prom` files into that same host directory, so the
metrics surface in Prometheus via node-exporter's normal scrape.

Apply / re-apply:

```bash
kubectl -n observability patch daemonset \
  kube-prometheus-stack-prometheus-node-exporter \
  --type=strategic \
  --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
kubectl -n observability rollout status daemonset \
  kube-prometheus-stack-prometheus-node-exporter
```

The kube-prometheus-stack chart is not currently tracked in ArgoCD;
if it moves under GitOps, fold these values into the chart values as
`prometheus-node-exporter.extraArgs` + `.extraHostVolumeMounts`
instead of maintaining this patch.

## `node-exporter-dns-config.yaml` (DEV-527)

Caps the `hostNetwork` node-exporter pod's DNS at three upstream
servers to silence Kubernetes' `DNSConfigForming` Warning event.
Hetzner's OS publishes four systemd-resolved upstreams
(`2a01:4ff:ff00::add:2`, `2a01:4ff:ff00::add:1`, `185.12.64.1`,
`185.12.64.2`), and kubelet drops the fourth because Kubernetes
pods are limited to three nameservers.

Sets:

1. `dnsPolicy: None` so `dnsConfig` is authoritative (with `Default`
   or the auto-coerced `ClusterFirst→Default` on a hostNetwork pod,
   kubelet still merges the node's resolv.conf on top and the fourth
   nameserver keeps re-triggering the warning), and
2. an explicit `dnsConfig` with the two Hetzner IPv6 anycast entries
   plus the first IPv4 entry, matching the three servers kubelet was
   already picking, plus `edns0`/`trust-ad` resolv.conf options.

Apply / re-apply:

```bash
kubectl -n observability patch daemonset \
  kube-prometheus-stack-prometheus-node-exporter \
  --type=strategic \
  --patch-file=apps/observability/patches/node-exporter-dns-config.yaml
kubectl -n observability rollout status daemonset \
  kube-prometheus-stack-prometheus-node-exporter
```

If the chart moves under GitOps, fold these values into the chart
values as `prometheus-node-exporter.dnsPolicy` +
`.dnsConfig` instead of maintaining this patch.

## `coredns-dns-config.yaml` (DEV-527)

Companion to the node-exporter patch for the k3s built-in CoreDNS
Deployment (`kube-system/coredns`, `dnsPolicy: Default`), which also
triggers the same `DNSConfigForming` warning on every pod restart.

Because CoreDNS is a **k3s Addon** whose source manifest lives at
`/var/lib/rancher/k3s/server/manifests/coredns.yaml` on each control
plane node, `kubectl patch` alone is NOT durable — the k3s addon
controller re-applies the source manifest and reverts `dnsPolicy`
back to `Default`. This patch file is kept in the repo as the
canonical description of the fix and can be used for a quick manual
re-apply (until the addon controller next reconciles), but the
authoritative fix is applied by editing the same `dnsPolicy` /
`dnsConfig` block into the CoreDNS Deployment stanza of the k3s
`coredns.yaml` on each CP node. See
`infrastructure/k3s-manifests/README-DEV-527.md` for the procedure.

Quick manual re-apply after a wrangler reconciliation reverted the
change (rare):

```bash
kubectl -n kube-system patch deployment coredns \
  --type=strategic \
  --patch-file=apps/observability/patches/coredns-dns-config.yaml
kubectl -n kube-system rollout status deployment coredns
```