1c3cd31381
Hetzner OS publishes 4 systemd-resolved upstreams and Kubernetes limits pod resolv.conf to 3 nameservers, so kubelet drops the 4th and fires a DNSConfigForming Warning event on every hostNetwork or dnsPolicy=Default pod restart. Silence the noise by pinning the pods to 3 explicit servers (same 3 kubelet was already picking). - apps/observability/patches/node-exporter-dns-config.yaml — strategic- merge patch adding dnsPolicy=None + dnsConfig to the kube-prometheus-stack node-exporter DaemonSet (Helm-managed, applied by hand) - apps/observability/patches/coredns-dns-config.yaml — companion patch for the k3s built-in CoreDNS Deployment. kubectl patch alone is not durable because the k3s addon controller reverts dnsPolicy; kept as a quick manual re-apply hook - infrastructure/k3s-manifests/coredns.yaml — the authoritative modified k3s addon manifest that must live at /var/lib/rancher/k3s/server/manifests/coredns.yaml on all 3 CP nodes - infrastructure/k3s-manifests/README-DEV-527.md — apply procedure, verification steps, and upgrade caveat Applied and verified on the live cluster: - node-exporter DaemonSet rolled with dnsPolicy=None; no DNSConfigForming events on current pods - coredns Deployment reconciled after pushing the modified manifest to all 3 CPs; new pod runs with dnsPolicy=None and 3-nameserver dnsConfig - internal + external DNS resolution still works Co-Authored-By: Paperclip <noreply@paperclip.ing>
100 lines
4 KiB
Markdown
100 lines
4 KiB
Markdown
# observability patches
|
|
|
|
Strategic-merge patches applied on top of Helm-managed observability
|
|
resources. Each file is idempotent (re-applying is a no-op) and is
|
|
reasserted by hand rather than by a controller, so re-run after any
|
|
`helm upgrade` of the affected release.
|
|
|
|
## `node-exporter-textfile-collector.yaml` (DEV-494)
|
|
|
|
Enables the node-exporter textfile collector on the
|
|
`kube-prometheus-stack-prometheus-node-exporter` DaemonSet by:
|
|
|
|
1. adding `--collector.textfile.directory=/host/textfile_collector`
|
|
to the container args, and
|
|
2. mounting the host directory `/var/lib/node_exporter/textfile_collector`
|
|
read-only at `/host/textfile_collector` (`hostPath` type
|
|
`DirectoryOrCreate`, kubelet creates it on nodes where the
|
|
directory does not exist yet).
|
|
|
|
The four monitoring backup CronJobs in `apps/monitoring/` write their
|
|
textfile-collector `.prom` files into that same host directory, so the
|
|
metrics surface in Prometheus via node-exporter's normal scrape.
|
|
|
|
Apply / re-apply:
|
|
|
|
```bash
|
|
kubectl -n observability patch daemonset \
|
|
kube-prometheus-stack-prometheus-node-exporter \
|
|
--type=strategic \
|
|
--patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
|
|
kubectl -n observability rollout status daemonset \
|
|
kube-prometheus-stack-prometheus-node-exporter
|
|
```
|
|
|
|
The kube-prometheus-stack chart is not currently tracked in ArgoCD;
|
|
if it moves under GitOps, fold these values into the chart values as
|
|
`prometheus-node-exporter.extraArgs` + `.extraHostVolumeMounts`
|
|
instead of maintaining this patch.
|
|
|
|
## `node-exporter-dns-config.yaml` (DEV-527)
|
|
|
|
Caps the `hostNetwork` node-exporter pod's DNS at three upstream
|
|
servers to silence Kubernetes' `DNSConfigForming` Warning event.
|
|
Hetzner's OS publishes four systemd-resolved upstreams
|
|
(`2a01:4ff:ff00::add:2`, `2a01:4ff:ff00::add:1`, `185.12.64.1`,
|
|
`185.12.64.2`), and kubelet drops the fourth because Kubernetes
|
|
pods are limited to three nameservers.
|
|
|
|
Sets:
|
|
|
|
1. `dnsPolicy: None` so `dnsConfig` is authoritative (with `Default`
|
|
or the auto-coerced `ClusterFirst→Default` on a hostNetwork pod,
|
|
kubelet still merges the node's resolv.conf on top and the fourth
|
|
nameserver keeps re-triggering the warning), and
|
|
2. an explicit `dnsConfig` with the two Hetzner IPv6 anycast entries
|
|
plus the first IPv4 entry, matching the three servers kubelet was
|
|
already picking, plus `edns0`/`trust-ad` resolv.conf options.
|
|
|
|
Apply / re-apply:
|
|
|
|
```bash
|
|
kubectl -n observability patch daemonset \
|
|
kube-prometheus-stack-prometheus-node-exporter \
|
|
--type=strategic \
|
|
--patch-file=apps/observability/patches/node-exporter-dns-config.yaml
|
|
kubectl -n observability rollout status daemonset \
|
|
kube-prometheus-stack-prometheus-node-exporter
|
|
```
|
|
|
|
If the chart moves under GitOps, fold these values into the chart
|
|
values as `prometheus-node-exporter.dnsPolicy` +
|
|
`.dnsConfig` instead of maintaining this patch.
|
|
|
|
## `coredns-dns-config.yaml` (DEV-527)
|
|
|
|
Companion to the node-exporter patch for the k3s built-in CoreDNS
|
|
Deployment (`kube-system/coredns`, `dnsPolicy: Default`), which also
|
|
triggers the same `DNSConfigForming` warning on every pod restart.
|
|
|
|
Because CoreDNS is a **k3s Addon** whose source manifest lives at
|
|
`/var/lib/rancher/k3s/server/manifests/coredns.yaml` on each control
|
|
plane node, `kubectl patch` alone is NOT durable — the k3s addon
|
|
controller re-applies the source manifest and reverts `dnsPolicy`
|
|
back to `Default`. This patch file is kept in the repo as the
|
|
canonical description of the fix and can be used for a quick manual
|
|
re-apply (until the addon controller next reconciles), but the
|
|
authoritative fix is applied by editing the same `dnsPolicy` /
|
|
`dnsConfig` block into the CoreDNS Deployment stanza of the k3s
|
|
`coredns.yaml` on each CP node. See
|
|
`infrastructure/k3s-manifests/README-DEV-527.md` for the procedure.
|
|
|
|
Quick manual re-apply after a wrangler reconciliation reverted the
|
|
change (rare):
|
|
|
|
```bash
|
|
kubectl -n kube-system patch deployment coredns \
|
|
--type=strategic \
|
|
--patch-file=apps/observability/patches/coredns-dns-config.yaml
|
|
kubectl -n kube-system rollout status deployment coredns
|
|
```
|