stack.basicstack.de/apps/observability/patches/README.md

observability patches

Strategic-merge patches applied on top of Helm-managed observability resources. Each file is idempotent (re-applying is a no-op) and is reasserted by hand rather than by a controller, so re-run after any helm upgrade of the affected release.

node-exporter-textfile-collector.yaml (DEV-494)

Enables the node-exporter textfile collector on the kube-prometheus-stack-prometheus-node-exporter DaemonSet by:

1. adding --collector.textfile.directory=/host/textfile_collector to the container args, and
2. mounting the host directory /var/lib/node_exporter/textfile_collector read-only at /host/textfile_collector (hostPath type DirectoryOrCreate, kubelet creates it on nodes where the directory does not exist yet).

The four monitoring backup CronJobs in apps/monitoring/ write their textfile-collector .prom files into that same host directory, so the metrics surface in Prometheus via node-exporter's normal scrape.

Apply / re-apply:

kubectl -n observability patch daemonset \
  kube-prometheus-stack-prometheus-node-exporter \
  --type=strategic \
  --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
kubectl -n observability rollout status daemonset \
  kube-prometheus-stack-prometheus-node-exporter

The kube-prometheus-stack chart is not currently tracked in ArgoCD; if it moves under GitOps, fold these values into the chart values as prometheus-node-exporter.extraArgs + .extraHostVolumeMounts instead of maintaining this patch.

node-exporter-dns-config.yaml (DEV-527)

Caps the hostNetwork node-exporter pod's DNS at three upstream servers to silence Kubernetes' DNSConfigForming Warning event. Hetzner's OS publishes four systemd-resolved upstreams (2a01:4ff:ff00::add:2, 2a01:4ff:ff00::add:1, 185.12.64.1, 185.12.64.2), and kubelet drops the fourth because Kubernetes pods are limited to three nameservers.

Sets:

1. dnsPolicy: None so dnsConfig is authoritative (with Default or the auto-coerced ClusterFirst→Default on a hostNetwork pod, kubelet still merges the node's resolv.conf on top and the fourth nameserver keeps re-triggering the warning), and
2. an explicit dnsConfig with the two Hetzner IPv6 anycast entries plus the first IPv4 entry, matching the three servers kubelet was already picking, plus edns0/trust-ad resolv.conf options.

Apply / re-apply:

kubectl -n observability patch daemonset \
  kube-prometheus-stack-prometheus-node-exporter \
  --type=strategic \
  --patch-file=apps/observability/patches/node-exporter-dns-config.yaml
kubectl -n observability rollout status daemonset \
  kube-prometheus-stack-prometheus-node-exporter

If the chart moves under GitOps, fold these values into the chart values as prometheus-node-exporter.dnsPolicy + .dnsConfig instead of maintaining this patch.

coredns-dns-config.yaml (DEV-527)

Companion to the node-exporter patch for the k3s built-in CoreDNS Deployment (kube-system/coredns, dnsPolicy: Default), which also triggers the same DNSConfigForming warning on every pod restart.

Because CoreDNS is a k3s Addon whose source manifest lives at /var/lib/rancher/k3s/server/manifests/coredns.yaml on each control plane node, kubectl patch alone is NOT durable — the k3s addon controller re-applies the source manifest and reverts dnsPolicy back to Default. This patch file is kept in the repo as the canonical description of the fix and can be used for a quick manual re-apply (until the addon controller next reconciles), but the authoritative fix is applied by editing the same dnsPolicy / dnsConfig block into the CoreDNS Deployment stanza of the k3s coredns.yaml on each CP node. See infrastructure/k3s-manifests/README-DEV-527.md for the procedure.

Quick manual re-apply after a wrangler reconciliation reverted the change (rare):

kubectl -n kube-system patch deployment coredns \
  --type=strategic \
  --patch-file=apps/observability/patches/coredns-dns-config.yaml
kubectl -n kube-system rollout status deployment coredns