stack.basicstack.de/infrastructure/k3s-manifests/README-DEV-527.md

k3s CoreDNS manifest override (DEV-527)

Context

The k3s server bundles a stock CoreDNS Deployment and reconciles it via the k3s addon controller. The manifest source is /var/lib/rancher/k3s/server/manifests/coredns.yaml on each control plane node. Any kubectl patch on the kube-system/coredns Deployment that touches a field the source manifest declares is reverted by the addon controller.

Hetzner Cloud nodes publish four systemd-resolved DNS upstreams:

2a01:4ff:ff00::add:2
2a01:4ff:ff00::add:1
185.12.64.1
185.12.64.2

Kubernetes limits pod resolv.conf to three nameservers, so kubelet drops one entry and fires a DNSConfigForming Warning event every time a pod using dnsPolicy: Default (or a hostNetwork pod that coerces ClusterFirst to Default) is created. The stock CoreDNS Deployment uses dnsPolicy: Default, so the warning fires on every CoreDNS pod restart.

Fix

The authoritative fix is to modify the source manifest so CoreDNS runs with an explicit three-nameserver dnsConfig and dnsPolicy: None. The three servers match what kubelet was already picking (both Hetzner IPv6 anycast entries plus the first IPv4 entry), so CoreDNS's forward . /etc/resolv.conf upstreams are unchanged; only the noisy Warning event is silenced.

The coredns.yaml file in this directory is the modified manifest that must live at /var/lib/rancher/k3s/server/manifests/coredns.yaml on all three k3s control plane nodes (k3s-cp-1, k3s-cp-2, k3s-cp-3).

Apply / re-apply procedure

Run from any host with SSH access to the CPs:

TS="$(date -u +%Y%m%dT%H%M%SZ)"
for IP in 178.105.17.239 188.245.85.199 49.13.92.162; do
  echo "== $IP =="
  scp infrastructure/k3s-manifests/coredns.yaml \
    root@$IP:/tmp/coredns.yaml.new
  ssh root@$IP "\
    cp -a /var/lib/rancher/k3s/server/manifests/coredns.yaml \
      /var/lib/rancher/k3s/server/manifests/coredns.yaml.bak-\$(date -u +%Y%m%dT%H%M%SZ) && \
    install -m 0600 -o root -g root /tmp/coredns.yaml.new \
      /var/lib/rancher/k3s/server/manifests/coredns.yaml && \
    rm /tmp/coredns.yaml.new && \
    sha256sum /var/lib/rancher/k3s/server/manifests/coredns.yaml"
done
kubectl -n kube-system rollout status deployment coredns --timeout=120s

The k3s addon controller detects the file's checksum change and re-applies the Deployment; the rolling update replaces the CoreDNS pod with the new spec.

Verification

kubectl get deploy -n kube-system coredns \
  -o jsonpath='{.spec.template.spec.dnsPolicy}{"\n"}{.spec.template.spec.dnsConfig}{"\n"}'
# Expect: dnsPolicy=None + dnsConfig with 3 nameservers.

# No DNSConfigForming event on the current CoreDNS pod:
POD=$(kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].metadata.name}')
kubectl get events -n kube-system \
  --field-selector involvedObject.name=$POD,reason=DNSConfigForming
# Expect: No resources found.

# Cluster DNS still resolves internal + external:
kubectl run dns-test --restart=Never --image=busybox:1.36 --rm=false --command -- \
  sh -c "nslookup kubernetes.default.svc.cluster.local && nslookup forgejo.basicstack.de && echo OK"
kubectl logs dns-test | tail -20
kubectl delete pod dns-test

Upgrade caveat

When the k3s server binary is upgraded to a version that ships a new CoreDNS manifest (new CoreDNS image, new resource limits, new probes, etc.), the upgrade will not overwrite this file — k3s only creates the file if it does not exist. Instead, k3s emits an event/log noting that the on-disk manifest differs from the bundled default. On such an upgrade, diff the bundled manifest against this one, fold the upstream changes in, and re-apply.

Related

• Companion strategic-merge patch: apps/observability/patches/coredns-dns-config.yaml
• Companion node-exporter patch: apps/observability/patches/node-exporter-dns-config.yaml
• Parent issue: DEV-525 (Kubernetes events)
• Sibling k3s coredns.yaml edit: DEV-526 (clusterIP change)