stack.basicstack.de/infrastructure/k3s-manifests/README-DEV-527.md

# k3s CoreDNS manifest override (DEV-527)

## Context

The k3s server bundles a stock CoreDNS Deployment and reconciles it
via the k3s addon controller. The manifest source is
`/var/lib/rancher/k3s/server/manifests/coredns.yaml` on each control
plane node. Any `kubectl patch` on the `kube-system/coredns`
Deployment that touches a field the source manifest declares is
reverted by the addon controller.

Hetzner Cloud nodes publish four systemd-resolved DNS upstreams:

```
2a01:4ff:ff00::add:2
2a01:4ff:ff00::add:1
185.12.64.1
185.12.64.2
```

Kubernetes limits pod `resolv.conf` to three nameservers, so kubelet
drops one entry and fires a `DNSConfigForming` Warning event every
time a pod using `dnsPolicy: Default` (or a `hostNetwork` pod that
coerces `ClusterFirst` to `Default`) is created. The stock CoreDNS
Deployment uses `dnsPolicy: Default`, so the warning fires on every
CoreDNS pod restart.

## Fix

The authoritative fix is to modify the source manifest so
CoreDNS runs with an explicit three-nameserver `dnsConfig` and
`dnsPolicy: None`. The three servers match what kubelet was already
picking (both Hetzner IPv6 anycast entries plus the first IPv4
entry), so CoreDNS's `forward . /etc/resolv.conf` upstreams are
unchanged; only the noisy Warning event is silenced.

The `coredns.yaml` file in this directory is the modified manifest
that must live at `/var/lib/rancher/k3s/server/manifests/coredns.yaml`
on **all three** k3s control plane nodes
(`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`).

## Apply / re-apply procedure

Run from any host with SSH access to the CPs:

```bash
TS="$(date -u +%Y%m%dT%H%M%SZ)"
for IP in 178.105.17.239 188.245.85.199 49.13.92.162; do
  echo "== $IP =="
  scp infrastructure/k3s-manifests/coredns.yaml \
    root@$IP:/tmp/coredns.yaml.new
  ssh root@$IP "\
    cp -a /var/lib/rancher/k3s/server/manifests/coredns.yaml \
      /var/lib/rancher/k3s/server/manifests/coredns.yaml.bak-\$(date -u +%Y%m%dT%H%M%SZ) && \
    install -m 0600 -o root -g root /tmp/coredns.yaml.new \
      /var/lib/rancher/k3s/server/manifests/coredns.yaml && \
    rm /tmp/coredns.yaml.new && \
    sha256sum /var/lib/rancher/k3s/server/manifests/coredns.yaml"
done
kubectl -n kube-system rollout status deployment coredns --timeout=120s
```

The k3s addon controller detects the file's checksum change and
re-applies the Deployment; the rolling update replaces the CoreDNS
pod with the new spec.

## Verification

```bash
kubectl get deploy -n kube-system coredns \
  -o jsonpath='{.spec.template.spec.dnsPolicy}{"\n"}{.spec.template.spec.dnsConfig}{"\n"}'
# Expect: dnsPolicy=None + dnsConfig with 3 nameservers.

# No DNSConfigForming event on the current CoreDNS pod:
POD=$(kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].metadata.name}')
kubectl get events -n kube-system \
  --field-selector involvedObject.name=$POD,reason=DNSConfigForming
# Expect: No resources found.

# Cluster DNS still resolves internal + external:
kubectl run dns-test --restart=Never --image=busybox:1.36 --rm=false --command -- \
  sh -c "nslookup kubernetes.default.svc.cluster.local && nslookup forgejo.basicstack.de && echo OK"
kubectl logs dns-test | tail -20
kubectl delete pod dns-test
```

## Upgrade caveat

When the k3s server binary is upgraded to a version that ships a new
CoreDNS manifest (new CoreDNS image, new resource limits, new probes,
etc.), the upgrade will **not** overwrite this file — k3s only creates
the file if it does not exist. Instead, k3s emits an event/log noting
that the on-disk manifest differs from the bundled default. On such an
upgrade, diff the bundled manifest against this one, fold the upstream
changes in, and re-apply.

## Related

- Companion strategic-merge patch:
  `apps/observability/patches/coredns-dns-config.yaml`
- Companion node-exporter patch:
  `apps/observability/patches/node-exporter-dns-config.yaml`
- Parent issue: `DEV-525` (Kubernetes events)
- Sibling k3s coredns.yaml edit: `DEV-526` (clusterIP change)