1c3cd31381
Hetzner OS publishes 4 systemd-resolved upstreams and Kubernetes limits pod resolv.conf to 3 nameservers, so kubelet drops the 4th and fires a DNSConfigForming Warning event on every hostNetwork or dnsPolicy=Default pod restart. Silence the noise by pinning the pods to 3 explicit servers (same 3 kubelet was already picking). - apps/observability/patches/node-exporter-dns-config.yaml — strategic- merge patch adding dnsPolicy=None + dnsConfig to the kube-prometheus-stack node-exporter DaemonSet (Helm-managed, applied by hand) - apps/observability/patches/coredns-dns-config.yaml — companion patch for the k3s built-in CoreDNS Deployment. kubectl patch alone is not durable because the k3s addon controller reverts dnsPolicy; kept as a quick manual re-apply hook - infrastructure/k3s-manifests/coredns.yaml — the authoritative modified k3s addon manifest that must live at /var/lib/rancher/k3s/server/manifests/coredns.yaml on all 3 CP nodes - infrastructure/k3s-manifests/README-DEV-527.md — apply procedure, verification steps, and upgrade caveat Applied and verified on the live cluster: - node-exporter DaemonSet rolled with dnsPolicy=None; no DNSConfigForming events on current pods - coredns Deployment reconciled after pushing the modified manifest to all 3 CPs; new pod runs with dnsPolicy=None and 3-nameserver dnsConfig - internal + external DNS resolution still works Co-Authored-By: Paperclip <noreply@paperclip.ing>
104 lines
3.9 KiB
Markdown
104 lines
3.9 KiB
Markdown
# k3s CoreDNS manifest override (DEV-527)
|
|
|
|
## Context
|
|
|
|
The k3s server bundles a stock CoreDNS Deployment and reconciles it
|
|
via the k3s addon controller. The manifest source is
|
|
`/var/lib/rancher/k3s/server/manifests/coredns.yaml` on each control
|
|
plane node. Any `kubectl patch` on the `kube-system/coredns`
|
|
Deployment that touches a field the source manifest declares is
|
|
reverted by the addon controller.
|
|
|
|
Hetzner Cloud nodes publish four systemd-resolved DNS upstreams:
|
|
|
|
```
|
|
2a01:4ff:ff00::add:2
|
|
2a01:4ff:ff00::add:1
|
|
185.12.64.1
|
|
185.12.64.2
|
|
```
|
|
|
|
Kubernetes limits pod `resolv.conf` to three nameservers, so kubelet
|
|
drops one entry and fires a `DNSConfigForming` Warning event every
|
|
time a pod using `dnsPolicy: Default` (or a `hostNetwork` pod that
|
|
coerces `ClusterFirst` to `Default`) is created. The stock CoreDNS
|
|
Deployment uses `dnsPolicy: Default`, so the warning fires on every
|
|
CoreDNS pod restart.
|
|
|
|
## Fix
|
|
|
|
The authoritative fix is to modify the source manifest so
|
|
CoreDNS runs with an explicit three-nameserver `dnsConfig` and
|
|
`dnsPolicy: None`. The three servers match what kubelet was already
|
|
picking (both Hetzner IPv6 anycast entries plus the first IPv4
|
|
entry), so CoreDNS's `forward . /etc/resolv.conf` upstreams are
|
|
unchanged; only the noisy Warning event is silenced.
|
|
|
|
The `coredns.yaml` file in this directory is the modified manifest
|
|
that must live at `/var/lib/rancher/k3s/server/manifests/coredns.yaml`
|
|
on **all three** k3s control plane nodes
|
|
(`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`).
|
|
|
|
## Apply / re-apply procedure
|
|
|
|
Run from any host with SSH access to the CPs:
|
|
|
|
```bash
|
|
TS="$(date -u +%Y%m%dT%H%M%SZ)"
|
|
for IP in 178.105.17.239 188.245.85.199 49.13.92.162; do
|
|
echo "== $IP =="
|
|
scp infrastructure/k3s-manifests/coredns.yaml \
|
|
root@$IP:/tmp/coredns.yaml.new
|
|
ssh root@$IP "\
|
|
cp -a /var/lib/rancher/k3s/server/manifests/coredns.yaml \
|
|
/var/lib/rancher/k3s/server/manifests/coredns.yaml.bak-\$(date -u +%Y%m%dT%H%M%SZ) && \
|
|
install -m 0600 -o root -g root /tmp/coredns.yaml.new \
|
|
/var/lib/rancher/k3s/server/manifests/coredns.yaml && \
|
|
rm /tmp/coredns.yaml.new && \
|
|
sha256sum /var/lib/rancher/k3s/server/manifests/coredns.yaml"
|
|
done
|
|
kubectl -n kube-system rollout status deployment coredns --timeout=120s
|
|
```
|
|
|
|
The k3s addon controller detects the file's checksum change and
|
|
re-applies the Deployment; the rolling update replaces the CoreDNS
|
|
pod with the new spec.
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
kubectl get deploy -n kube-system coredns \
|
|
-o jsonpath='{.spec.template.spec.dnsPolicy}{"\n"}{.spec.template.spec.dnsConfig}{"\n"}'
|
|
# Expect: dnsPolicy=None + dnsConfig with 3 nameservers.
|
|
|
|
# No DNSConfigForming event on the current CoreDNS pod:
|
|
POD=$(kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].metadata.name}')
|
|
kubectl get events -n kube-system \
|
|
--field-selector involvedObject.name=$POD,reason=DNSConfigForming
|
|
# Expect: No resources found.
|
|
|
|
# Cluster DNS still resolves internal + external:
|
|
kubectl run dns-test --restart=Never --image=busybox:1.36 --rm=false --command -- \
|
|
sh -c "nslookup kubernetes.default.svc.cluster.local && nslookup forgejo.basicstack.de && echo OK"
|
|
kubectl logs dns-test | tail -20
|
|
kubectl delete pod dns-test
|
|
```
|
|
|
|
## Upgrade caveat
|
|
|
|
When the k3s server binary is upgraded to a version that ships a new
|
|
CoreDNS manifest (new CoreDNS image, new resource limits, new probes,
|
|
etc.), the upgrade will **not** overwrite this file — k3s only creates
|
|
the file if it does not exist. Instead, k3s emits an event/log noting
|
|
that the on-disk manifest differs from the bundled default. On such an
|
|
upgrade, diff the bundled manifest against this one, fold the upstream
|
|
changes in, and re-apply.
|
|
|
|
## Related
|
|
|
|
- Companion strategic-merge patch:
|
|
`apps/observability/patches/coredns-dns-config.yaml`
|
|
- Companion node-exporter patch:
|
|
`apps/observability/patches/node-exporter-dns-config.yaml`
|
|
- Parent issue: `DEV-525` (Kubernetes events)
|
|
- Sibling k3s coredns.yaml edit: `DEV-526` (clusterIP change)
|