# DNS Configuration ## Overview The cluster DNS configuration requires alignment between: 1. The kube-dns service ClusterIP (10.96.0.10) 2. The kubelet cluster-dns setting (must match the service IP) 3. The CoreDNS upstream forwarders Misalignment causes cluster-wide DNS failures, manifesting as HTTP 502 errors for ingresses and service-to-service communication failures. ## Current Configuration - **kube-dns Service IP:** 10.96.0.10 - **Service CIDR:** 10.96.0.0/16 - **CoreDNS Upstream:** 8.8.8.8, 1.1.1.1 ## Kubelet DNS Configuration All k3s nodes (control plane and workers) must be configured to use the correct DNS service IP. ### Control Plane Configuration File: `/etc/rancher/k3s/config.yaml` ```yaml # k3s server configuration kube-apiserver-arg: - "encryption-provider-config=/etc/rancher/k3s/encryption-config.yaml" # DNS configuration kubelet-arg: - "cluster-dns=10.96.0.10" ``` ### Worker Node Configuration File: `/etc/rancher/k3s/config.yaml` ```yaml kubelet-arg: - "cluster-dns=10.96.0.10" ``` ### Applying Configuration Changes After updating the configuration file: ```bash # Control plane systemctl restart k3s # Worker nodes systemctl restart k3s-agent ``` ## CoreDNS Configuration CoreDNS is managed by k3s through the auto-deploy manifest system. ### Forward Configuration File: `/var/lib/rancher/k3s/server/manifests/coredns.yaml` The Corefile must use explicit upstream DNS servers instead of `/etc/resolv.conf`: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } hosts /etc/coredns/NodeHosts { ttl 60 reload 15s fallthrough } prometheus :9153 cache 30 loop reload loadbalance import /etc/coredns/custom/*.override forward . 8.8.8.8 1.1.1.1 } import /etc/coredns/custom/*.server ``` **Critical:** The `forward . 8.8.8.8 1.1.1.1` line must NOT use `/etc/resolv.conf` because the host's resolv.conf points to `127.0.0.53` (systemd-resolved on localhost), which does not work from inside containers. ### Applying CoreDNS Changes K3s automatically reconciles the coredns ConfigMap from the manifest file. After editing: ```bash # CoreDNS will reload automatically (reload plugin) # Or restart CoreDNS for immediate effect: kubectl rollout restart deployment -n kube-system coredns ``` ## Verification ### Check Pod DNS Configuration ```bash kubectl run test-dns --image=busybox:latest --rm -i --restart=Never -- cat /etc/resolv.conf ``` Expected output: ``` nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5 ``` ### Test DNS Resolution ```bash # Test cluster DNS kubectl run test-dns-resolve --image=busybox:latest --rm -i --restart=Never -- \ nslookup kubernetes.default.svc.cluster.local # Test external DNS kubectl run test-external-dns --image=busybox:latest --rm -i --restart=Never -- \ nslookup google.com ``` Both should resolve successfully. ### Check CoreDNS Logs ```bash kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 ``` Should not show errors like: - `Failed to watch: apiserver not ready` - `plugin/kubernetes: Failed to list` ## Troubleshooting ### Symptom: Pods Cannot Resolve DNS **Check 1:** Verify pod DNS configuration ```bash kubectl run debug --image=busybox:latest --rm -i --restart=Never -- cat /etc/resolv.conf ``` If nameserver is wrong (e.g., 10.43.0.10 instead of 10.96.0.10): 1. Check kubelet configuration on all nodes 2. Restart k3s/k3s-agent services 3. Delete and recreate test pods (existing pods keep old DNS config) **Check 2:** Verify kube-dns service exists ```bash kubectl get svc -n kube-system kube-dns ``` Should show ClusterIP 10.96.0.10 **Check 3:** Test CoreDNS directly ```bash kubectl run test-dns-direct --image=busybox:latest --rm -i --restart=Never -- \ nslookup kubernetes.default.svc.cluster.local 10.96.0.10 ``` If this works but normal DNS doesn't, the issue is kubelet configuration. ### Symptom: CoreDNS Returns NXDOMAIN **Check 1:** Verify CoreDNS can reach Kubernetes API ```bash kubectl logs -n kube-system -l k8s-app=kube-dns | grep -i error ``` Look for API connection errors. **Check 2:** Verify CoreDNS configuration ```bash kubectl get configmap -n kube-system coredns -o yaml | grep -A 5 "forward" ``` Should show `forward . 8.8.8.8 1.1.1.1`, NOT `/etc/resolv.conf` **Check 3:** Restart CoreDNS ```bash kubectl rollout restart deployment -n kube-system coredns kubectl wait --for=condition=available deployment/coredns -n kube-system --timeout=60s ``` ### Symptom: HTTP 502 Bad Gateway for Ingresses This is often caused by DNS failures. Traefik cannot resolve backend service names. **Fix:** 1. Verify DNS is working (see above checks) 2. Restart Traefik to pick up DNS fixes: ```bash kubectl rollout restart deployment -n kube-system traefik ``` ## Adding New Nodes When provisioning new worker nodes, ensure DNS configuration is included: ```bash ssh root@ # Create k3s config directory mkdir -p /etc/rancher/k3s # Configure DNS cat > /etc/rancher/k3s/config.yaml <