From 27d26fb11b7cf060dc627f021ec47e25dcfd9150 Mon Sep 17 00:00:00 2001 From: Paperclip CTO Date: Mon, 6 Jul 2026 18:22:53 +0000 Subject: [PATCH] Document DNS configuration and service CIDR fix - Fixed Service CIDR documentation (10.96.0.0/16, not 10.43.0.0/16) - Added comprehensive DNS configuration guide - Documented kubelet cluster-dns requirements - Added CoreDNS forward configuration details - Documented DNS CIDR mismatch troubleshooting (DEV-223) Co-Authored-By: Paperclip --- infrastructure/K3S_OPERATIONS.md | 10 +- .../networking/DNS_CONFIGURATION.md | 260 ++++++++++++++++++ 2 files changed, 267 insertions(+), 3 deletions(-) create mode 100644 infrastructure/networking/DNS_CONFIGURATION.md diff --git a/infrastructure/K3S_OPERATIONS.md b/infrastructure/K3S_OPERATIONS.md index 45e7886..741a14e 100644 --- a/infrastructure/K3S_OPERATIONS.md +++ b/infrastructure/K3S_OPERATIONS.md @@ -399,11 +399,14 @@ failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16" ### Network Configuration - **Pod CIDR:** 10.244.0.0/16 -- **Service CIDR:** 10.43.0.0/16 +- **Service CIDR:** 10.96.0.0/16 +- **Cluster DNS:** 10.96.0.10 (kube-dns service) - **CNI:** Flannel (VXLAN mode) - **Flannel Backend:** Uses **public IPs** for VXLAN tunnels (not private IPs) -**Important:** The cluster-cidr must be explicitly set in the k3s service configuration to prevent bootstrap/runtime mismatch. See Troubleshooting section for details. +**Important:** +- The cluster-cidr must be explicitly set in the k3s service configuration to prevent bootstrap/runtime mismatch. See Troubleshooting section for details. +- The kubelet cluster-dns must be configured to match the kube-dns service ClusterIP. See `infrastructure/networking/DNS_CONFIGURATION.md` for details. ### Node CIDRs @@ -425,7 +428,8 @@ failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16" | Date | Version | Changes | By | |------|---------|---------|-----| -| 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1, corrected Service CIDR documentation to 10.43.0.0/16 | CTO Agent | +| 2026-07-06 | v1.36.2+k3s1 | Fixed DNS service CIDR mismatch (kubelet cluster-dns configuration), corrected Service CIDR documentation to 10.96.0.0/16, added DNS configuration guide | CTO Agent | +| 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1 | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent | | 2026-06-10 | v1.35.5+k3s1 | Original cluster deployment | - | diff --git a/infrastructure/networking/DNS_CONFIGURATION.md b/infrastructure/networking/DNS_CONFIGURATION.md new file mode 100644 index 0000000..6249155 --- /dev/null +++ b/infrastructure/networking/DNS_CONFIGURATION.md @@ -0,0 +1,260 @@ +# DNS Configuration + +## Overview + +The cluster DNS configuration requires alignment between: +1. The kube-dns service ClusterIP (10.96.0.10) +2. The kubelet cluster-dns setting (must match the service IP) +3. The CoreDNS upstream forwarders + +Misalignment causes cluster-wide DNS failures, manifesting as HTTP 502 errors for ingresses and service-to-service communication failures. + +## Current Configuration + +- **kube-dns Service IP:** 10.96.0.10 +- **Service CIDR:** 10.96.0.0/16 +- **CoreDNS Upstream:** 8.8.8.8, 1.1.1.1 + +## Kubelet DNS Configuration + +All k3s nodes (control plane and workers) must be configured to use the correct DNS service IP. + +### Control Plane Configuration + +File: `/etc/rancher/k3s/config.yaml` + +```yaml +# k3s server configuration +kube-apiserver-arg: + - "encryption-provider-config=/etc/rancher/k3s/encryption-config.yaml" +# DNS configuration +kubelet-arg: + - "cluster-dns=10.96.0.10" +``` + +### Worker Node Configuration + +File: `/etc/rancher/k3s/config.yaml` + +```yaml +kubelet-arg: + - "cluster-dns=10.96.0.10" +``` + +### Applying Configuration Changes + +After updating the configuration file: + +```bash +# Control plane +systemctl restart k3s + +# Worker nodes +systemctl restart k3s-agent +``` + +## CoreDNS Configuration + +CoreDNS is managed by k3s through the auto-deploy manifest system. + +### Forward Configuration + +File: `/var/lib/rancher/k3s/server/manifests/coredns.yaml` + +The Corefile must use explicit upstream DNS servers instead of `/etc/resolv.conf`: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: coredns + namespace: kube-system +data: + Corefile: | + .:53 { + errors + health + ready + kubernetes cluster.local in-addr.arpa ip6.arpa { + pods insecure + fallthrough in-addr.arpa ip6.arpa + } + hosts /etc/coredns/NodeHosts { + ttl 60 + reload 15s + fallthrough + } + prometheus :9153 + cache 30 + loop + reload + loadbalance + import /etc/coredns/custom/*.override + forward . 8.8.8.8 1.1.1.1 + } + import /etc/coredns/custom/*.server +``` + +**Critical:** The `forward . 8.8.8.8 1.1.1.1` line must NOT use `/etc/resolv.conf` because the host's resolv.conf points to `127.0.0.53` (systemd-resolved on localhost), which does not work from inside containers. + +### Applying CoreDNS Changes + +K3s automatically reconciles the coredns ConfigMap from the manifest file. After editing: + +```bash +# CoreDNS will reload automatically (reload plugin) +# Or restart CoreDNS for immediate effect: +kubectl rollout restart deployment -n kube-system coredns +``` + +## Verification + +### Check Pod DNS Configuration + +```bash +kubectl run test-dns --image=busybox:latest --rm -i --restart=Never -- cat /etc/resolv.conf +``` + +Expected output: +``` +nameserver 10.96.0.10 +search default.svc.cluster.local svc.cluster.local cluster.local +options ndots:5 +``` + +### Test DNS Resolution + +```bash +# Test cluster DNS +kubectl run test-dns-resolve --image=busybox:latest --rm -i --restart=Never -- \ + nslookup kubernetes.default.svc.cluster.local + +# Test external DNS +kubectl run test-external-dns --image=busybox:latest --rm -i --restart=Never -- \ + nslookup google.com +``` + +Both should resolve successfully. + +### Check CoreDNS Logs + +```bash +kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 +``` + +Should not show errors like: +- `Failed to watch: apiserver not ready` +- `plugin/kubernetes: Failed to list` + +## Troubleshooting + +### Symptom: Pods Cannot Resolve DNS + +**Check 1:** Verify pod DNS configuration +```bash +kubectl run debug --image=busybox:latest --rm -i --restart=Never -- cat /etc/resolv.conf +``` + +If nameserver is wrong (e.g., 10.43.0.10 instead of 10.96.0.10): +1. Check kubelet configuration on all nodes +2. Restart k3s/k3s-agent services +3. Delete and recreate test pods (existing pods keep old DNS config) + +**Check 2:** Verify kube-dns service exists +```bash +kubectl get svc -n kube-system kube-dns +``` + +Should show ClusterIP 10.96.0.10 + +**Check 3:** Test CoreDNS directly +```bash +kubectl run test-dns-direct --image=busybox:latest --rm -i --restart=Never -- \ + nslookup kubernetes.default.svc.cluster.local 10.96.0.10 +``` + +If this works but normal DNS doesn't, the issue is kubelet configuration. + +### Symptom: CoreDNS Returns NXDOMAIN + +**Check 1:** Verify CoreDNS can reach Kubernetes API +```bash +kubectl logs -n kube-system -l k8s-app=kube-dns | grep -i error +``` + +Look for API connection errors. + +**Check 2:** Verify CoreDNS configuration +```bash +kubectl get configmap -n kube-system coredns -o yaml | grep -A 5 "forward" +``` + +Should show `forward . 8.8.8.8 1.1.1.1`, NOT `/etc/resolv.conf` + +**Check 3:** Restart CoreDNS +```bash +kubectl rollout restart deployment -n kube-system coredns +kubectl wait --for=condition=available deployment/coredns -n kube-system --timeout=60s +``` + +### Symptom: HTTP 502 Bad Gateway for Ingresses + +This is often caused by DNS failures. Traefik cannot resolve backend service names. + +**Fix:** +1. Verify DNS is working (see above checks) +2. Restart Traefik to pick up DNS fixes: + ```bash + kubectl rollout restart deployment -n kube-system traefik + ``` + +## Adding New Nodes + +When provisioning new worker nodes, ensure DNS configuration is included: + +```bash +ssh root@ + +# Create k3s config directory +mkdir -p /etc/rancher/k3s + +# Configure DNS +cat > /etc/rancher/k3s/config.yaml <