basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Document DNS configuration and service CIDR fix

Browse source 
- Fixed Service CIDR documentation (10.96.0.0/16, not 10.43.0.0/16)
- Added comprehensive DNS configuration guide
- Documented kubelet cluster-dns requirements
- Added CoreDNS forward configuration details
- Documented DNS CIDR mismatch troubleshooting (DEV-223)

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Paperclip CTO 2026-07-06 18:22:53 +00:00
parent f3034af04a
commit 27d26fb11b
2 changed files with 267 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
infrastructure/K3S_OPERATIONS.md
View file

@ -399,11 +399,14 @@ failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16"
### Network Configuration ### Network Configuration
- **Pod CIDR:** 10.244.0.0/16 - **Pod CIDR:** 10.244.0.0/16
- **Service CIDR:** 10.43.0.0/16 - **Service CIDR:** 10.96.0.0/16
- **Cluster DNS:** 10.96.0.10 (kube-dns service)
- **CNI:** Flannel (VXLAN mode) - **CNI:** Flannel (VXLAN mode)
- **Flannel Backend:** Uses **public IPs** for VXLAN tunnels (not private IPs) - **Flannel Backend:** Uses **public IPs** for VXLAN tunnels (not private IPs)
**Important:** The cluster-cidr must be explicitly set in the k3s service configuration to prevent bootstrap/runtime mismatch. See Troubleshooting section for details. **Important:**
- The cluster-cidr must be explicitly set in the k3s service configuration to prevent bootstrap/runtime mismatch. See Troubleshooting section for details.
- The kubelet cluster-dns must be configured to match the kube-dns service ClusterIP. See `infrastructure/networking/DNS_CONFIGURATION.md` for details.
### Node CIDRs ### Node CIDRs
@ -425,7 +428,8 @@ failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16"
| Date | Version | Changes | By | | Date | Version | Changes | By |
|------|---------|---------|-----| |------|---------|---------|-----|
| 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1, corrected Service CIDR documentation to 10.43.0.0/16 | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Fixed DNS service CIDR mismatch (kubelet cluster-dns configuration), corrected Service CIDR documentation to 10.96.0.0/16, added DNS configuration guide | CTO Agent |
| 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1 | CTO Agent |
| 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent |
| 2026-06-10 | v1.35.5+k3s1 | Original cluster deployment | - | | 2026-06-10 | v1.35.5+k3s1 | Original cluster deployment | - |

260
infrastructure/networking/DNS_CONFIGURATION.md Normal file
View file

@ -0,0 +1,260 @@
# DNS Configuration
## Overview
The cluster DNS configuration requires alignment between:
1. The kube-dns service ClusterIP (10.96.0.10)
2. The kubelet cluster-dns setting (must match the service IP)
3. The CoreDNS upstream forwarders
Misalignment causes cluster-wide DNS failures, manifesting as HTTP 502 errors for ingresses and service-to-service communication failures.
## Current Configuration
- **kube-dns Service IP:** 10.96.0.10
- **Service CIDR:** 10.96.0.0/16
- **CoreDNS Upstream:** 8.8.8.8, 1.1.1.1
## Kubelet DNS Configuration
All k3s nodes (control plane and workers) must be configured to use the correct DNS service IP.
### Control Plane Configuration
File: `/etc/rancher/k3s/config.yaml`
```yaml
# k3s server configuration
kube-apiserver-arg:
- "encryption-provider-config=/etc/rancher/k3s/encryption-config.yaml"
# DNS configuration
kubelet-arg:
- "cluster-dns=10.96.0.10"
```
### Worker Node Configuration
File: `/etc/rancher/k3s/config.yaml`
```yaml
kubelet-arg:
- "cluster-dns=10.96.0.10"
```
### Applying Configuration Changes
After updating the configuration file:
```bash
# Control plane
systemctl restart k3s
# Worker nodes
systemctl restart k3s-agent
```
## CoreDNS Configuration
CoreDNS is managed by k3s through the auto-deploy manifest system.
### Forward Configuration
File: `/var/lib/rancher/k3s/server/manifests/coredns.yaml`
The Corefile must use explicit upstream DNS servers instead of `/etc/resolv.conf`:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
hosts /etc/coredns/NodeHosts {
ttl 60
reload 15s
fallthrough
}
prometheus :9153
cache 30
loop
reload
loadbalance
import /etc/coredns/custom/*.override
forward . 8.8.8.8 1.1.1.1
}
import /etc/coredns/custom/*.server
```
**Critical:** The `forward . 8.8.8.8 1.1.1.1` line must NOT use `/etc/resolv.conf` because the host's resolv.conf points to `127.0.0.53` (systemd-resolved on localhost), which does not work from inside containers.
### Applying CoreDNS Changes
K3s automatically reconciles the coredns ConfigMap from the manifest file. After editing:
```bash
# CoreDNS will reload automatically (reload plugin)
# Or restart CoreDNS for immediate effect:
kubectl rollout restart deployment -n kube-system coredns
```
## Verification
### Check Pod DNS Configuration
```bash
kubectl run test-dns --image=busybox:latest --rm -i --restart=Never -- cat /etc/resolv.conf
```
Expected output:
```
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
```
### Test DNS Resolution
```bash
# Test cluster DNS
kubectl run test-dns-resolve --image=busybox:latest --rm -i --restart=Never -- \
nslookup kubernetes.default.svc.cluster.local
# Test external DNS
kubectl run test-external-dns --image=busybox:latest --rm -i --restart=Never -- \
nslookup google.com
```
Both should resolve successfully.
### Check CoreDNS Logs
```bash
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
```
Should not show errors like:
- `Failed to watch: apiserver not ready`
- `plugin/kubernetes: Failed to list`
## Troubleshooting
### Symptom: Pods Cannot Resolve DNS
**Check 1:** Verify pod DNS configuration
```bash
kubectl run debug --image=busybox:latest --rm -i --restart=Never -- cat /etc/resolv.conf
```
If nameserver is wrong (e.g., 10.43.0.10 instead of 10.96.0.10):
1. Check kubelet configuration on all nodes
2. Restart k3s/k3s-agent services
3. Delete and recreate test pods (existing pods keep old DNS config)
**Check 2:** Verify kube-dns service exists
```bash
kubectl get svc -n kube-system kube-dns
```
Should show ClusterIP 10.96.0.10
**Check 3:** Test CoreDNS directly
```bash
kubectl run test-dns-direct --image=busybox:latest --rm -i --restart=Never -- \
nslookup kubernetes.default.svc.cluster.local 10.96.0.10
```
If this works but normal DNS doesn't, the issue is kubelet configuration.
### Symptom: CoreDNS Returns NXDOMAIN
**Check 1:** Verify CoreDNS can reach Kubernetes API
```bash
kubectl logs -n kube-system -l k8s-app=kube-dns | grep -i error
```
Look for API connection errors.
**Check 2:** Verify CoreDNS configuration
```bash
kubectl get configmap -n kube-system coredns -o yaml | grep -A 5 "forward"
```
Should show `forward . 8.8.8.8 1.1.1.1`, NOT `/etc/resolv.conf`
**Check 3:** Restart CoreDNS
```bash
kubectl rollout restart deployment -n kube-system coredns
kubectl wait --for=condition=available deployment/coredns -n kube-system --timeout=60s
```
### Symptom: HTTP 502 Bad Gateway for Ingresses
This is often caused by DNS failures. Traefik cannot resolve backend service names.
**Fix:**
1. Verify DNS is working (see above checks)
2. Restart Traefik to pick up DNS fixes:
```bash
kubectl rollout restart deployment -n kube-system traefik
```
## Adding New Nodes
When provisioning new worker nodes, ensure DNS configuration is included:
```bash
ssh root@<new-node-ip>
# Create k3s config directory
mkdir -p /etc/rancher/k3s
# Configure DNS
cat > /etc/rancher/k3s/config.yaml <<EOF
kubelet-arg:
- "cluster-dns=10.96.0.10"
EOF
# Then proceed with k3s agent installation
```
## Root Cause History
### DEV-223: DNS Service CIDR Mismatch (2026-07-06)
**Problem:** Cluster-wide DNS failure causing HTTP 502 errors for all ingresses and service communication failures.
**Root Cause:**
- The cluster was configured with service CIDR 10.96.0.0/16 (standard Kubernetes range)
- k3s defaults to 10.43.0.0/16, and kubelet was never reconfigured
- Pods received DNS server 10.43.0.10, but kube-dns service was at 10.96.0.10
- All DNS queries timed out
**Fix:**
1. Added `kubelet-arg: ["cluster-dns=10.96.0.10"]` to `/etc/rancher/k3s/config.yaml` on control plane and all workers
2. Updated CoreDNS forward configuration to use `8.8.8.8 1.1.1.1` instead of `/etc/resolv.conf`
3. Restarted k3s services cluster-wide
**Verification:**
- HTTPS services returned HTTP 302/200 instead of 502
- IMAP/SMTP ports accessible
- DNS resolution working from all pods
**Prevention:** Always explicitly configure kubelet cluster-dns when using non-default service CIDR.
---
## Change History
| Date | Change | By |
|------|--------|-----|
| 2026-07-06 | Initial DNS configuration documentation, fixed DNS CIDR mismatch | CTO Agent |