From c755bc1a45730dddab73ff4779a21af7900bd25e Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Mon, 13 Jul 2026 20:00:19 +0000 Subject: [PATCH] Document SMTP/IMAP external access issue and solution options Root cause: k3s ServiceLB assigns internal VIPs (10.42.1.x) that are not publicly routable. External traffic to mail.basicstack.de cannot reach the SMTP/IMAP services. Investigation shows: - Web-UI works (goes through Traefik IngressRoute) - SMTP/IMAP ports have correct firewall rules - iptables DNAT rules exist but don't help external traffic - Internal connectivity works correctly - LoadBalancer 'external IPs' are actually pod network IPs Three solution options documented: A. NodePort + iptables REDIRECT (recommended) B. Deploy MetalLB for true LoadBalancer IPs C. Hetzner Cloud LoadBalancers (not recommended, cost) Awaiting approval on approach before implementing fix. Related: DEV-235 Co-Authored-By: Paperclip --- ...UE-2026-07-13-smtp-imap-external-access.md | 251 ++++++++++++++++++ 1 file changed, 251 insertions(+) create mode 100644 apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md diff --git a/apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md b/apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md new file mode 100644 index 0000000..09b3ddb --- /dev/null +++ b/apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md @@ -0,0 +1,251 @@ +# Issue: SMTP/IMAP External Access Not Working + +**Date**: 2026-07-13 19:55 UTC +**Discovered by**: CTO Agent +**Related Task**: DEV-235 + +## Symptom + +- Web-UI at https://mail.basicstack.de is **REACHABLE** ✅ +- SMTP ports (25, 587, 465) are **NOT accessible** from external clients ❌ +- IMAP ports (143, 993) are **NOT accessible** from external clients ❌ + +## Investigation + +### Architecture Discovery + +Current Stalwart deployment uses k3s ServiceLB (klipper-lb) with LoadBalancer-type services: + +``` +External Client + ↓ (to 178.105.17.239:25/587/465/143/993) + ??? → No direct route + +Actual Architecture: +k3s ServiceLB (svclb) on each node + - Uses hostPort binding + iptables DNAT + - Forwards to Service ClusterIP + - Service forwards to Stalwart pod +``` + +### Root Cause Analysis + +1. **LoadBalancer VIPs are Internal Pod Network IPs**: + ```bash + $ kubectl get svc -n stalwart + stalwart-smtp LoadBalancer 10.102.204.35 10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5 + stalwart-imap LoadBalancer 10.108.223.98 10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5 + ``` + + The "external IPs" (10.42.1.x) are actually internal pod network IPs from the Flannel VXLAN network, NOT publicly routable IPs. + +2. **Firewall Rules Are Correct** ✅: + ```bash + fw-k3s allows: + - Port 25 (SMTP) + - Port 587 (SMTP submission) + - Port 465 (SMTPS) + - Port 143 (IMAP) + - Port 993 (IMAPS) + ``` + +3. **iptables DNAT Rules Exist** ✅: + ```bash + CNI-HOSTPORT-DNAT rules: + - Ports 25,587,465 → DNAT to 10.244.0.155 (svclb-stalwart-smtp pod) + - Ports 143,993 → DNAT to 10.244.0.153 (svclb-stalwart-imap pod) + ``` + +4. **Internal Connectivity Works** ✅: + ```bash + # From control plane node + - Direct to pod: ✅ WORKS + - Via localhost DNAT: ✅ WORKS + - Via service ClusterIP: ✅ WORKS + ``` + +5. **External Connectivity Path Unclear**: + - DNS points mail.basicstack.de → 178.105.17.239 ✅ + - Public IP is configured on eth0 ✅ + - But external traffic may not be reaching the DNAT rules + - Packet counters on KUBE-EXT-* chains show 0 packets for LoadBalancer IPs + +### Diagnostic Evidence + +#### Service Configuration +```yaml +apiVersion: v1 +kind: Service +metadata: + name: stalwart-smtp + namespace: stalwart +spec: + type: LoadBalancer + selector: + app: stalwart + ports: + - name: smtp + port: 25 + targetPort: 25 + - name: submission + port: 587 + targetPort: 587 + - name: submissions + port: 465 + targetPort: 465 +``` + +#### iptables Status (2026-07-13 19:55 UTC) +```bash +# PREROUTING chain order (matters!): +1. KUBE-SERVICES (k8s service routing) +2. DOCKER +3. CNI-HOSTPORT-DNAT (hostPort binding) +4. CNI-HOSTPORT-DNAT (duplicate) + +# DNAT rules for SMTP (port 25): +36 packets → DNAT to 10.244.0.155:25 (from all sources) + +# K8s service rules: +6 packets → stalwart-smtp ClusterIP (10.102.204.35:25) +0 packets → stalwart-smtp LoadBalancer IPs (10.42.1.x:25) +``` + +#### svclb Pod Status +```bash +svclb-stalwart-smtp-681908de-lx5mj 3/3 Running 1 k3s-cp-1 + - Restart count: 1 (containers exited with code 1 initially) + - Now running normally + - hostPort: 25, 587, 465 + - Setting up iptables-legacy rules correctly +``` + +## Root Cause + +**k3s ServiceLB is NOT a true external LoadBalancer.** It's an in-cluster service proxy that: +- Assigns internal VIPs (10.42.1.x) from the pod network +- Uses hostPort bindings + iptables DNAT to forward traffic +- **Does NOT provision external IPs or integrate with cloud providers** + +For external SMTP/IMAP access, we need one of: + +### Option A: Use NodePort Services (RECOMMENDED) + +Change service type from `LoadBalancer` to `NodePort` and: +1. Update DNS to point directly to node IPs +2. Use the NodePort numbers (already assigned: 31363, 31771, 30342, 31795, 30456) +3. Document that clients must use standard ports (25, 587, 465, 143, 993) via iptables REDIRECT + +**Pros**: +- Simple, no additional infrastructure +- Works with existing firewall rules +- No cloud provider dependency + +**Cons**: +- Clients see non-standard ports unless we add iptables REDIRECT rules on nodes + +### Option B: Deploy MetalLB + +Install MetalLB to provide real LoadBalancer IPs: +1. Configure IP pool with public IPs (178.105.17.239 or additional IPs) +2. MetalLB assigns real external IPs to LoadBalancer services +3. No DNS or client changes needed + +**Pros**: +- True LoadBalancer functionality +- Standard port usage +- More "Kubernetes-native" + +**Cons**: +- Additional complexity +- Requires IP pool management +- May conflict with existing k3s ServiceLB + +### Option C: Use Hetzner Cloud LoadBalancers (NOT RECOMMENDED) + +Provision actual Hetzner Cloud LoadBalancers via hcloud-cloud-controller-manager. + +**Pros**: +- True external LoadBalancers +- Managed by Hetzner + +**Cons**: +- Additional cost (~€5-10/month per LB) +- Requires ccm installation and configuration +- Overkill for this use case + +## Fix Applied + +**NONE YET** - Awaiting approval on approach. + +## Verification Plan + +Once fix is applied: + +1. **External SMTP Test**: + ```bash + telnet mail.basicstack.de 25 + # Should see: 220 mail.basicstack.de ESMTP Stalwart + ``` + +2. **External IMAP Test**: + ```bash + openssl s_client -connect mail.basicstack.de:993 -crlf + # Should connect to IMAPS + ``` + +3. **Mail Client Test**: + - Configure Thunderbird/Outlook with: + - SMTP: mail.basicstack.de:587 (STARTTLS) + - IMAP: mail.basicstack.de:993 (SSL/TLS) + - Send and receive test emails + +## Follow-up + +- [ ] Choose fix approach (A, B, or C) +- [ ] Implement chosen solution +- [ ] Update DNS_REQUIREMENTS.md with correct configuration +- [ ] Update STABILITY-CHECK.md diagnostic playbook +- [ ] Test external connectivity from multiple networks +- [ ] Document the working architecture + +## Related Files + +- `stalwart-fresh-deployment.yaml` - Service definitions +- `infrastructure/networking/DNS_REQUIREMENTS.md` - DNS configuration +- `STABILITY-CHECK.md` - Diagnostic procedures + +## Technical Details + +### Current Service Configuration +```bash +kubectl get svc -n stalwart +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) +stalwart-smtp LoadBalancer 10.102.204.35 10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5 25:31363/TCP,587:31771/TCP,465:30342/TCP +stalwart-imap LoadBalancer 10.108.223.98 10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5 143:31795/TCP,993:30456/TCP +stalwart-http ClusterIP 10.99.161.29 8080/TCP +``` + +### Node IPs +``` +k3s-cp-1: 178.105.17.239 (control plane, target for DNS) +k3s-worker-1: 178.105.216.48 +k3s-worker-2: 49.13.134.255 +k3s-worker-3: 167.233.121.121 +``` + +### Required iptables Rules for Option A (NodePort + REDIRECT) +```bash +# On each node, redirect standard ports to NodePorts +iptables -t nat -A PREROUTING -p tcp --dport 25 -j REDIRECT --to-port 31363 +iptables -t nat -A PREROUTING -p tcp --dport 587 -j REDIRECT --to-port 31771 +iptables -t nat -A PREROUTING -p tcp --dport 465 -j REDIRECT --to-port 30342 +iptables -t nat -A PREROUTING -p tcp --dport 143 -j REDIRECT --to-port 31795 +iptables -t nat -A PREROUTING -p tcp --dport 993 -j REDIRECT --to-port 30456 +``` + +**NOTE**: These rules must be made persistent (via `/etc/iptables/rules.v4` or similar). + +--- + +**Next Action**: Request approval from CEO/board on fix approach before implementing.