# Issue: SMTP/IMAP External Access Not Working **Date**: 2026-07-13 19:55 UTC **Discovered by**: CTO Agent **Related Task**: DEV-235 ## Symptom - Web-UI at https://mail.basicstack.de is **REACHABLE** ✅ - SMTP ports (25, 587, 465) are **NOT accessible** from external clients ❌ - IMAP ports (143, 993) are **NOT accessible** from external clients ❌ ## Investigation ### Architecture Discovery Current Stalwart deployment uses k3s ServiceLB (klipper-lb) with LoadBalancer-type services: ``` External Client ↓ (to 178.105.17.239:25/587/465/143/993) ??? → No direct route Actual Architecture: k3s ServiceLB (svclb) on each node - Uses hostPort binding + iptables DNAT - Forwards to Service ClusterIP - Service forwards to Stalwart pod ``` ### Root Cause Analysis 1. **LoadBalancer VIPs are Internal Pod Network IPs**: ```bash $ kubectl get svc -n stalwart stalwart-smtp LoadBalancer 10.102.204.35 10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5 stalwart-imap LoadBalancer 10.108.223.98 10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5 ``` The "external IPs" (10.42.1.x) are actually internal pod network IPs from the Flannel VXLAN network, NOT publicly routable IPs. 2. **Firewall Rules Are Correct** ✅: ```bash fw-k3s allows: - Port 25 (SMTP) - Port 587 (SMTP submission) - Port 465 (SMTPS) - Port 143 (IMAP) - Port 993 (IMAPS) ``` 3. **iptables DNAT Rules Exist** ✅: ```bash CNI-HOSTPORT-DNAT rules: - Ports 25,587,465 → DNAT to 10.244.0.155 (svclb-stalwart-smtp pod) - Ports 143,993 → DNAT to 10.244.0.153 (svclb-stalwart-imap pod) ``` 4. **Internal Connectivity Works** ✅: ```bash # From control plane node - Direct to pod: ✅ WORKS - Via localhost DNAT: ✅ WORKS - Via service ClusterIP: ✅ WORKS ``` 5. **External Connectivity Path Unclear**: - DNS points mail.basicstack.de → 178.105.17.239 ✅ - Public IP is configured on eth0 ✅ - But external traffic may not be reaching the DNAT rules - Packet counters on KUBE-EXT-* chains show 0 packets for LoadBalancer IPs ### Diagnostic Evidence #### Service Configuration ```yaml apiVersion: v1 kind: Service metadata: name: stalwart-smtp namespace: stalwart spec: type: LoadBalancer selector: app: stalwart ports: - name: smtp port: 25 targetPort: 25 - name: submission port: 587 targetPort: 587 - name: submissions port: 465 targetPort: 465 ``` #### iptables Status (2026-07-13 19:55 UTC) ```bash # PREROUTING chain order (matters!): 1. KUBE-SERVICES (k8s service routing) 2. DOCKER 3. CNI-HOSTPORT-DNAT (hostPort binding) 4. CNI-HOSTPORT-DNAT (duplicate) # DNAT rules for SMTP (port 25): 36 packets → DNAT to 10.244.0.155:25 (from all sources) # K8s service rules: 6 packets → stalwart-smtp ClusterIP (10.102.204.35:25) 0 packets → stalwart-smtp LoadBalancer IPs (10.42.1.x:25) ``` #### svclb Pod Status ```bash svclb-stalwart-smtp-681908de-lx5mj 3/3 Running 1 k3s-cp-1 - Restart count: 1 (containers exited with code 1 initially) - Now running normally - hostPort: 25, 587, 465 - Setting up iptables-legacy rules correctly ``` ## Root Cause **k3s ServiceLB is NOT a true external LoadBalancer.** It's an in-cluster service proxy that: - Assigns internal VIPs (10.42.1.x) from the pod network - Uses hostPort bindings + iptables DNAT to forward traffic - **Does NOT provision external IPs or integrate with cloud providers** For external SMTP/IMAP access, we need one of: ### Option A: Use NodePort Services (RECOMMENDED) Change service type from `LoadBalancer` to `NodePort` and: 1. Update DNS to point directly to node IPs 2. Use the NodePort numbers (already assigned: 31363, 31771, 30342, 31795, 30456) 3. Document that clients must use standard ports (25, 587, 465, 143, 993) via iptables REDIRECT **Pros**: - Simple, no additional infrastructure - Works with existing firewall rules - No cloud provider dependency **Cons**: - Clients see non-standard ports unless we add iptables REDIRECT rules on nodes ### Option B: Deploy MetalLB Install MetalLB to provide real LoadBalancer IPs: 1. Configure IP pool with public IPs (178.105.17.239 or additional IPs) 2. MetalLB assigns real external IPs to LoadBalancer services 3. No DNS or client changes needed **Pros**: - True LoadBalancer functionality - Standard port usage - More "Kubernetes-native" **Cons**: - Additional complexity - Requires IP pool management - May conflict with existing k3s ServiceLB ### Option C: Use Hetzner Cloud LoadBalancers (NOT RECOMMENDED) Provision actual Hetzner Cloud LoadBalancers via hcloud-cloud-controller-manager. **Pros**: - True external LoadBalancers - Managed by Hetzner **Cons**: - Additional cost (~€5-10/month per LB) - Requires ccm installation and configuration - Overkill for this use case ## Fix Applied **UPDATE 2026-07-13 20:10 UTC**: Investigation revealed that k3s ServiceLB was **working correctly all along** via hostPort bindings! **Commit**: ae3f16498736ed6ba0733e6656d82cb47d8eb5a6 **Applied**: 2026-07-13 20:07 UTC (ArgoCD sync) The confusion was caused by misunderstanding the `EXTERNAL-IP` field showing `10.42.1.1` (flannel pod network IP): - k3s ServiceLB does NOT route traffic through that VIP - Instead, it creates svclb DaemonSet pods with hostPort bindings on each node's public IP - External traffic to `178.105.17.239:25` → hostPort binding → service ClusterIP → pod **Verification**: ```bash # All ports responding correctly: $ nc -zv 178.105.17.239 25 587 465 143 993 Connection to 178.105.17.239 25 port [tcp/smtp] succeeded! Connection to 178.105.17.239 587 port [tcp/submission] succeeded! Connection to 178.105.17.239 465 port [tcp/submissions] succeeded! Connection to 178.105.17.239 143 port [tcp/imap] succeeded! Connection to 178.105.17.239 993 port [tcp/imaps] succeeded! # Stalwart responding correctly: $ echo "QUIT" | nc 178.105.17.239 25 220 mail.basicstack.de Stalwart ESMTP at your service 221 2.0.0 Bye. ``` **Change Made**: Added `externalTrafficPolicy: Local` to both LoadBalancer services in git. **Why This Matters**: With `externalTrafficPolicy: Local`, traffic only routes to nodes where the pod is actually running. With the default `Cluster` policy, ArgoCD would create svclb pods on all nodes even when Stalwart is only on one node, causing potential routing issues. **Root Cause of Confusion**: ArgoCD continuously reverted manual `kubectl patch` changes because `externalTrafficPolicy: Local` was not in the git manifest. Now it's committed to git, so ArgoCD won't revert it. ## Verification Plan Once fix is applied: 1. **External SMTP Test**: ```bash telnet mail.basicstack.de 25 # Should see: 220 mail.basicstack.de ESMTP Stalwart ``` 2. **External IMAP Test**: ```bash openssl s_client -connect mail.basicstack.de:993 -crlf # Should connect to IMAPS ``` 3. **Mail Client Test**: - Configure Thunderbird/Outlook with: - SMTP: mail.basicstack.de:587 (STARTTLS) - IMAP: mail.basicstack.de:993 (SSL/TLS) - Send and receive test emails ## Follow-up - [ ] Choose fix approach (A, B, or C) - [ ] Implement chosen solution - [ ] Update DNS_REQUIREMENTS.md with correct configuration - [ ] Update STABILITY-CHECK.md diagnostic playbook - [ ] Test external connectivity from multiple networks - [ ] Document the working architecture ## Related Files - `stalwart-fresh-deployment.yaml` - Service definitions - `infrastructure/networking/DNS_REQUIREMENTS.md` - DNS configuration - `STABILITY-CHECK.md` - Diagnostic procedures ## Technical Details ### Current Service Configuration ```bash kubectl get svc -n stalwart NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) stalwart-smtp LoadBalancer 10.102.204.35 10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5 25:31363/TCP,587:31771/TCP,465:30342/TCP stalwart-imap LoadBalancer 10.108.223.98 10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5 143:31795/TCP,993:30456/TCP stalwart-http ClusterIP 10.99.161.29 8080/TCP ``` ### Node IPs ``` k3s-cp-1: 178.105.17.239 (control plane, target for DNS) k3s-worker-1: 178.105.216.48 k3s-worker-2: 49.13.134.255 k3s-worker-3: 167.233.121.121 ``` ### Required iptables Rules for Option A (NodePort + REDIRECT) ```bash # On each node, redirect standard ports to NodePorts iptables -t nat -A PREROUTING -p tcp --dport 25 -j REDIRECT --to-port 31363 iptables -t nat -A PREROUTING -p tcp --dport 587 -j REDIRECT --to-port 31771 iptables -t nat -A PREROUTING -p tcp --dport 465 -j REDIRECT --to-port 30342 iptables -t nat -A PREROUTING -p tcp --dport 143 -j REDIRECT --to-port 31795 iptables -t nat -A PREROUTING -p tcp --dport 993 -j REDIRECT --to-port 30456 ``` **NOTE**: These rules must be made persistent (via `/etc/iptables/rules.v4` or similar). --- **Next Action**: Request approval from CEO/board on fix approach before implementing.