stack.basicstack.de/apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md

Issue: SMTP/IMAP External Access Not Working

Date: 2026-07-13 19:55 UTC
Discovered by: CTO Agent
Related Task: DEV-235

Symptom

• Web-UI at https://mail.basicstack.de is REACHABLE ✅
• SMTP ports (25, 587, 465) are NOT accessible from external clients ❌
• IMAP ports (143, 993) are NOT accessible from external clients ❌

Investigation

Architecture Discovery

Current Stalwart deployment uses k3s ServiceLB (klipper-lb) with LoadBalancer-type services:

External Client
    ↓ (to 178.105.17.239:25/587/465/143/993)
    ??? → No direct route
    
Actual Architecture:
k3s ServiceLB (svclb) on each node
    - Uses hostPort binding + iptables DNAT
    - Forwards to Service ClusterIP
    - Service forwards to Stalwart pod

Root Cause Analysis

1. LoadBalancer VIPs are Internal Pod Network IPs:

   $ kubectl get svc -n stalwart
   stalwart-smtp   LoadBalancer   10.102.204.35   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5
   stalwart-imap   LoadBalancer   10.108.223.98   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5
   

   The "external IPs" (10.42.1.x) are actually internal pod network IPs from the Flannel VXLAN network, NOT publicly routable IPs.

2. Firewall Rules Are Correct ✅:

   fw-k3s allows:
   - Port 25 (SMTP)
   - Port 587 (SMTP submission)
   - Port 465 (SMTPS)
   - Port 143 (IMAP)
   - Port 993 (IMAPS)
   

3. iptables DNAT Rules Exist ✅:

   CNI-HOSTPORT-DNAT rules:
   - Ports 25,587,465 → DNAT to 10.244.0.155 (svclb-stalwart-smtp pod)
   - Ports 143,993 → DNAT to 10.244.0.153 (svclb-stalwart-imap pod)
   

4. Internal Connectivity Works ✅:

   # From control plane node
   - Direct to pod: ✅ WORKS
   - Via localhost DNAT: ✅ WORKS
   - Via service ClusterIP: ✅ WORKS
   

5. External Connectivity Path Unclear:

   • DNS points mail.basicstack.de → 178.105.17.239 ✅
   • Public IP is configured on eth0 ✅
   • But external traffic may not be reaching the DNAT rules
   • Packet counters on KUBE-EXT-* chains show 0 packets for LoadBalancer IPs

Diagnostic Evidence

Service Configuration

apiVersion: v1
kind: Service
metadata:
  name: stalwart-smtp
  namespace: stalwart
spec:
  type: LoadBalancer
  selector:
    app: stalwart
  ports:
    - name: smtp
      port: 25
      targetPort: 25
    - name: submission
      port: 587
      targetPort: 587
    - name: submissions
      port: 465
      targetPort: 465

iptables Status (2026-07-13 19:55 UTC)

# PREROUTING chain order (matters!):
1. KUBE-SERVICES (k8s service routing)
2. DOCKER
3. CNI-HOSTPORT-DNAT (hostPort binding)
4. CNI-HOSTPORT-DNAT (duplicate)

# DNAT rules for SMTP (port 25):
36 packets → DNAT to 10.244.0.155:25 (from all sources)

# K8s service rules:
6 packets → stalwart-smtp ClusterIP (10.102.204.35:25)
0 packets → stalwart-smtp LoadBalancer IPs (10.42.1.x:25)

svclb Pod Status

svclb-stalwart-smtp-681908de-lx5mj   3/3   Running   1   k3s-cp-1
  - Restart count: 1 (containers exited with code 1 initially)
  - Now running normally
  - hostPort: 25, 587, 465
  - Setting up iptables-legacy rules correctly

Root Cause

k3s ServiceLB is NOT a true external LoadBalancer. It's an in-cluster service proxy that:

• Assigns internal VIPs (10.42.1.x) from the pod network
• Uses hostPort bindings + iptables DNAT to forward traffic
• Does NOT provision external IPs or integrate with cloud providers

For external SMTP/IMAP access, we need one of:

Option A: Use NodePort Services (RECOMMENDED)

Change service type from LoadBalancer to NodePort and:

1. Update DNS to point directly to node IPs
2. Use the NodePort numbers (already assigned: 31363, 31771, 30342, 31795, 30456)
3. Document that clients must use standard ports (25, 587, 465, 143, 993) via iptables REDIRECT

Pros:

• Simple, no additional infrastructure
• Works with existing firewall rules
• No cloud provider dependency

Cons:

• Clients see non-standard ports unless we add iptables REDIRECT rules on nodes

Option B: Deploy MetalLB

Install MetalLB to provide real LoadBalancer IPs:

1. Configure IP pool with public IPs (178.105.17.239 or additional IPs)
2. MetalLB assigns real external IPs to LoadBalancer services
3. No DNS or client changes needed

Pros:

• True LoadBalancer functionality
• Standard port usage
• More "Kubernetes-native"

Cons:

• Additional complexity
• Requires IP pool management
• May conflict with existing k3s ServiceLB

Option C: Use Hetzner Cloud LoadBalancers (NOT RECOMMENDED)

Provision actual Hetzner Cloud LoadBalancers via hcloud-cloud-controller-manager.

Pros:

• True external LoadBalancers
• Managed by Hetzner

Cons:

• Additional cost (~€5-10/month per LB)
• Requires ccm installation and configuration
• Overkill for this use case

Fix Applied

UPDATE 2026-07-13 20:10 UTC: Investigation revealed that k3s ServiceLB was working correctly all along via hostPort bindings!

Commit: ae3f164987
Applied: 2026-07-13 20:07 UTC (ArgoCD sync)

The confusion was caused by misunderstanding the EXTERNAL-IP field showing 10.42.1.1 (flannel pod network IP):

• k3s ServiceLB does NOT route traffic through that VIP
• Instead, it creates svclb DaemonSet pods with hostPort bindings on each node's public IP
• External traffic to 178.105.17.239:25 → hostPort binding → service ClusterIP → pod

Verification:

# All ports responding correctly:
$ nc -zv 178.105.17.239 25 587 465 143 993
Connection to 178.105.17.239 25 port [tcp/smtp] succeeded!
Connection to 178.105.17.239 587 port [tcp/submission] succeeded!
Connection to 178.105.17.239 465 port [tcp/submissions] succeeded!
Connection to 178.105.17.239 143 port [tcp/imap] succeeded!
Connection to 178.105.17.239 993 port [tcp/imaps] succeeded!

# Stalwart responding correctly:
$ echo "QUIT" | nc 178.105.17.239 25
220 mail.basicstack.de Stalwart ESMTP at your service
221 2.0.0 Bye.

Change Made: Added externalTrafficPolicy: Local to both LoadBalancer services in git.

Why This Matters: With externalTrafficPolicy: Local, traffic only routes to nodes where the pod is actually running. With the default Cluster policy, ArgoCD would create svclb pods on all nodes even when Stalwart is only on one node, causing potential routing issues.

Root Cause of Confusion: ArgoCD continuously reverted manual kubectl patch changes because externalTrafficPolicy: Local was not in the git manifest. Now it's committed to git, so ArgoCD won't revert it.

Verification Plan

Once fix is applied:

1. External SMTP Test:

   telnet mail.basicstack.de 25
   # Should see: 220 mail.basicstack.de ESMTP Stalwart
   

2. External IMAP Test:

   openssl s_client -connect mail.basicstack.de:993 -crlf
   # Should connect to IMAPS
   

3. Mail Client Test:

   • Configure Thunderbird/Outlook with:
     • SMTP: mail.basicstack.de:587 (STARTTLS)
     • IMAP: mail.basicstack.de:993 (SSL/TLS)
   • Send and receive test emails

Follow-up

• Choose fix approach (A, B, or C)
• Implement chosen solution
• Update DNS_REQUIREMENTS.md with correct configuration
• Update STABILITY-CHECK.md diagnostic playbook
• Test external connectivity from multiple networks
• Document the working architecture

Related Files

• stalwart-fresh-deployment.yaml - Service definitions
• infrastructure/networking/DNS_REQUIREMENTS.md - DNS configuration
• STABILITY-CHECK.md - Diagnostic procedures

Technical Details

Current Service Configuration

kubectl get svc -n stalwart
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP                               PORT(S)
stalwart-smtp             LoadBalancer   10.102.204.35   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5   25:31363/TCP,587:31771/TCP,465:30342/TCP
stalwart-imap             LoadBalancer   10.108.223.98   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5   143:31795/TCP,993:30456/TCP
stalwart-http             ClusterIP      10.99.161.29    <none>                                    8080/TCP

Node IPs

k3s-cp-1:       178.105.17.239 (control plane, target for DNS)
k3s-worker-1:   178.105.216.48
k3s-worker-2:   49.13.134.255
k3s-worker-3:   167.233.121.121

Required iptables Rules for Option A (NodePort + REDIRECT)

# On each node, redirect standard ports to NodePorts
iptables -t nat -A PREROUTING -p tcp --dport 25 -j REDIRECT --to-port 31363
iptables -t nat -A PREROUTING -p tcp --dport 587 -j REDIRECT --to-port 31771
iptables -t nat -A PREROUTING -p tcp --dport 465 -j REDIRECT --to-port 30342
iptables -t nat -A PREROUTING -p tcp --dport 143 -j REDIRECT --to-port 31795
iptables -t nat -A PREROUTING -p tcp --dport 993 -j REDIRECT --to-port 30456

NOTE: These rules must be made persistent (via /etc/iptables/rules.v4 or similar).

---

Next Action: Request approval from CEO/board on fix approach before implementing.