stack.basicstack.de/apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md

Issue: SMTP/IMAP External Access Not Working

Date: 2026-07-13 19:55 UTC
Discovered by: CTO Agent
Related Task: DEV-235

Symptom

• Web-UI at https://mail.basicstack.de is REACHABLE ✅
• SMTP ports (25, 587, 465) are NOT accessible from external clients ❌
• IMAP ports (143, 993) are NOT accessible from external clients ❌

Investigation

Architecture Discovery

Current Stalwart deployment uses k3s ServiceLB (klipper-lb) with LoadBalancer-type services:

External Client
    ↓ (to 178.105.17.239:25/587/465/143/993)
    ??? → No direct route
    
Actual Architecture:
k3s ServiceLB (svclb) on each node
    - Uses hostPort binding + iptables DNAT
    - Forwards to Service ClusterIP
    - Service forwards to Stalwart pod

Root Cause Analysis

1. LoadBalancer VIPs are Internal Pod Network IPs:

   $ kubectl get svc -n stalwart
   stalwart-smtp   LoadBalancer   10.102.204.35   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5
   stalwart-imap   LoadBalancer   10.108.223.98   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5
   

   The "external IPs" (10.42.1.x) are actually internal pod network IPs from the Flannel VXLAN network, NOT publicly routable IPs.

2. Firewall Rules Are Correct ✅:

   fw-k3s allows:
   - Port 25 (SMTP)
   - Port 587 (SMTP submission)
   - Port 465 (SMTPS)
   - Port 143 (IMAP)
   - Port 993 (IMAPS)
   

3. iptables DNAT Rules Exist ✅:

   CNI-HOSTPORT-DNAT rules:
   - Ports 25,587,465 → DNAT to 10.244.0.155 (svclb-stalwart-smtp pod)
   - Ports 143,993 → DNAT to 10.244.0.153 (svclb-stalwart-imap pod)
   

4. Internal Connectivity Works ✅:

   # From control plane node
   - Direct to pod: ✅ WORKS
   - Via localhost DNAT: ✅ WORKS
   - Via service ClusterIP: ✅ WORKS
   

5. External Connectivity Path Unclear:

   • DNS points mail.basicstack.de → 178.105.17.239 ✅
   • Public IP is configured on eth0 ✅
   • But external traffic may not be reaching the DNAT rules
   • Packet counters on KUBE-EXT-* chains show 0 packets for LoadBalancer IPs

Diagnostic Evidence

Service Configuration

apiVersion: v1
kind: Service
metadata:
  name: stalwart-smtp
  namespace: stalwart
spec:
  type: LoadBalancer
  selector:
    app: stalwart
  ports:
    - name: smtp
      port: 25
      targetPort: 25
    - name: submission
      port: 587
      targetPort: 587
    - name: submissions
      port: 465
      targetPort: 465

iptables Status (2026-07-13 19:55 UTC)

# PREROUTING chain order (matters!):
1. KUBE-SERVICES (k8s service routing)
2. DOCKER
3. CNI-HOSTPORT-DNAT (hostPort binding)
4. CNI-HOSTPORT-DNAT (duplicate)

# DNAT rules for SMTP (port 25):
36 packets → DNAT to 10.244.0.155:25 (from all sources)

# K8s service rules:
6 packets → stalwart-smtp ClusterIP (10.102.204.35:25)
0 packets → stalwart-smtp LoadBalancer IPs (10.42.1.x:25)

svclb Pod Status

svclb-stalwart-smtp-681908de-lx5mj   3/3   Running   1   k3s-cp-1
  - Restart count: 1 (containers exited with code 1 initially)
  - Now running normally
  - hostPort: 25, 587, 465
  - Setting up iptables-legacy rules correctly

Root Cause

k3s ServiceLB is NOT a true external LoadBalancer. It's an in-cluster service proxy that:

• Assigns internal VIPs (10.42.1.x) from the pod network
• Uses hostPort bindings + iptables DNAT to forward traffic
• Does NOT provision external IPs or integrate with cloud providers

For external SMTP/IMAP access, we need one of:

Option A: Use NodePort Services (RECOMMENDED)

Change service type from LoadBalancer to NodePort and:

1. Update DNS to point directly to node IPs
2. Use the NodePort numbers (already assigned: 31363, 31771, 30342, 31795, 30456)
3. Document that clients must use standard ports (25, 587, 465, 143, 993) via iptables REDIRECT

Pros:

• Simple, no additional infrastructure
• Works with existing firewall rules
• No cloud provider dependency

Cons:

• Clients see non-standard ports unless we add iptables REDIRECT rules on nodes

Option B: Deploy MetalLB

Install MetalLB to provide real LoadBalancer IPs:

1. Configure IP pool with public IPs (178.105.17.239 or additional IPs)
2. MetalLB assigns real external IPs to LoadBalancer services
3. No DNS or client changes needed

Pros:

• True LoadBalancer functionality
• Standard port usage
• More "Kubernetes-native"

Cons:

• Additional complexity
• Requires IP pool management
• May conflict with existing k3s ServiceLB

Option C: Use Hetzner Cloud LoadBalancers (NOT RECOMMENDED)

Provision actual Hetzner Cloud LoadBalancers via hcloud-cloud-controller-manager.

Pros:

• True external LoadBalancers
• Managed by Hetzner

Cons:

• Additional cost (~€5-10/month per LB)
• Requires ccm installation and configuration
• Overkill for this use case

Fix Applied

NONE YET - Awaiting approval on approach.

Verification Plan

Once fix is applied:

1. External SMTP Test:

   telnet mail.basicstack.de 25
   # Should see: 220 mail.basicstack.de ESMTP Stalwart
   

2. External IMAP Test:

   openssl s_client -connect mail.basicstack.de:993 -crlf
   # Should connect to IMAPS
   

3. Mail Client Test:

   • Configure Thunderbird/Outlook with:
     • SMTP: mail.basicstack.de:587 (STARTTLS)
     • IMAP: mail.basicstack.de:993 (SSL/TLS)
   • Send and receive test emails

Follow-up

• Choose fix approach (A, B, or C)
• Implement chosen solution
• Update DNS_REQUIREMENTS.md with correct configuration
• Update STABILITY-CHECK.md diagnostic playbook
• Test external connectivity from multiple networks
• Document the working architecture

Related Files

• stalwart-fresh-deployment.yaml - Service definitions
• infrastructure/networking/DNS_REQUIREMENTS.md - DNS configuration
• STABILITY-CHECK.md - Diagnostic procedures

Technical Details

Current Service Configuration

kubectl get svc -n stalwart
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP                               PORT(S)
stalwart-smtp             LoadBalancer   10.102.204.35   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5   25:31363/TCP,587:31771/TCP,465:30342/TCP
stalwart-imap             LoadBalancer   10.108.223.98   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5   143:31795/TCP,993:30456/TCP
stalwart-http             ClusterIP      10.99.161.29    <none>                                    8080/TCP

Node IPs

k3s-cp-1:       178.105.17.239 (control plane, target for DNS)
k3s-worker-1:   178.105.216.48
k3s-worker-2:   49.13.134.255
k3s-worker-3:   167.233.121.121

Required iptables Rules for Option A (NodePort + REDIRECT)

# On each node, redirect standard ports to NodePorts
iptables -t nat -A PREROUTING -p tcp --dport 25 -j REDIRECT --to-port 31363
iptables -t nat -A PREROUTING -p tcp --dport 587 -j REDIRECT --to-port 31771
iptables -t nat -A PREROUTING -p tcp --dport 465 -j REDIRECT --to-port 30342
iptables -t nat -A PREROUTING -p tcp --dport 143 -j REDIRECT --to-port 31795
iptables -t nat -A PREROUTING -p tcp --dport 993 -j REDIRECT --to-port 30456

NOTE: These rules must be made persistent (via /etc/iptables/rules.v4 or similar).

---

Next Action: Request approval from CEO/board on fix approach before implementing.