stack.basicstack.de/apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md

# Issue: SMTP/IMAP External Access Not Working

**Date**: 2026-07-13 19:55 UTC  
**Discovered by**: CTO Agent  
**Related Task**: DEV-235

## Symptom

- Web-UI at https://mail.basicstack.de is **REACHABLE** ✅
- SMTP ports (25, 587, 465) are **NOT accessible** from external clients ❌
- IMAP ports (143, 993) are **NOT accessible** from external clients ❌

## Investigation

### Architecture Discovery

Current Stalwart deployment uses k3s ServiceLB (klipper-lb) with LoadBalancer-type services:

```
External Client
    ↓ (to 178.105.17.239:25/587/465/143/993)
    ??? → No direct route
    
Actual Architecture:
k3s ServiceLB (svclb) on each node
    - Uses hostPort binding + iptables DNAT
    - Forwards to Service ClusterIP
    - Service forwards to Stalwart pod
```

### Root Cause Analysis

1. **LoadBalancer VIPs are Internal Pod Network IPs**:
   ```bash
   $ kubectl get svc -n stalwart
   stalwart-smtp   LoadBalancer   10.102.204.35   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5
   stalwart-imap   LoadBalancer   10.108.223.98   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5
   ```
   
   The "external IPs" (10.42.1.x) are actually internal pod network IPs from the Flannel VXLAN network, NOT publicly routable IPs.

2. **Firewall Rules Are Correct** ✅:
   ```bash
   fw-k3s allows:
   - Port 25 (SMTP)
   - Port 587 (SMTP submission)
   - Port 465 (SMTPS)
   - Port 143 (IMAP)
   - Port 993 (IMAPS)
   ```

3. **iptables DNAT Rules Exist** ✅:
   ```bash
   CNI-HOSTPORT-DNAT rules:
   - Ports 25,587,465 → DNAT to 10.244.0.155 (svclb-stalwart-smtp pod)
   - Ports 143,993 → DNAT to 10.244.0.153 (svclb-stalwart-imap pod)
   ```

4. **Internal Connectivity Works** ✅:
   ```bash
   # From control plane node
   - Direct to pod: ✅ WORKS
   - Via localhost DNAT: ✅ WORKS
   - Via service ClusterIP: ✅ WORKS
   ```

5. **External Connectivity Path Unclear**:
   - DNS points mail.basicstack.de → 178.105.17.239 ✅
   - Public IP is configured on eth0 ✅
   - But external traffic may not be reaching the DNAT rules
   - Packet counters on KUBE-EXT-* chains show 0 packets for LoadBalancer IPs

### Diagnostic Evidence

#### Service Configuration
```yaml
apiVersion: v1
kind: Service
metadata:
  name: stalwart-smtp
  namespace: stalwart
spec:
  type: LoadBalancer
  selector:
    app: stalwart
  ports:
    - name: smtp
      port: 25
      targetPort: 25
    - name: submission
      port: 587
      targetPort: 587
    - name: submissions
      port: 465
      targetPort: 465
```

#### iptables Status (2026-07-13 19:55 UTC)
```bash
# PREROUTING chain order (matters!):
1. KUBE-SERVICES (k8s service routing)
2. DOCKER
3. CNI-HOSTPORT-DNAT (hostPort binding)
4. CNI-HOSTPORT-DNAT (duplicate)

# DNAT rules for SMTP (port 25):
36 packets → DNAT to 10.244.0.155:25 (from all sources)

# K8s service rules:
6 packets → stalwart-smtp ClusterIP (10.102.204.35:25)
0 packets → stalwart-smtp LoadBalancer IPs (10.42.1.x:25)
```

#### svclb Pod Status
```bash
svclb-stalwart-smtp-681908de-lx5mj   3/3   Running   1   k3s-cp-1
  - Restart count: 1 (containers exited with code 1 initially)
  - Now running normally
  - hostPort: 25, 587, 465
  - Setting up iptables-legacy rules correctly
```

## Root Cause

**k3s ServiceLB is NOT a true external LoadBalancer.** It's an in-cluster service proxy that:
- Assigns internal VIPs (10.42.1.x) from the pod network
- Uses hostPort bindings + iptables DNAT to forward traffic
- **Does NOT provision external IPs or integrate with cloud providers**

For external SMTP/IMAP access, we need one of:

### Option A: Use NodePort Services (RECOMMENDED)

Change service type from `LoadBalancer` to `NodePort` and:
1. Update DNS to point directly to node IPs
2. Use the NodePort numbers (already assigned: 31363, 31771, 30342, 31795, 30456)
3. Document that clients must use standard ports (25, 587, 465, 143, 993) via iptables REDIRECT

**Pros**:
- Simple, no additional infrastructure
- Works with existing firewall rules
- No cloud provider dependency

**Cons**:
- Clients see non-standard ports unless we add iptables REDIRECT rules on nodes

### Option B: Deploy MetalLB

Install MetalLB to provide real LoadBalancer IPs:
1. Configure IP pool with public IPs (178.105.17.239 or additional IPs)
2. MetalLB assigns real external IPs to LoadBalancer services
3. No DNS or client changes needed

**Pros**:
- True LoadBalancer functionality
- Standard port usage
- More "Kubernetes-native"

**Cons**:
- Additional complexity
- Requires IP pool management
- May conflict with existing k3s ServiceLB

### Option C: Use Hetzner Cloud LoadBalancers (NOT RECOMMENDED)

Provision actual Hetzner Cloud LoadBalancers via hcloud-cloud-controller-manager.

**Pros**:
- True external LoadBalancers
- Managed by Hetzner

**Cons**:
- Additional cost (~€5-10/month per LB)
- Requires ccm installation and configuration
- Overkill for this use case

## Fix Applied

**UPDATE 2026-07-13 20:10 UTC**: Investigation revealed that k3s ServiceLB was **working correctly all along** via hostPort bindings!

**Commit**: ae3f16498736ed6ba0733e6656d82cb47d8eb5a6  
**Applied**: 2026-07-13 20:07 UTC (ArgoCD sync)

The confusion was caused by misunderstanding the `EXTERNAL-IP` field showing `10.42.1.1` (flannel pod network IP):
- k3s ServiceLB does NOT route traffic through that VIP
- Instead, it creates svclb DaemonSet pods with hostPort bindings on each node's public IP
- External traffic to `178.105.17.239:25` → hostPort binding → service ClusterIP → pod

**Verification**:
```bash
# All ports responding correctly:
$ nc -zv 178.105.17.239 25 587 465 143 993
Connection to 178.105.17.239 25 port [tcp/smtp] succeeded!
Connection to 178.105.17.239 587 port [tcp/submission] succeeded!
Connection to 178.105.17.239 465 port [tcp/submissions] succeeded!
Connection to 178.105.17.239 143 port [tcp/imap] succeeded!
Connection to 178.105.17.239 993 port [tcp/imaps] succeeded!

# Stalwart responding correctly:
$ echo "QUIT" | nc 178.105.17.239 25
220 mail.basicstack.de Stalwart ESMTP at your service
221 2.0.0 Bye.
```

**Change Made**: Added `externalTrafficPolicy: Local` to both LoadBalancer services in git.

**Why This Matters**: With `externalTrafficPolicy: Local`, traffic only routes to nodes where the pod is actually running. With the default `Cluster` policy, ArgoCD would create svclb pods on all nodes even when Stalwart is only on one node, causing potential routing issues.

**Root Cause of Confusion**: ArgoCD continuously reverted manual `kubectl patch` changes because `externalTrafficPolicy: Local` was not in the git manifest. Now it's committed to git, so ArgoCD won't revert it.

## Verification Plan

Once fix is applied:

1. **External SMTP Test**:
   ```bash
   telnet mail.basicstack.de 25
   # Should see: 220 mail.basicstack.de ESMTP Stalwart
   ```

2. **External IMAP Test**:
   ```bash
   openssl s_client -connect mail.basicstack.de:993 -crlf
   # Should connect to IMAPS
   ```

3. **Mail Client Test**:
   - Configure Thunderbird/Outlook with:
     - SMTP: mail.basicstack.de:587 (STARTTLS)
     - IMAP: mail.basicstack.de:993 (SSL/TLS)
   - Send and receive test emails

## Follow-up

- [ ] Choose fix approach (A, B, or C)
- [ ] Implement chosen solution
- [ ] Update DNS_REQUIREMENTS.md with correct configuration
- [ ] Update STABILITY-CHECK.md diagnostic playbook
- [ ] Test external connectivity from multiple networks
- [ ] Document the working architecture

## Related Files

- `stalwart-fresh-deployment.yaml` - Service definitions
- `infrastructure/networking/DNS_REQUIREMENTS.md` - DNS configuration
- `STABILITY-CHECK.md` - Diagnostic procedures

## Technical Details

### Current Service Configuration
```bash
kubectl get svc -n stalwart
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP                               PORT(S)
stalwart-smtp             LoadBalancer   10.102.204.35   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5   25:31363/TCP,587:31771/TCP,465:30342/TCP
stalwart-imap             LoadBalancer   10.108.223.98   10.42.1.1,10.42.1.2,10.42.1.3,10.42.1.5   143:31795/TCP,993:30456/TCP
stalwart-http             ClusterIP      10.99.161.29    <none>                                    8080/TCP
```

### Node IPs
```
k3s-cp-1:       178.105.17.239 (control plane, target for DNS)
k3s-worker-1:   178.105.216.48
k3s-worker-2:   49.13.134.255
k3s-worker-3:   167.233.121.121
```

### Required iptables Rules for Option A (NodePort + REDIRECT)
```bash
# On each node, redirect standard ports to NodePorts
iptables -t nat -A PREROUTING -p tcp --dport 25 -j REDIRECT --to-port 31363
iptables -t nat -A PREROUTING -p tcp --dport 587 -j REDIRECT --to-port 31771
iptables -t nat -A PREROUTING -p tcp --dport 465 -j REDIRECT --to-port 30342
iptables -t nat -A PREROUTING -p tcp --dport 143 -j REDIRECT --to-port 31795
iptables -t nat -A PREROUTING -p tcp --dport 993 -j REDIRECT --to-port 30456
```

**NOTE**: These rules must be made persistent (via `/etc/iptables/rules.v4` or similar).

---

**Next Action**: Request approval from CEO/board on fix approach before implementing.