stack.basicstack.de/apps/stalwart/STABLE-ROUTING-SOLUTION.md

# Stalwart Mail - Stable Routing Solution

**Created**: 2026-07-22  
**Issue**: DEV-359  
**Author**: CTO Agent

## Problem Statement

k3s ServiceLB (svclb) with hostPort bindings can leave orphaned CNI-HOSTPORT iptables rules when pods restart, causing intermittent mail service outages.

### Root Cause

1. **Architecture**: k3s ServiceLB uses DaemonSet pods with hostPort to expose LoadBalancer services
2. **CNI Plugin**: hostPort creates CNI-HOSTPORT iptables DNAT rules
3. **Failure Mode**: When svclb pods restart, the CNI plugin may fail to clean up old DNAT rules
4. **Impact**: External traffic gets DNATed to old (non-existent) pod IPs, causing connection timeouts

### Historical Issues

- **2026-07-20**: Stalwart pod restart left orphaned rules pointing to old svclb pod IPs
- **2026-07-22**: Orphaned rules caused complete IMAP/SMTP outage
- **Recovery**: Required manual deletion of orphaned rules + svclb pod recreation

## Recommended Long-Term Solutions

### Option 1: Migrate to Hetzner Cloud Load Balancer (RECOMMENDED)

**Architecture**:
```
External Traffic → Hetzner LB11 (public IP)
  ↓
k3s worker nodes (private IPs via TargetGroup)
  ↓
Stalwart LoadBalancer service (standard k8s)
  ↓
Stalwart pod
```

**Benefits**:
- True external load balancing with HA
- No CNI-HOSTPORT complexity
- Automatic health checks
- SSL termination option
- Multi-node distribution
- No manual iptables management

**Cost**: ~€8.91/month (LB11 - 5 services)

**Implementation**: See `/hetzner-lb-migration-plan.md` in workspace root

**Status**: Plan approved, awaiting implementation (tracked in DEV-357)

### Option 2: Traefik TCP IngressRoute (ALTERNATIVE)

**Architecture**:
```
External Traffic → Traefik (ports 25, 587, 993)
  ↓ (TCP routing via IngressRouteTCP)
Stalwart ClusterIP service
  ↓
Stalwart pod
```

**Benefits**:
- No additional infrastructure
- Traefik already deployed
- Unified ingress for HTTP + TCP
- No CNI-HOSTPORT dependencies

**Drawbacks**:
- Requires Traefik configuration changes
- Must expose additional ports on Traefik service
- Single point of failure (unless Traefik is HA)

**Implementation Steps**:
1. Add SMTP/IMAP entryPoints to Traefik deployment args
2. Add ports 25, 587, 993 to Traefik LoadBalancer service
3. Create IngressRouteTCP resources (already exist in cluster)
4. Change Stalwart services from LoadBalancer to ClusterIP

**Status**: Not implemented, IngressRouteTCP resources exist but Traefik entryPoints missing

### Option 3: MetalLB (NOT RECOMMENDED)

Installing MetalLB to provide real LoadBalancer IPs.

**Drawbacks**:
- Additional complexity
- Potential conflicts with k3s ServiceLB
- Still uses node IPs (no true external LB)
- Requires IP pool management

**Status**: Rejected - Hetzner LB or Traefik are better options

## Current State (2026-07-22)

**Active Solution**: k3s ServiceLB with hostPort bindings

**Configuration**:
- Service type: LoadBalancer with `externalTrafficPolicy: Local`
- svclb DaemonSet pods on all nodes
- CNI-HOSTPORT iptables rules for port forwarding
- Exposed ports: SMTP (25, 587), IMAPS (993)
- Removed insecure ports: SMTPS (465), IMAP (143)

**Known Issues**:
- CNI-HOSTPORT rules can become orphaned on pod restart
- Requires manual intervention to clean up orphaned rules
- Not truly multi-node load balanced

**Recovery Procedure** (if orphaned rules occur):
```bash
# 1. Delete all svclb pods to trigger CNI cleanup
kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp'
kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-imap'

# 2. Wait for pods to recreate (CNI will create new hostPort rules)
kubectl wait --for=condition=ready pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp' --timeout=60s

# 3. Verify CNI-HOSTPORT rules on each node
for node in k3s-cp-1 k3s-worker-1 k3s-worker-2 k3s-worker-3; do
  ssh root@$node "iptables-legacy -t nat -L CNI-HOSTPORT-DNAT -n"
done

# 4. Test external connectivity
telnet mail.basicstack.de 587
openssl s_client -connect mail.basicstack.de:993
```

## Recommendation

**Implement Option 1 (Hetzner Load Balancer)** for the following reasons:

1. **Stability**: Eliminates CNI-HOSTPORT complexity entirely
2. **Scalability**: True external load balancing across multiple nodes
3. **Reliability**: Hetzner-managed infrastructure with automatic health checks
4. **Cost-effective**: €8.91/month is acceptable for a production mail service
5. **Future-proof**: Enables future HA Stalwart deployments

**Timeline**: Can be implemented in ~1 day (see DEV-357 migration plan)

**Fallback**: If budget/approval is blocked, implement Option 2 (Traefik TCP) as interim solution

---

## Related

- Migration Plan: `/hetzner-lb-migration-plan.md`
- Implementation Task: DEV-357
- Historical Investigation: `apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md`