stack.basicstack.de/apps/stalwart/STABLE-ROUTING-SOLUTION.md

Stalwart Mail - Stable Routing Solution

Created: 2026-07-22
Issue: DEV-359
Author: CTO Agent

Problem Statement

k3s ServiceLB (svclb) with hostPort bindings can leave orphaned CNI-HOSTPORT iptables rules when pods restart, causing intermittent mail service outages.

Root Cause

1. Architecture: k3s ServiceLB uses DaemonSet pods with hostPort to expose LoadBalancer services
2. CNI Plugin: hostPort creates CNI-HOSTPORT iptables DNAT rules
3. Failure Mode: When svclb pods restart, the CNI plugin may fail to clean up old DNAT rules
4. Impact: External traffic gets DNATed to old (non-existent) pod IPs, causing connection timeouts

Historical Issues

• 2026-07-20: Stalwart pod restart left orphaned rules pointing to old svclb pod IPs
• 2026-07-22: Orphaned rules caused complete IMAP/SMTP outage
• Recovery: Required manual deletion of orphaned rules + svclb pod recreation

Recommended Long-Term Solutions

Option 1: Migrate to Hetzner Cloud Load Balancer (RECOMMENDED)

Architecture:

External Traffic → Hetzner LB11 (public IP)
  ↓
k3s worker nodes (private IPs via TargetGroup)
  ↓
Stalwart LoadBalancer service (standard k8s)
  ↓
Stalwart pod

Benefits:

• True external load balancing with HA
• No CNI-HOSTPORT complexity
• Automatic health checks
• SSL termination option
• Multi-node distribution
• No manual iptables management

Cost: ~€8.91/month (LB11 - 5 services)

Implementation: See /hetzner-lb-migration-plan.md in workspace root

Status: Plan approved, awaiting implementation (tracked in DEV-357)

Option 2: Traefik TCP IngressRoute (ALTERNATIVE)

Architecture:

External Traffic → Traefik (ports 25, 587, 993)
  ↓ (TCP routing via IngressRouteTCP)
Stalwart ClusterIP service
  ↓
Stalwart pod

Benefits:

• No additional infrastructure
• Traefik already deployed
• Unified ingress for HTTP + TCP
• No CNI-HOSTPORT dependencies

Drawbacks:

• Requires Traefik configuration changes
• Must expose additional ports on Traefik service
• Single point of failure (unless Traefik is HA)

Implementation Steps:

1. Add SMTP/IMAP entryPoints to Traefik deployment args
2. Add ports 25, 587, 993 to Traefik LoadBalancer service
3. Create IngressRouteTCP resources (already exist in cluster)
4. Change Stalwart services from LoadBalancer to ClusterIP

Status: Not implemented, IngressRouteTCP resources exist but Traefik entryPoints missing

Option 3: MetalLB (NOT RECOMMENDED)

Installing MetalLB to provide real LoadBalancer IPs.

Drawbacks:

• Additional complexity
• Potential conflicts with k3s ServiceLB
• Still uses node IPs (no true external LB)
• Requires IP pool management

Status: Rejected - Hetzner LB or Traefik are better options

Current State (2026-07-22)

Active Solution: k3s ServiceLB with hostPort bindings

Configuration:

• Service type: LoadBalancer with externalTrafficPolicy: Local
• svclb DaemonSet pods on all nodes
• CNI-HOSTPORT iptables rules for port forwarding
• Exposed ports: SMTP (25, 587), IMAPS (993)
• Removed insecure ports: SMTPS (465), IMAP (143)

Known Issues:

• CNI-HOSTPORT rules can become orphaned on pod restart
• Requires manual intervention to clean up orphaned rules
• Not truly multi-node load balanced

Recovery Procedure (if orphaned rules occur):

# 1. Delete all svclb pods to trigger CNI cleanup
kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp'
kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-imap'

# 2. Wait for pods to recreate (CNI will create new hostPort rules)
kubectl wait --for=condition=ready pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp' --timeout=60s

# 3. Verify CNI-HOSTPORT rules on each node
for node in k3s-cp-1 k3s-worker-1 k3s-worker-2 k3s-worker-3; do
  ssh root@$node "iptables-legacy -t nat -L CNI-HOSTPORT-DNAT -n"
done

# 4. Test external connectivity
telnet mail.basicstack.de 587
openssl s_client -connect mail.basicstack.de:993

Recommendation

Implement Option 1 (Hetzner Load Balancer) for the following reasons:

1. Stability: Eliminates CNI-HOSTPORT complexity entirely
2. Scalability: True external load balancing across multiple nodes
3. Reliability: Hetzner-managed infrastructure with automatic health checks
4. Cost-effective: €8.91/month is acceptable for a production mail service
5. Future-proof: Enables future HA Stalwart deployments

Timeline: Can be implemented in ~1 day (see DEV-357 migration plan)

Fallback: If budget/approval is blocked, implement Option 2 (Traefik TCP) as interim solution

---

Related

• Migration Plan: /hetzner-lb-migration-plan.md
• Implementation Task: DEV-357
• Historical Investigation: apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md