diff --git a/apps/stalwart/STABLE-ROUTING-SOLUTION.md b/apps/stalwart/STABLE-ROUTING-SOLUTION.md new file mode 100644 index 0000000..e6d5900 --- /dev/null +++ b/apps/stalwart/STABLE-ROUTING-SOLUTION.md @@ -0,0 +1,150 @@ +# Stalwart Mail - Stable Routing Solution + +**Created**: 2026-07-22 +**Issue**: DEV-359 +**Author**: CTO Agent + +## Problem Statement + +k3s ServiceLB (svclb) with hostPort bindings can leave orphaned CNI-HOSTPORT iptables rules when pods restart, causing intermittent mail service outages. + +### Root Cause + +1. **Architecture**: k3s ServiceLB uses DaemonSet pods with hostPort to expose LoadBalancer services +2. **CNI Plugin**: hostPort creates CNI-HOSTPORT iptables DNAT rules +3. **Failure Mode**: When svclb pods restart, the CNI plugin may fail to clean up old DNAT rules +4. **Impact**: External traffic gets DNATed to old (non-existent) pod IPs, causing connection timeouts + +### Historical Issues + +- **2026-07-20**: Stalwart pod restart left orphaned rules pointing to old svclb pod IPs +- **2026-07-22**: Orphaned rules caused complete IMAP/SMTP outage +- **Recovery**: Required manual deletion of orphaned rules + svclb pod recreation + +## Recommended Long-Term Solutions + +### Option 1: Migrate to Hetzner Cloud Load Balancer (RECOMMENDED) + +**Architecture**: +``` +External Traffic → Hetzner LB11 (public IP) + ↓ +k3s worker nodes (private IPs via TargetGroup) + ↓ +Stalwart LoadBalancer service (standard k8s) + ↓ +Stalwart pod +``` + +**Benefits**: +- True external load balancing with HA +- No CNI-HOSTPORT complexity +- Automatic health checks +- SSL termination option +- Multi-node distribution +- No manual iptables management + +**Cost**: ~€8.91/month (LB11 - 5 services) + +**Implementation**: See `/hetzner-lb-migration-plan.md` in workspace root + +**Status**: Plan approved, awaiting implementation (tracked in DEV-357) + +### Option 2: Traefik TCP IngressRoute (ALTERNATIVE) + +**Architecture**: +``` +External Traffic → Traefik (ports 25, 587, 993) + ↓ (TCP routing via IngressRouteTCP) +Stalwart ClusterIP service + ↓ +Stalwart pod +``` + +**Benefits**: +- No additional infrastructure +- Traefik already deployed +- Unified ingress for HTTP + TCP +- No CNI-HOSTPORT dependencies + +**Drawbacks**: +- Requires Traefik configuration changes +- Must expose additional ports on Traefik service +- Single point of failure (unless Traefik is HA) + +**Implementation Steps**: +1. Add SMTP/IMAP entryPoints to Traefik deployment args +2. Add ports 25, 587, 993 to Traefik LoadBalancer service +3. Create IngressRouteTCP resources (already exist in cluster) +4. Change Stalwart services from LoadBalancer to ClusterIP + +**Status**: Not implemented, IngressRouteTCP resources exist but Traefik entryPoints missing + +### Option 3: MetalLB (NOT RECOMMENDED) + +Installing MetalLB to provide real LoadBalancer IPs. + +**Drawbacks**: +- Additional complexity +- Potential conflicts with k3s ServiceLB +- Still uses node IPs (no true external LB) +- Requires IP pool management + +**Status**: Rejected - Hetzner LB or Traefik are better options + +## Current State (2026-07-22) + +**Active Solution**: k3s ServiceLB with hostPort bindings + +**Configuration**: +- Service type: LoadBalancer with `externalTrafficPolicy: Local` +- svclb DaemonSet pods on all nodes +- CNI-HOSTPORT iptables rules for port forwarding +- Exposed ports: SMTP (25, 587), IMAPS (993) +- Removed insecure ports: SMTPS (465), IMAP (143) + +**Known Issues**: +- CNI-HOSTPORT rules can become orphaned on pod restart +- Requires manual intervention to clean up orphaned rules +- Not truly multi-node load balanced + +**Recovery Procedure** (if orphaned rules occur): +```bash +# 1. Delete all svclb pods to trigger CNI cleanup +kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp' +kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-imap' + +# 2. Wait for pods to recreate (CNI will create new hostPort rules) +kubectl wait --for=condition=ready pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp' --timeout=60s + +# 3. Verify CNI-HOSTPORT rules on each node +for node in k3s-cp-1 k3s-worker-1 k3s-worker-2 k3s-worker-3; do + ssh root@$node "iptables-legacy -t nat -L CNI-HOSTPORT-DNAT -n" +done + +# 4. Test external connectivity +telnet mail.basicstack.de 587 +openssl s_client -connect mail.basicstack.de:993 +``` + +## Recommendation + +**Implement Option 1 (Hetzner Load Balancer)** for the following reasons: + +1. **Stability**: Eliminates CNI-HOSTPORT complexity entirely +2. **Scalability**: True external load balancing across multiple nodes +3. **Reliability**: Hetzner-managed infrastructure with automatic health checks +4. **Cost-effective**: €8.91/month is acceptable for a production mail service +5. **Future-proof**: Enables future HA Stalwart deployments + +**Timeline**: Can be implemented in ~1 day (see DEV-357 migration plan) + +**Fallback**: If budget/approval is blocked, implement Option 2 (Traefik TCP) as interim solution + +--- + +## Related + +- Migration Plan: `/hetzner-lb-migration-plan.md` +- Implementation Task: DEV-357 +- Historical Investigation: `apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md`