Document stable routing solution for Stalwart Mail
Add comprehensive analysis of current k3s ServiceLB issues and long-term routing solutions to prevent CNI-HOSTPORT orphaned rules. Recommended approach: Migrate to Hetzner Cloud Load Balancer (DEV-357) - Eliminates CNI-HOSTPORT complexity - True external load balancing - ~€8.91/month cost Alternative: Traefik TCP IngressRoute (interim solution) Includes recovery procedures for orphaned iptables rules. Related: DEV-359, DEV-357 Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
a551ee5235
commit
38639f369e
1 changed files with 150 additions and 0 deletions
150
apps/stalwart/STABLE-ROUTING-SOLUTION.md
Normal file
150
apps/stalwart/STABLE-ROUTING-SOLUTION.md
Normal file
|
|
@ -0,0 +1,150 @@
|
||||||
|
# Stalwart Mail - Stable Routing Solution
|
||||||
|
|
||||||
|
**Created**: 2026-07-22
|
||||||
|
**Issue**: DEV-359
|
||||||
|
**Author**: CTO Agent
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
k3s ServiceLB (svclb) with hostPort bindings can leave orphaned CNI-HOSTPORT iptables rules when pods restart, causing intermittent mail service outages.
|
||||||
|
|
||||||
|
### Root Cause
|
||||||
|
|
||||||
|
1. **Architecture**: k3s ServiceLB uses DaemonSet pods with hostPort to expose LoadBalancer services
|
||||||
|
2. **CNI Plugin**: hostPort creates CNI-HOSTPORT iptables DNAT rules
|
||||||
|
3. **Failure Mode**: When svclb pods restart, the CNI plugin may fail to clean up old DNAT rules
|
||||||
|
4. **Impact**: External traffic gets DNATed to old (non-existent) pod IPs, causing connection timeouts
|
||||||
|
|
||||||
|
### Historical Issues
|
||||||
|
|
||||||
|
- **2026-07-20**: Stalwart pod restart left orphaned rules pointing to old svclb pod IPs
|
||||||
|
- **2026-07-22**: Orphaned rules caused complete IMAP/SMTP outage
|
||||||
|
- **Recovery**: Required manual deletion of orphaned rules + svclb pod recreation
|
||||||
|
|
||||||
|
## Recommended Long-Term Solutions
|
||||||
|
|
||||||
|
### Option 1: Migrate to Hetzner Cloud Load Balancer (RECOMMENDED)
|
||||||
|
|
||||||
|
**Architecture**:
|
||||||
|
```
|
||||||
|
External Traffic → Hetzner LB11 (public IP)
|
||||||
|
↓
|
||||||
|
k3s worker nodes (private IPs via TargetGroup)
|
||||||
|
↓
|
||||||
|
Stalwart LoadBalancer service (standard k8s)
|
||||||
|
↓
|
||||||
|
Stalwart pod
|
||||||
|
```
|
||||||
|
|
||||||
|
**Benefits**:
|
||||||
|
- True external load balancing with HA
|
||||||
|
- No CNI-HOSTPORT complexity
|
||||||
|
- Automatic health checks
|
||||||
|
- SSL termination option
|
||||||
|
- Multi-node distribution
|
||||||
|
- No manual iptables management
|
||||||
|
|
||||||
|
**Cost**: ~€8.91/month (LB11 - 5 services)
|
||||||
|
|
||||||
|
**Implementation**: See `/hetzner-lb-migration-plan.md` in workspace root
|
||||||
|
|
||||||
|
**Status**: Plan approved, awaiting implementation (tracked in DEV-357)
|
||||||
|
|
||||||
|
### Option 2: Traefik TCP IngressRoute (ALTERNATIVE)
|
||||||
|
|
||||||
|
**Architecture**:
|
||||||
|
```
|
||||||
|
External Traffic → Traefik (ports 25, 587, 993)
|
||||||
|
↓ (TCP routing via IngressRouteTCP)
|
||||||
|
Stalwart ClusterIP service
|
||||||
|
↓
|
||||||
|
Stalwart pod
|
||||||
|
```
|
||||||
|
|
||||||
|
**Benefits**:
|
||||||
|
- No additional infrastructure
|
||||||
|
- Traefik already deployed
|
||||||
|
- Unified ingress for HTTP + TCP
|
||||||
|
- No CNI-HOSTPORT dependencies
|
||||||
|
|
||||||
|
**Drawbacks**:
|
||||||
|
- Requires Traefik configuration changes
|
||||||
|
- Must expose additional ports on Traefik service
|
||||||
|
- Single point of failure (unless Traefik is HA)
|
||||||
|
|
||||||
|
**Implementation Steps**:
|
||||||
|
1. Add SMTP/IMAP entryPoints to Traefik deployment args
|
||||||
|
2. Add ports 25, 587, 993 to Traefik LoadBalancer service
|
||||||
|
3. Create IngressRouteTCP resources (already exist in cluster)
|
||||||
|
4. Change Stalwart services from LoadBalancer to ClusterIP
|
||||||
|
|
||||||
|
**Status**: Not implemented, IngressRouteTCP resources exist but Traefik entryPoints missing
|
||||||
|
|
||||||
|
### Option 3: MetalLB (NOT RECOMMENDED)
|
||||||
|
|
||||||
|
Installing MetalLB to provide real LoadBalancer IPs.
|
||||||
|
|
||||||
|
**Drawbacks**:
|
||||||
|
- Additional complexity
|
||||||
|
- Potential conflicts with k3s ServiceLB
|
||||||
|
- Still uses node IPs (no true external LB)
|
||||||
|
- Requires IP pool management
|
||||||
|
|
||||||
|
**Status**: Rejected - Hetzner LB or Traefik are better options
|
||||||
|
|
||||||
|
## Current State (2026-07-22)
|
||||||
|
|
||||||
|
**Active Solution**: k3s ServiceLB with hostPort bindings
|
||||||
|
|
||||||
|
**Configuration**:
|
||||||
|
- Service type: LoadBalancer with `externalTrafficPolicy: Local`
|
||||||
|
- svclb DaemonSet pods on all nodes
|
||||||
|
- CNI-HOSTPORT iptables rules for port forwarding
|
||||||
|
- Exposed ports: SMTP (25, 587), IMAPS (993)
|
||||||
|
- Removed insecure ports: SMTPS (465), IMAP (143)
|
||||||
|
|
||||||
|
**Known Issues**:
|
||||||
|
- CNI-HOSTPORT rules can become orphaned on pod restart
|
||||||
|
- Requires manual intervention to clean up orphaned rules
|
||||||
|
- Not truly multi-node load balanced
|
||||||
|
|
||||||
|
**Recovery Procedure** (if orphaned rules occur):
|
||||||
|
```bash
|
||||||
|
# 1. Delete all svclb pods to trigger CNI cleanup
|
||||||
|
kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp'
|
||||||
|
kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-imap'
|
||||||
|
|
||||||
|
# 2. Wait for pods to recreate (CNI will create new hostPort rules)
|
||||||
|
kubectl wait --for=condition=ready pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp' --timeout=60s
|
||||||
|
|
||||||
|
# 3. Verify CNI-HOSTPORT rules on each node
|
||||||
|
for node in k3s-cp-1 k3s-worker-1 k3s-worker-2 k3s-worker-3; do
|
||||||
|
ssh root@$node "iptables-legacy -t nat -L CNI-HOSTPORT-DNAT -n"
|
||||||
|
done
|
||||||
|
|
||||||
|
# 4. Test external connectivity
|
||||||
|
telnet mail.basicstack.de 587
|
||||||
|
openssl s_client -connect mail.basicstack.de:993
|
||||||
|
```
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
**Implement Option 1 (Hetzner Load Balancer)** for the following reasons:
|
||||||
|
|
||||||
|
1. **Stability**: Eliminates CNI-HOSTPORT complexity entirely
|
||||||
|
2. **Scalability**: True external load balancing across multiple nodes
|
||||||
|
3. **Reliability**: Hetzner-managed infrastructure with automatic health checks
|
||||||
|
4. **Cost-effective**: €8.91/month is acceptable for a production mail service
|
||||||
|
5. **Future-proof**: Enables future HA Stalwart deployments
|
||||||
|
|
||||||
|
**Timeline**: Can be implemented in ~1 day (see DEV-357 migration plan)
|
||||||
|
|
||||||
|
**Fallback**: If budget/approval is blocked, implement Option 2 (Traefik TCP) as interim solution
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- Migration Plan: `/hetzner-lb-migration-plan.md`
|
||||||
|
- Implementation Task: DEV-357
|
||||||
|
- Historical Investigation: `apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md`
|
||||||
Loading…
Add table
Reference in a new issue