Document stable routing solution for Stalwart Mail
Add comprehensive analysis of current k3s ServiceLB issues and long-term routing solutions to prevent CNI-HOSTPORT orphaned rules. Recommended approach: Migrate to Hetzner Cloud Load Balancer (DEV-357) - Eliminates CNI-HOSTPORT complexity - True external load balancing - ~€8.91/month cost Alternative: Traefik TCP IngressRoute (interim solution) Includes recovery procedures for orphaned iptables rules. Related: DEV-359, DEV-357 Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
a551ee5235
commit
38639f369e
1 changed files with 150 additions and 0 deletions
150
apps/stalwart/STABLE-ROUTING-SOLUTION.md
Normal file
150
apps/stalwart/STABLE-ROUTING-SOLUTION.md
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
# Stalwart Mail - Stable Routing Solution
|
||||
|
||||
**Created**: 2026-07-22
|
||||
**Issue**: DEV-359
|
||||
**Author**: CTO Agent
|
||||
|
||||
## Problem Statement
|
||||
|
||||
k3s ServiceLB (svclb) with hostPort bindings can leave orphaned CNI-HOSTPORT iptables rules when pods restart, causing intermittent mail service outages.
|
||||
|
||||
### Root Cause
|
||||
|
||||
1. **Architecture**: k3s ServiceLB uses DaemonSet pods with hostPort to expose LoadBalancer services
|
||||
2. **CNI Plugin**: hostPort creates CNI-HOSTPORT iptables DNAT rules
|
||||
3. **Failure Mode**: When svclb pods restart, the CNI plugin may fail to clean up old DNAT rules
|
||||
4. **Impact**: External traffic gets DNATed to old (non-existent) pod IPs, causing connection timeouts
|
||||
|
||||
### Historical Issues
|
||||
|
||||
- **2026-07-20**: Stalwart pod restart left orphaned rules pointing to old svclb pod IPs
|
||||
- **2026-07-22**: Orphaned rules caused complete IMAP/SMTP outage
|
||||
- **Recovery**: Required manual deletion of orphaned rules + svclb pod recreation
|
||||
|
||||
## Recommended Long-Term Solutions
|
||||
|
||||
### Option 1: Migrate to Hetzner Cloud Load Balancer (RECOMMENDED)
|
||||
|
||||
**Architecture**:
|
||||
```
|
||||
External Traffic → Hetzner LB11 (public IP)
|
||||
↓
|
||||
k3s worker nodes (private IPs via TargetGroup)
|
||||
↓
|
||||
Stalwart LoadBalancer service (standard k8s)
|
||||
↓
|
||||
Stalwart pod
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- True external load balancing with HA
|
||||
- No CNI-HOSTPORT complexity
|
||||
- Automatic health checks
|
||||
- SSL termination option
|
||||
- Multi-node distribution
|
||||
- No manual iptables management
|
||||
|
||||
**Cost**: ~€8.91/month (LB11 - 5 services)
|
||||
|
||||
**Implementation**: See `/hetzner-lb-migration-plan.md` in workspace root
|
||||
|
||||
**Status**: Plan approved, awaiting implementation (tracked in DEV-357)
|
||||
|
||||
### Option 2: Traefik TCP IngressRoute (ALTERNATIVE)
|
||||
|
||||
**Architecture**:
|
||||
```
|
||||
External Traffic → Traefik (ports 25, 587, 993)
|
||||
↓ (TCP routing via IngressRouteTCP)
|
||||
Stalwart ClusterIP service
|
||||
↓
|
||||
Stalwart pod
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- No additional infrastructure
|
||||
- Traefik already deployed
|
||||
- Unified ingress for HTTP + TCP
|
||||
- No CNI-HOSTPORT dependencies
|
||||
|
||||
**Drawbacks**:
|
||||
- Requires Traefik configuration changes
|
||||
- Must expose additional ports on Traefik service
|
||||
- Single point of failure (unless Traefik is HA)
|
||||
|
||||
**Implementation Steps**:
|
||||
1. Add SMTP/IMAP entryPoints to Traefik deployment args
|
||||
2. Add ports 25, 587, 993 to Traefik LoadBalancer service
|
||||
3. Create IngressRouteTCP resources (already exist in cluster)
|
||||
4. Change Stalwart services from LoadBalancer to ClusterIP
|
||||
|
||||
**Status**: Not implemented, IngressRouteTCP resources exist but Traefik entryPoints missing
|
||||
|
||||
### Option 3: MetalLB (NOT RECOMMENDED)
|
||||
|
||||
Installing MetalLB to provide real LoadBalancer IPs.
|
||||
|
||||
**Drawbacks**:
|
||||
- Additional complexity
|
||||
- Potential conflicts with k3s ServiceLB
|
||||
- Still uses node IPs (no true external LB)
|
||||
- Requires IP pool management
|
||||
|
||||
**Status**: Rejected - Hetzner LB or Traefik are better options
|
||||
|
||||
## Current State (2026-07-22)
|
||||
|
||||
**Active Solution**: k3s ServiceLB with hostPort bindings
|
||||
|
||||
**Configuration**:
|
||||
- Service type: LoadBalancer with `externalTrafficPolicy: Local`
|
||||
- svclb DaemonSet pods on all nodes
|
||||
- CNI-HOSTPORT iptables rules for port forwarding
|
||||
- Exposed ports: SMTP (25, 587), IMAPS (993)
|
||||
- Removed insecure ports: SMTPS (465), IMAP (143)
|
||||
|
||||
**Known Issues**:
|
||||
- CNI-HOSTPORT rules can become orphaned on pod restart
|
||||
- Requires manual intervention to clean up orphaned rules
|
||||
- Not truly multi-node load balanced
|
||||
|
||||
**Recovery Procedure** (if orphaned rules occur):
|
||||
```bash
|
||||
# 1. Delete all svclb pods to trigger CNI cleanup
|
||||
kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp'
|
||||
kubectl delete pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-imap'
|
||||
|
||||
# 2. Wait for pods to recreate (CNI will create new hostPort rules)
|
||||
kubectl wait --for=condition=ready pods -n kube-system -l 'svccontroller.k3s.cattle.io/svcname=stalwart-smtp' --timeout=60s
|
||||
|
||||
# 3. Verify CNI-HOSTPORT rules on each node
|
||||
for node in k3s-cp-1 k3s-worker-1 k3s-worker-2 k3s-worker-3; do
|
||||
ssh root@$node "iptables-legacy -t nat -L CNI-HOSTPORT-DNAT -n"
|
||||
done
|
||||
|
||||
# 4. Test external connectivity
|
||||
telnet mail.basicstack.de 587
|
||||
openssl s_client -connect mail.basicstack.de:993
|
||||
```
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Implement Option 1 (Hetzner Load Balancer)** for the following reasons:
|
||||
|
||||
1. **Stability**: Eliminates CNI-HOSTPORT complexity entirely
|
||||
2. **Scalability**: True external load balancing across multiple nodes
|
||||
3. **Reliability**: Hetzner-managed infrastructure with automatic health checks
|
||||
4. **Cost-effective**: €8.91/month is acceptable for a production mail service
|
||||
5. **Future-proof**: Enables future HA Stalwart deployments
|
||||
|
||||
**Timeline**: Can be implemented in ~1 day (see DEV-357 migration plan)
|
||||
|
||||
**Fallback**: If budget/approval is blocked, implement Option 2 (Traefik TCP) as interim solution
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Migration Plan: `/hetzner-lb-migration-plan.md`
|
||||
- Implementation Task: DEV-357
|
||||
- Historical Investigation: `apps/stalwart/ISSUE-2026-07-13-smtp-imap-external-access.md`
|
||||
Loading…
Add table
Reference in a new issue