From fd0b589c104e2f1f28f4464922443006ad31a59f Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 2 Aug 2026 11:16:01 +0000 Subject: [PATCH] Document Hetzner Load Balancer architecture for Stalwart Added comprehensive documentation of the two-tier load balancing setup: - Hetzner Cloud Load Balancer (external layer, managed by Hetzner CCM) - Kubernetes LoadBalancer services (internal layer, k3s ServiceLB) Key points documented: - Traffic flow from external client through both LB layers to pod - Why LoadBalancer service type is required (CCM integration) - Historical context of the migration from hostPort to Hetzner LB - Service definitions and port configurations Updated: - apps/stalwart/README.md: Added Network Architecture section - infrastructure/networking/NETWORK_ARCHITECTURE.md: Enhanced Stalwart section with two-tier architecture details and updated traffic flows Resolves documentation gap identified in DEV-439. Co-Authored-By: Paperclip --- apps/stalwart/README.md | 48 +++++++++++++- .../networking/NETWORK_ARCHITECTURE.md | 64 ++++++++++++++++--- 2 files changed, 100 insertions(+), 12 deletions(-) diff --git a/apps/stalwart/README.md b/apps/stalwart/README.md index 65fa1aa..a8e5290 100644 --- a/apps/stalwart/README.md +++ b/apps/stalwart/README.md @@ -74,10 +74,54 @@ The API access via /jmap seems to be too complex for the agent and the configrua Refer to [manual configuration steps](manual_config_steps.md) +## Network Architecture + +### Load Balancing Setup + +Stalwart uses a two-tier load balancing architecture: + +1. **Hetzner Cloud Load Balancer** (External Layer) + - Provides the public-facing IP for mail services + - Configured with k3s-cp-1 and all worker nodes as targets + - Forwards traffic to Kubernetes NodePorts on the cluster nodes + +2. **Kubernetes LoadBalancer Services** (Internal Layer) + - Service type: LoadBalancer (managed by k3s ServiceLB) + - Automatically configured by Hetzner CCM (Cloud Controller Manager) + - Creates NodePorts that the Hetzner LB targets + +### Traffic Flow + +``` +External Mail Client + ↓ +Hetzner Load Balancer (public IP) + ↓ +k3s Node NodePort (automatically assigned) + ↓ +Kubernetes LoadBalancer Service (stalwart-smtp / stalwart-imap) + ↓ +Stalwart Pod +``` + +### Why LoadBalancer Service Type is Required + +The Kubernetes services MUST remain type `LoadBalancer` because: +- The Hetzner CCM automatically manages the Hetzner Load Balancer configuration +- When a service is type LoadBalancer, the CCM creates/updates the Hetzner LB targets +- Changing to NodePort would break this automatic management +- Manual Hetzner LB configuration would be required and error-prone + +### Services + +- `stalwart-smtp` (LoadBalancer): Ports 25, 587 +- `stalwart-imap` (LoadBalancer): Port 993 +- `stalwart-http` (ClusterIP): Port 8080 (web UI via Traefik Ingress) + ## Ports -- SMTP: 25, 587, 465 -- IMAP: 143, 993 +- SMTP: 25, 587 +- IMAPS: 993 (secure only, port 143 disabled per DEV-359) - HTTP: 8080 (web UI) ## Storage diff --git a/infrastructure/networking/NETWORK_ARCHITECTURE.md b/infrastructure/networking/NETWORK_ARCHITECTURE.md index f85df2f..43c3b2e 100644 --- a/infrastructure/networking/NETWORK_ARCHITECTURE.md +++ b/infrastructure/networking/NETWORK_ARCHITECTURE.md @@ -198,27 +198,60 @@ Ports: ### Stalwart Mail Server +Stalwart uses a **two-tier load balancing architecture** combining Hetzner Cloud Load Balancer with Kubernetes LoadBalancer services. + +#### Architecture + +``` +External Mail Client + ↓ +Hetzner Cloud Load Balancer (Managed) + - Targets: k3s-cp-1, k3s-worker-1 through k3s-worker-5 + - Health checks enabled + ↓ +k3s Node NodePorts (auto-assigned by k3s ServiceLB) + ↓ +Kubernetes LoadBalancer Services (stalwart-smtp / stalwart-imap) + ↓ +Stalwart Pod (fsn1 datacenter, PV affinity) +``` + +#### Why This Architecture? + +The Kubernetes services **MUST** be type `LoadBalancer` (not NodePort) because: +- **Hetzner CCM Integration**: The Hetzner Cloud Controller Manager automatically configures the Hetzner Load Balancer when it detects a Kubernetes LoadBalancer service +- **Automatic Target Management**: CCM keeps the Hetzner LB targets synchronized with cluster node changes +- **Health Check Automation**: CCM configures health checks based on the service configuration +- **No Manual Intervention**: Changing to NodePort would require manual Hetzner LB management + +See DEV-439 for detailed investigation. + #### SMTP Service ```yaml Service: stalwart/stalwart-smtp -Type: LoadBalancer +Type: LoadBalancer # Required for Hetzner CCM integration External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239 Ports: - 25/TCP (SMTP) - 587/TCP (Submission) - - 465/TCP (SMTPS) ``` #### IMAP Service ```yaml Service: stalwart/stalwart-imap -Type: LoadBalancer +Type: LoadBalancer # Required for Hetzner CCM integration External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239 Ports: - - 143/TCP (IMAP) - - 993/TCP (IMAPS) + - 993/TCP (IMAPS) # Port 143 disabled per DEV-359 ``` +#### Historical Context + +- **Initial Setup**: Used k3s ServiceLB with hostPort bindings (DEV-359) +- **Migration**: Migrated to Hetzner Load Balancer for stability (DEV-357) +- **Current State**: Hetzner LB + Kubernetes LoadBalancer services (managed by Hetzner CCM) +- **Key Fix**: Resolved CNI-HOSTPORT orphaned rules issue by adding external LB layer + ## DNS Configuration ### Domain: basicstack.de @@ -398,16 +431,26 @@ Internet → DNS → Hetzner Firewall (fw-k3s) → Application Pods ``` -### Direct LoadBalancer Traffic (SMTP/IMAP) +### Mail Traffic (SMTP/IMAP) - Stalwart + +**Two-Tier Architecture** (Hetzner Load Balancer + Kubernetes LoadBalancer): ``` -Internet → DNS → Hetzner Firewall (fw-k3s) - → LoadBalancer IP (any worker or control-plane node) +Internet → DNS (mail.basicstack.de) + → Hetzner Cloud Load Balancer (managed by Hetzner CCM) + → k3s Node (any target: k3s-cp-1 or k3s-worker-1 through k3s-worker-5) + → NodePort (auto-assigned by k3s ServiceLB) + → Kubernetes LoadBalancer Service (stalwart-smtp / stalwart-imap) → Service DaemonSet Pod (svclb-stalwart-*) - → Backend Service (ClusterIP) - → Stalwart Pods + → Stalwart Pod (in fsn1 datacenter due to PV affinity) ``` +**Key Points:** +- Hetzner CCM automatically manages the Hetzner Load Balancer configuration +- Service type MUST be LoadBalancer (not NodePort) for CCM integration +- k3s ServiceLB creates the NodePorts that the Hetzner LB targets +- Traffic flows through two load balancing layers for reliability + ### Internal Service-to-Service ``` @@ -669,4 +712,5 @@ kubectl get certificate -n -w | Date | Change | Author | |------|--------|--------| +| 2026-08-02 | Document Hetzner Load Balancer architecture for Stalwart mail services | CTO (DEV-439) | | 2026-07-06 | Initial comprehensive network architecture documentation | CTO (DEV-225) |