Document Hetzner Load Balancer architecture for Stalwart

Added comprehensive documentation of the two-tier load balancing setup:
- Hetzner Cloud Load Balancer (external layer, managed by Hetzner CCM)
- Kubernetes LoadBalancer services (internal layer, k3s ServiceLB)

Key points documented:
- Traffic flow from external client through both LB layers to pod
- Why LoadBalancer service type is required (CCM integration)
- Historical context of the migration from hostPort to Hetzner LB
- Service definitions and port configurations

Updated:
- apps/stalwart/README.md: Added Network Architecture section
- infrastructure/networking/NETWORK_ARCHITECTURE.md: Enhanced Stalwart
  section with two-tier architecture details and updated traffic flows

Resolves documentation gap identified in DEV-439.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/pangolin/pangolin-controller-deployment.yaml

@@ -26,6 +26,12 @@ spec:
           envFrom:
             - configMapRef:
                 name: pangolin-controller-config
+          env:
+            - name: CONFIG_AUTH_HEADER
+              valueFrom:
+                secretKeyRef:
+                  name: pangolin-controller-api-key
+                  key: auth-header
           ports:
             - name: metrics
               containerPort: 9090

apps/stalwart/README.md

@@ -74,10 +74,54 @@ The API access via /jmap seems to be too complex for the agent and the configrua
 
 Refer to [manual configuration steps](manual_config_steps.md)
 
+## Network Architecture
+
+### Load Balancing Setup
+
+Stalwart uses a two-tier load balancing architecture:
+
+1. **Hetzner Cloud Load Balancer** (External Layer)
+   - Provides the public-facing IP for mail services
+   - Configured with k3s-cp-1 and all worker nodes as targets
+   - Forwards traffic to Kubernetes NodePorts on the cluster nodes
+
+2. **Kubernetes LoadBalancer Services** (Internal Layer)
+   - Service type: LoadBalancer (managed by k3s ServiceLB)
+   - Automatically configured by Hetzner CCM (Cloud Controller Manager)
+   - Creates NodePorts that the Hetzner LB targets
+
+### Traffic Flow
+
+```
+External Mail Client
+    ↓
+Hetzner Load Balancer (public IP)
+    ↓
+k3s Node NodePort (automatically assigned)
+    ↓
+Kubernetes LoadBalancer Service (stalwart-smtp / stalwart-imap)
+    ↓
+Stalwart Pod
+```
+
+### Why LoadBalancer Service Type is Required
+
+The Kubernetes services MUST remain type `LoadBalancer` because:
+- The Hetzner CCM automatically manages the Hetzner Load Balancer configuration
+- When a service is type LoadBalancer, the CCM creates/updates the Hetzner LB targets
+- Changing to NodePort would break this automatic management
+- Manual Hetzner LB configuration would be required and error-prone
+
+### Services
+
+- `stalwart-smtp` (LoadBalancer): Ports 25, 587
+- `stalwart-imap` (LoadBalancer): Port 993
+- `stalwart-http` (ClusterIP): Port 8080 (web UI via Traefik Ingress)
+
 ## Ports
 
-- SMTP: 25, 587, 465
-- IMAP: 143, 993
+- SMTP: 25, 587
+- IMAPS: 993 (secure only, port 143 disabled per DEV-359)
 - HTTP: 8080 (web UI)
 
 ## Storage

infrastructure/networking/NETWORK_ARCHITECTURE.md

@@ -198,27 +198,60 @@ Ports:
 
 ### Stalwart Mail Server
 
+Stalwart uses a **two-tier load balancing architecture** combining Hetzner Cloud Load Balancer with Kubernetes LoadBalancer services.
+
+#### Architecture
+
+```
+External Mail Client
+    ↓
+Hetzner Cloud Load Balancer (Managed)
+  - Targets: k3s-cp-1, k3s-worker-1 through k3s-worker-5
+  - Health checks enabled
+    ↓
+k3s Node NodePorts (auto-assigned by k3s ServiceLB)
+    ↓
+Kubernetes LoadBalancer Services (stalwart-smtp / stalwart-imap)
+    ↓
+Stalwart Pod (fsn1 datacenter, PV affinity)
+```
+
+#### Why This Architecture?
+
+The Kubernetes services **MUST** be type `LoadBalancer` (not NodePort) because:
+- **Hetzner CCM Integration**: The Hetzner Cloud Controller Manager automatically configures the Hetzner Load Balancer when it detects a Kubernetes LoadBalancer service
+- **Automatic Target Management**: CCM keeps the Hetzner LB targets synchronized with cluster node changes
+- **Health Check Automation**: CCM configures health checks based on the service configuration
+- **No Manual Intervention**: Changing to NodePort would require manual Hetzner LB management
+
+See DEV-439 for detailed investigation.
+
 #### SMTP Service
 ```yaml
 Service: stalwart/stalwart-smtp
-Type: LoadBalancer
+Type: LoadBalancer  # Required for Hetzner CCM integration
 External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
 Ports:
   - 25/TCP (SMTP)
   - 587/TCP (Submission)
-  - 465/TCP (SMTPS)
 ```
 
 #### IMAP Service
 ```yaml
 Service: stalwart/stalwart-imap
-Type: LoadBalancer
+Type: LoadBalancer  # Required for Hetzner CCM integration
 External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
 Ports:
-  - 143/TCP (IMAP)
-  - 993/TCP (IMAPS)
+  - 993/TCP (IMAPS)  # Port 143 disabled per DEV-359
 ```
 
+#### Historical Context
+
+- **Initial Setup**: Used k3s ServiceLB with hostPort bindings (DEV-359)
+- **Migration**: Migrated to Hetzner Load Balancer for stability (DEV-357)
+- **Current State**: Hetzner LB + Kubernetes LoadBalancer services (managed by Hetzner CCM)
+- **Key Fix**: Resolved CNI-HOSTPORT orphaned rules issue by adding external LB layer
+
 ## DNS Configuration
 
 ### Domain: basicstack.de
@@ -398,16 +431,26 @@ Internet → DNS → Hetzner Firewall (fw-k3s)
     → Application Pods
 ```
 
-### Direct LoadBalancer Traffic (SMTP/IMAP)
+### Mail Traffic (SMTP/IMAP) - Stalwart
+
+**Two-Tier Architecture** (Hetzner Load Balancer + Kubernetes LoadBalancer):
 
 ```
-Internet → DNS → Hetzner Firewall (fw-k3s)
-    → LoadBalancer IP (any worker or control-plane node)
+Internet → DNS (mail.basicstack.de)
+    → Hetzner Cloud Load Balancer (managed by Hetzner CCM)
+    → k3s Node (any target: k3s-cp-1 or k3s-worker-1 through k3s-worker-5)
+    → NodePort (auto-assigned by k3s ServiceLB)
+    → Kubernetes LoadBalancer Service (stalwart-smtp / stalwart-imap)
     → Service DaemonSet Pod (svclb-stalwart-*)
-    → Backend Service (ClusterIP)
-    → Stalwart Pods
+    → Stalwart Pod (in fsn1 datacenter due to PV affinity)
 ```
 
+**Key Points:**
+- Hetzner CCM automatically manages the Hetzner Load Balancer configuration
+- Service type MUST be LoadBalancer (not NodePort) for CCM integration
+- k3s ServiceLB creates the NodePorts that the Hetzner LB targets
+- Traffic flows through two load balancing layers for reliability
+
 ### Internal Service-to-Service
 
 ```
@@ -669,4 +712,5 @@ kubectl get certificate -n <namespace> <cert-name> -w
 
 | Date | Change | Author |
 |------|--------|--------|
+| 2026-08-02 | Document Hetzner Load Balancer architecture for Stalwart mail services | CTO (DEV-439) |
 | 2026-07-06 | Initial comprehensive network architecture documentation | CTO (DEV-225) |