Add comprehensive Network Architecture section explaining: - Two-tier load balancing (K8s LoadBalancer + Hetzner LB) - Why LoadBalancer service type is required (Hetzner CCM auto-management) - Traffic flow from external clients to Stalwart pods - Warning against changing to NodePort (breaks CCM integration) This documents the investigation from DEV-439, confirming that the current LoadBalancer service setup is correct and necessary for our infrastructure. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|---|---|---|
| .. | ||
| CERTIFICATE-RENEWAL.md | ||
| CONFIGMAP-HARDENING-TEST-2026-08-01.md | ||
| ISSUE-2026-07-13-smtp-imap-external-access.md | ||
| manual_config_steps.md | ||
| README.md | ||
| SETUP_COMPLETE.md | ||
| STABILITY-CHECK.md | ||
| STABLE-ROUTING-SOLUTION.md | ||
| stalwart-admin-credentials-sealed.yaml | ||
| stalwart-allow-cluster-ips-job.yaml | ||
| stalwart-bootstrap-config.yaml | ||
| stalwart-config.yaml | ||
| stalwart-fresh-deployment.yaml | ||
| stalwart-monitoring.yaml | ||
| stalwart-s3-backup-sealed.yaml | ||
Stalwart Mail Server v0.16.11
Clean deployment of Stalwart mail server with username/password authentication only.
Architecture
- Version: v0.16.11
- Authentication: Username/password only (NO OAuth/OIDC)
- Configuration: API-based (stored in RocksDB)
- Storage: Encrypted hcloud-volumes (20Gi)
- Backup: Daily restic backup to S3 at 3 AM
- Web UI: https://mail.basicstack.de
Network Architecture
Stalwart uses a two-tier load balancing setup:
-
Kubernetes LoadBalancer Services: Four separate LoadBalancer services expose SMTP (25, 587, 465) and IMAP (143, 993) ports. The service type
LoadBalanceris required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type. -
Hetzner Load Balancer: Automatically managed by the Hetzner CCM. The load balancer distributes traffic across:
- k3s-cp-1 (control plane node)
- k3s-worker-1 through k3s-worker-5 (worker nodes)
Traffic Flow
External Client → Hetzner Load Balancer → NodePort → K8s Service → Stalwart Pod
The Hetzner CCM watches for LoadBalancer-type services and automatically:
- Creates/updates the Hetzner LB configuration
- Configures health checks
- Maps service ports to node ports
- Manages target nodes
Important: Changing the service type from LoadBalancer to NodePort would break the automatic Hetzner LB management. The current setup is the correct configuration for our infrastructure.
Files
stalwart-fresh-deployment.yaml- Main deployment manifeststalwart-admin-credentials-sealed.yaml- Sealed secret for admin passwordstalwart-s3-backup-sealed.yaml- Sealed secret for S3 backup credentials
Deployment
# Apply sealed secrets first
kubectl apply -f stalwart-admin-credentials-sealed.yaml
kubectl apply -f stalwart-s3-backup-sealed.yaml
# Create bootstrap config
kubectl create configmap stalwart-bootstrap-config \
--from-literal=config.json='{"@type":"RocksDb","path":"/var/lib/stalwart"}' \
-n stalwart
# Deploy Stalwart
kubectl apply -f stalwart-fresh-deployment.yaml
Initial Admin Login
After deployment, log in at https://mail.basicstack.de with:
- Username:
admin - Password: (from stalwart-admin-credentials secret)
Configuration
IMPORTANT: The configuration is stored in the RocksDB and cannot be overwritten by a configuration file!
All configuration is done via the web UI or API. The bootstrap config only points to the RocksDB database location. NO config.toml files are used.
Check https://stalw.art/docs/ref/ for configuration possibilities. The API access via /jmap seems to be too complex for the agent and the configruation has been done manually.
Refer to manual configuration steps
Ports
- SMTP: 25, 587, 465
- IMAP: 143, 993
- HTTP: 8080 (web UI)
Storage
Data is stored in /var/lib/stalwart using the RocksDB database format. This includes:
- Email messages
- User accounts
- Server configuration
- TLS certificates configuration
Certificate Renewal
Refer to Automatic Renewal TLS Certificate