stack.basicstack.de/apps/stalwart/README.md

Stalwart Mail Server v0.16.11

Clean deployment of Stalwart mail server with username/password authentication only.

Architecture

• Version: v0.16.11
• Authentication: Username/password only (NO OAuth/OIDC)
• Configuration: API-based (stored in RocksDB)
• Storage: Encrypted hcloud-volumes (20Gi)
• Backup: Daily restic backup to S3 at 3 AM
• Web UI: https://mail.basicstack.de

Network Architecture

Stalwart uses a two-tier load balancing setup:

1. Kubernetes LoadBalancer Services: Four separate LoadBalancer services expose SMTP (25, 587, 465) and IMAP (143, 993) ports. The service type LoadBalancer is required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type.

2. Hetzner Load Balancer: Automatically managed by the Hetzner CCM. The load balancer distributes traffic across:

   • k3s-cp-1 (control plane node)
   • k3s-worker-1 through k3s-worker-5 (worker nodes)

Traffic Flow

External Client → Hetzner Load Balancer → NodePort → K8s Service → Stalwart Pod

The Hetzner CCM watches for LoadBalancer-type services and automatically:

• Creates/updates the Hetzner LB configuration
• Configures health checks
• Maps service ports to node ports
• Manages target nodes

Important: Changing the service type from LoadBalancer to NodePort would break the automatic Hetzner LB management. The current setup is the correct configuration for our infrastructure.

Files

• stalwart-fresh-deployment.yaml - Main deployment manifest
• stalwart-admin-credentials-sealed.yaml - Sealed secret for admin password
• stalwart-s3-backup-sealed.yaml - Sealed secret for S3 backup credentials

Deployment

# Apply sealed secrets first
kubectl apply -f stalwart-admin-credentials-sealed.yaml
kubectl apply -f stalwart-s3-backup-sealed.yaml

# Create bootstrap config
kubectl create configmap stalwart-bootstrap-config \
  --from-literal=config.json='{"@type":"RocksDb","path":"/var/lib/stalwart"}' \
  -n stalwart

# Deploy Stalwart
kubectl apply -f stalwart-fresh-deployment.yaml

Initial Admin Login

After deployment, log in at https://mail.basicstack.de with:

• Username: admin
• Password: (from stalwart-admin-credentials secret)

Configuration

IMPORTANT: The configuration is stored in the RocksDB and cannot be overwritten by a configuration file!

All configuration is done via the web UI or API. The bootstrap config only points to the RocksDB database location. NO config.toml files are used.

Check https://stalw.art/docs/ref/ for configuration possibilities. The API access via /jmap seems to be too complex for the agent and the configruation has been done manually.

Refer to manual configuration steps

Network Architecture

Load Balancing Setup

Stalwart uses a two-tier load balancing architecture:

1. Hetzner Cloud Load Balancer (External Layer)

   • Provides the public-facing IP for mail services
   • Configured with k3s-cp-1 and all worker nodes as targets
   • Forwards traffic to Kubernetes NodePorts on the cluster nodes

2. Kubernetes LoadBalancer Services (Internal Layer)

   • Service type: LoadBalancer (managed by k3s ServiceLB)
   • Automatically configured by Hetzner CCM (Cloud Controller Manager)
   • Creates NodePorts that the Hetzner LB targets

Traffic Flow

External Mail Client
    ↓
Hetzner Load Balancer (public IP)
    ↓
k3s Node NodePort (automatically assigned)
    ↓
Kubernetes LoadBalancer Service (stalwart-smtp / stalwart-imap)
    ↓
Stalwart Pod

Why LoadBalancer Service Type is Required

The Kubernetes services MUST remain type LoadBalancer because:

• The Hetzner CCM automatically manages the Hetzner Load Balancer configuration
• When a service is type LoadBalancer, the CCM creates/updates the Hetzner LB targets
• Changing to NodePort would break this automatic management
• Manual Hetzner LB configuration would be required and error-prone

Services

• stalwart-smtp (LoadBalancer): Ports 25, 587
• stalwart-imap (LoadBalancer): Port 993
• stalwart-http (ClusterIP): Port 8080 (web UI via Traefik Ingress)

Ports

• SMTP: 25, 587
• IMAPS: 993 (secure only, port 143 disabled per DEV-359)
• HTTP: 8080 (web UI)

Storage

Data is stored in /var/lib/stalwart using the RocksDB database format. This includes:

• Email messages
• User accounts
• Server configuration
• TLS certificates configuration

Certificate Renewal

Refer to Automatic Renewal TLS Certificate