stack.basicstack.de/apps/stalwart

Stalwart Mail Server v0.16.18

Deployment of Stalwart mail server on a PostgreSQL backend (RocksDB was retired in DEV-467/DEV-473; teardown completed in DEV-474).

Architecture

• Version: v0.16.18
• Authentication: OIDC via Pocket-ID (backed by Postgres accounts store)
• Configuration + state: PostgreSQL (stalwart-postgres StatefulSet in-namespace)
• Storage: Encrypted hcloud-volumes attached to stalwart-postgres-0 (10Gi). Stalwart itself uses an emptyDir at /var/lib/stalwart — no persistent block volume on stalwart-0
• Backup: Daily Postgres logical dump to S3 via stalwart-postgres-backup CronJob (restic tag stalwart-postgres) at 02:30 UTC
• Web UI: https://mail.basicstack.de

Network Architecture

Stalwart uses a two-tier load balancing setup:

1. Kubernetes LoadBalancer Services: Two LoadBalancer services expose SMTP (25, 587) and IMAPS (993) ports (SMTPS/465 removed per DEV-359 approval, plaintext IMAP/143 disabled). The service type LoadBalancer is required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type.

2. Hetzner Load Balancer: Automatically managed by the Hetzner CCM. The load balancer distributes traffic across:

   • k3s-cp-1 (control plane node)
   • k3s-worker-1 through k3s-worker-5 (worker nodes)

Traffic Flow

External Client → Hetzner Load Balancer → NodePort → K8s Service → Stalwart Pod

The Hetzner CCM watches for LoadBalancer-type services and automatically:

• Creates/updates the Hetzner LB configuration
• Configures health checks
• Maps service ports to node ports
• Manages target nodes

Important: Changing the service type from LoadBalancer to NodePort would break the automatic Hetzner LB management. The current setup is the correct configuration for our infrastructure.

Files

• stalwart-fresh-deployment.yaml - Main deployment manifest (StatefulSet, services, ingress)
• stalwart-postgres.yaml - Postgres backend StatefulSet + service (state store)
• stalwart-postgres-backup.yaml - Daily pg_dump → restic CronJob (tag stalwart-postgres)
• stalwart-config.yaml - Stalwart configmap (stalwart.toml)
• stalwart-bootstrap-config.yaml - Bootstrap configmap pointing Stalwart at Postgres
• stalwart-admin-credentials-sealed.yaml - Sealed secret for admin password
• stalwart-postgres-credentials-sealed.yaml - Sealed secret for Postgres role
• stalwart-s3-backup-sealed.yaml - Sealed secret for S3 backup credentials

Deployment

# Apply sealed secrets first
kubectl apply -f stalwart-admin-credentials-sealed.yaml
kubectl apply -f stalwart-postgres-credentials-sealed.yaml
kubectl apply -f stalwart-s3-backup-sealed.yaml

# Configmaps: bootstrap points Stalwart at Postgres; stalwart-config carries stalwart.toml
kubectl apply -f stalwart-bootstrap-config.yaml
kubectl apply -f stalwart-config.yaml

# Postgres backend + backup
kubectl apply -f stalwart-postgres.yaml
kubectl apply -f stalwart-postgres-backup.yaml

# Deploy Stalwart
kubectl apply -f stalwart-fresh-deployment.yaml

Initial Admin Login

After deployment, log in at https://mail.basicstack.de with:

• Username: admin
• Password: (from stalwart-admin-credentials secret)

Configuration

IMPORTANT: Runtime configuration and account/queue state now live in PostgreSQL (stalwart-postgres-0), not RocksDB.

The stalwart-bootstrap-config configmap points Stalwart at the Postgres store; runtime edits made in the web UI or API are persisted to Postgres. The stalwart-config configmap carries the boot-time stalwart.toml.

The historical RocksDB backend (PVC stalwart-data, restic tag stalwart, stalwart-backup CronJob, and the fix-permissions init-container) was retired on 2026-08-28 after 14 healthy post-cutover days (DEV-474). No RocksDB restic chain remains; rollback is now via the Postgres pg_dump restic chain (tag=stalwart-postgres).

Check https://stalw.art/docs/ref/ for configuration possibilities. The API access via /jmap seems to be too complex for the agent and the configruation has been done manually.

Refer to manual configuration steps

Network Architecture

Load Balancing Setup

Stalwart uses a two-tier load balancing architecture:

1. Hetzner Cloud Load Balancer (External Layer)

   • Provides the public-facing IP for mail services
   • Configured with k3s-cp-1 and all worker nodes as targets
   • Forwards traffic to Kubernetes NodePorts on the cluster nodes

2. Kubernetes LoadBalancer Services (Internal Layer)

   • Service type: LoadBalancer (managed by k3s ServiceLB)
   • Automatically configured by Hetzner CCM (Cloud Controller Manager)
   • Creates NodePorts that the Hetzner LB targets

Traffic Flow

External Mail Client
    ↓
Hetzner Load Balancer (public IP)
    ↓
k3s Node NodePort (automatically assigned)
    ↓
Kubernetes LoadBalancer Service (stalwart-smtp / stalwart-imap)
    ↓
Stalwart Pod

Why LoadBalancer Service Type is Required

The Kubernetes services MUST remain type LoadBalancer because:

• The Hetzner CCM automatically manages the Hetzner Load Balancer configuration
• When a service is type LoadBalancer, the CCM creates/updates the Hetzner LB targets
• Changing to NodePort would break this automatic management
• Manual Hetzner LB configuration would be required and error-prone

Services

• stalwart-smtp (LoadBalancer): Ports 25, 587
• stalwart-imap (LoadBalancer): Port 993
• stalwart-http (ClusterIP): Port 8080 (web UI via Traefik Ingress)

Ports

• SMTP: 25, 587
• IMAPS: 993 (secure only, port 143 disabled per DEV-359)
• HTTP: 8080 (web UI)

Storage

All persistent state lives in PostgreSQL on stalwart-postgres-0 (PVC postgres-data-stalwart-postgres-0, 10Gi, hcloud-volumes-encrypted). Stalwart connects to Postgres via the bootstrap config and stores:

• Email messages (blob + metadata)
• User accounts
• Server configuration
• Queue / spool state

/var/lib/stalwart on stalwart-0 is an emptyDir mount kept only as a writable path if Stalwart needs one at runtime; no data of value lives there.

Certificate Renewal

Refer to Automatic Renewal TLS Certificate