stack.basicstack.de/apps/stalwart/README.md

# Stalwart Mail Server v0.16.18

Deployment of Stalwart mail server on a PostgreSQL backend (RocksDB was retired in DEV-467/DEV-473; teardown completed in DEV-474).

## Architecture

- **Version**: v0.16.18
- **Authentication**: OIDC via Pocket-ID (backed by Postgres accounts store)
- **Configuration + state**: PostgreSQL (`stalwart-postgres` StatefulSet in-namespace)
- **Storage**: Encrypted hcloud-volumes attached to `stalwart-postgres-0` (10Gi). Stalwart itself uses an emptyDir at `/var/lib/stalwart` — no persistent block volume on `stalwart-0`
- **Backup**: Daily Postgres logical dump to S3 via `stalwart-postgres-backup` CronJob (restic tag `stalwart-postgres`) at 02:30 UTC
- **Web UI**: https://mail.basicstack.de

## Network Architecture

Stalwart uses a two-tier load balancing setup:

1. **Kubernetes LoadBalancer Services**: Two LoadBalancer services expose SMTP (25, 587) and IMAPS (993) ports (SMTPS/465 removed per DEV-359 approval, plaintext IMAP/143 disabled). The service type `LoadBalancer` is required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type.

2. **Hetzner Load Balancer**: Automatically managed by the Hetzner CCM. The load balancer distributes traffic across:
   - k3s-cp-1 (control plane node)
   - k3s-worker-1 through k3s-worker-5 (worker nodes)

### Traffic Flow

```
External Client → Hetzner Load Balancer → NodePort → K8s Service → Stalwart Pod
```

The Hetzner CCM watches for LoadBalancer-type services and automatically:
- Creates/updates the Hetzner LB configuration
- Configures health checks
- Maps service ports to node ports
- Manages target nodes

**Important**: Changing the service type from `LoadBalancer` to `NodePort` would break the automatic Hetzner LB management. The current setup is the correct configuration for our infrastructure.

## Files

- `stalwart-fresh-deployment.yaml` - Main deployment manifest (StatefulSet, services, ingress)
- `stalwart-postgres.yaml` - Postgres backend StatefulSet + service (state store)
- `stalwart-postgres-backup.yaml` - Daily `pg_dump` → restic CronJob (tag `stalwart-postgres`)
- `stalwart-config.yaml` - Stalwart configmap (`stalwart.toml`)
- `stalwart-bootstrap-config.yaml` - Bootstrap configmap pointing Stalwart at Postgres
- `stalwart-admin-credentials-sealed.yaml` - Sealed secret for admin password
- `stalwart-postgres-credentials-sealed.yaml` - Sealed secret for Postgres role
- `stalwart-s3-backup-sealed.yaml` - Sealed secret for S3 backup credentials

## Deployment

```bash
# Apply sealed secrets first
kubectl apply -f stalwart-admin-credentials-sealed.yaml
kubectl apply -f stalwart-postgres-credentials-sealed.yaml
kubectl apply -f stalwart-s3-backup-sealed.yaml

# Configmaps: bootstrap points Stalwart at Postgres; stalwart-config carries stalwart.toml
kubectl apply -f stalwart-bootstrap-config.yaml
kubectl apply -f stalwart-config.yaml

# Postgres backend + backup
kubectl apply -f stalwart-postgres.yaml
kubectl apply -f stalwart-postgres-backup.yaml

# Deploy Stalwart
kubectl apply -f stalwart-fresh-deployment.yaml
```

## Initial Admin Login

After deployment, log in at https://mail.basicstack.de with:
- Username: `admin`
- Password: (from stalwart-admin-credentials secret)

## Configuration

> **IMPORTANT:** Runtime configuration and account/queue state now live in **PostgreSQL** (`stalwart-postgres-0`), not RocksDB.

The `stalwart-bootstrap-config` configmap points Stalwart at the Postgres store; runtime edits made in the web UI or API are persisted to Postgres. The `stalwart-config` configmap carries the boot-time `stalwart.toml`.

The historical RocksDB backend (PVC `stalwart-data`, restic tag `stalwart`, `stalwart-backup` CronJob, and the `fix-permissions` init-container) was retired on 2026-08-28 after 14 healthy post-cutover days (DEV-474). No RocksDB restic chain remains; rollback is now via the Postgres `pg_dump` restic chain (`tag=stalwart-postgres`).

Check https://stalw.art/docs/ref/ for configuration possibilities.
The API access via /jmap seems to be too complex for the agent and the configruation has been done manually.

Refer to [manual configuration steps](manual_config_steps.md)

## Network Architecture

### Load Balancing Setup

Stalwart uses a two-tier load balancing architecture:

1. **Hetzner Cloud Load Balancer** (External Layer)
   - Provides the public-facing IP for mail services
   - Configured with k3s-cp-1 and all worker nodes as targets
   - Forwards traffic to Kubernetes NodePorts on the cluster nodes

2. **Kubernetes LoadBalancer Services** (Internal Layer)
   - Service type: LoadBalancer (managed by k3s ServiceLB)
   - Automatically configured by Hetzner CCM (Cloud Controller Manager)
   - Creates NodePorts that the Hetzner LB targets

### Traffic Flow

```
External Mail Client
    ↓
Hetzner Load Balancer (public IP)
    ↓
k3s Node NodePort (automatically assigned)
    ↓
Kubernetes LoadBalancer Service (stalwart-smtp / stalwart-imap)
    ↓
Stalwart Pod
```

### Why LoadBalancer Service Type is Required

The Kubernetes services MUST remain type `LoadBalancer` because:
- The Hetzner CCM automatically manages the Hetzner Load Balancer configuration
- When a service is type LoadBalancer, the CCM creates/updates the Hetzner LB targets
- Changing to NodePort would break this automatic management
- Manual Hetzner LB configuration would be required and error-prone

### Services

- `stalwart-smtp` (LoadBalancer): Ports 25, 587
- `stalwart-imap` (LoadBalancer): Port 993
- `stalwart-http` (ClusterIP): Port 8080 (web UI via Traefik Ingress)

## Ports

- SMTP: 25, 587
- IMAPS: 993 (secure only, port 143 disabled per DEV-359)
- HTTP: 8080 (web UI)

## Storage

All persistent state lives in **PostgreSQL** on `stalwart-postgres-0` (PVC `postgres-data-stalwart-postgres-0`, 10Gi, `hcloud-volumes-encrypted`). Stalwart connects to Postgres via the bootstrap config and stores:
- Email messages (blob + metadata)
- User accounts
- Server configuration
- Queue / spool state

`/var/lib/stalwart` on `stalwart-0` is an emptyDir mount kept only as a writable path if Stalwart needs one at runtime; no data of value lives there.

## Certificate Renewal

Refer to [Automatic Renewal TLS Certificate](CERTIFICATE-RENEWAL.md)