diff --git a/apps/stalwart/README.md b/apps/stalwart/README.md index a8e5290..a1009cc 100644 --- a/apps/stalwart/README.md +++ b/apps/stalwart/README.md @@ -1,21 +1,21 @@ -# Stalwart Mail Server v0.16.11 +# Stalwart Mail Server v0.16.18 -Clean deployment of Stalwart mail server with username/password authentication only. +Deployment of Stalwart mail server on a PostgreSQL backend (RocksDB was retired in DEV-467/DEV-473; teardown completed in DEV-474). ## Architecture -- **Version**: v0.16.11 -- **Authentication**: Username/password only (NO OAuth/OIDC) -- **Configuration**: API-based (stored in RocksDB) -- **Storage**: Encrypted hcloud-volumes (20Gi) -- **Backup**: Daily restic backup to S3 at 3 AM +- **Version**: v0.16.18 +- **Authentication**: OIDC via Pocket-ID (backed by Postgres accounts store) +- **Configuration + state**: PostgreSQL (`stalwart-postgres` StatefulSet in-namespace) +- **Storage**: Encrypted hcloud-volumes attached to `stalwart-postgres-0` (10Gi). Stalwart itself uses an emptyDir at `/var/lib/stalwart` — no persistent block volume on `stalwart-0` +- **Backup**: Daily Postgres logical dump to S3 via `stalwart-postgres-backup` CronJob (restic tag `stalwart-postgres`) at 02:30 UTC - **Web UI**: https://mail.basicstack.de ## Network Architecture Stalwart uses a two-tier load balancing setup: -1. **Kubernetes LoadBalancer Services**: Four separate LoadBalancer services expose SMTP (25, 587, 465) and IMAP (143, 993) ports. The service type `LoadBalancer` is required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type. +1. **Kubernetes LoadBalancer Services**: Two LoadBalancer services expose SMTP (25, 587) and IMAPS (993) ports (SMTPS/465 removed per DEV-359 approval, plaintext IMAP/143 disabled). The service type `LoadBalancer` is required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type. 2. **Hetzner Load Balancer**: Automatically managed by the Hetzner CCM. The load balancer distributes traffic across: - k3s-cp-1 (control plane node) @@ -37,8 +37,13 @@ The Hetzner CCM watches for LoadBalancer-type services and automatically: ## Files -- `stalwart-fresh-deployment.yaml` - Main deployment manifest +- `stalwart-fresh-deployment.yaml` - Main deployment manifest (StatefulSet, services, ingress) +- `stalwart-postgres.yaml` - Postgres backend StatefulSet + service (state store) +- `stalwart-postgres-backup.yaml` - Daily `pg_dump` → restic CronJob (tag `stalwart-postgres`) +- `stalwart-config.yaml` - Stalwart configmap (`stalwart.toml`) +- `stalwart-bootstrap-config.yaml` - Bootstrap configmap pointing Stalwart at Postgres - `stalwart-admin-credentials-sealed.yaml` - Sealed secret for admin password +- `stalwart-postgres-credentials-sealed.yaml` - Sealed secret for Postgres role - `stalwart-s3-backup-sealed.yaml` - Sealed secret for S3 backup credentials ## Deployment @@ -46,12 +51,16 @@ The Hetzner CCM watches for LoadBalancer-type services and automatically: ```bash # Apply sealed secrets first kubectl apply -f stalwart-admin-credentials-sealed.yaml +kubectl apply -f stalwart-postgres-credentials-sealed.yaml kubectl apply -f stalwart-s3-backup-sealed.yaml -# Create bootstrap config -kubectl create configmap stalwart-bootstrap-config \ - --from-literal=config.json='{"@type":"RocksDb","path":"/var/lib/stalwart"}' \ - -n stalwart +# Configmaps: bootstrap points Stalwart at Postgres; stalwart-config carries stalwart.toml +kubectl apply -f stalwart-bootstrap-config.yaml +kubectl apply -f stalwart-config.yaml + +# Postgres backend + backup +kubectl apply -f stalwart-postgres.yaml +kubectl apply -f stalwart-postgres-backup.yaml # Deploy Stalwart kubectl apply -f stalwart-fresh-deployment.yaml @@ -65,9 +74,11 @@ After deployment, log in at https://mail.basicstack.de with: ## Configuration -> **IMPORTANT:** The configuration is stored in the RocksDB and cannot be overwritten by a configuration file! +> **IMPORTANT:** Runtime configuration and account/queue state now live in **PostgreSQL** (`stalwart-postgres-0`), not RocksDB. -All configuration is done via the web UI or API. The bootstrap config only points to the RocksDB database location. NO config.toml files are used. +The `stalwart-bootstrap-config` configmap points Stalwart at the Postgres store; runtime edits made in the web UI or API are persisted to Postgres. The `stalwart-config` configmap carries the boot-time `stalwart.toml`. + +The historical RocksDB backend (PVC `stalwart-data`, restic tag `stalwart`, `stalwart-backup` CronJob, and the `fix-permissions` init-container) was retired on 2026-08-28 after 14 healthy post-cutover days (DEV-474). No RocksDB restic chain remains; rollback is now via the Postgres `pg_dump` restic chain (`tag=stalwart-postgres`). Check https://stalw.art/docs/ref/ for configuration possibilities. The API access via /jmap seems to be too complex for the agent and the configruation has been done manually. @@ -126,11 +137,13 @@ The Kubernetes services MUST remain type `LoadBalancer` because: ## Storage -Data is stored in `/var/lib/stalwart` using the RocksDB database format. This includes: -- Email messages +All persistent state lives in **PostgreSQL** on `stalwart-postgres-0` (PVC `postgres-data-stalwart-postgres-0`, 10Gi, `hcloud-volumes-encrypted`). Stalwart connects to Postgres via the bootstrap config and stores: +- Email messages (blob + metadata) - User accounts - Server configuration -- TLS certificates configuration +- Queue / spool state + +`/var/lib/stalwart` on `stalwart-0` is an emptyDir mount kept only as a writable path if Stalwart needs one at runtime; no data of value lives there. ## Certificate Renewal diff --git a/apps/stalwart/stalwart-fresh-deployment.yaml b/apps/stalwart/stalwart-fresh-deployment.yaml index 384ac11..0496baf 100644 --- a/apps/stalwart/stalwart-fresh-deployment.yaml +++ b/apps/stalwart/stalwart-fresh-deployment.yaml @@ -5,19 +5,6 @@ metadata: name: stalwart --- apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: stalwart-data - namespace: stalwart -spec: - accessModes: - - ReadWriteOnce - storageClassName: hcloud-volumes-encrypted - resources: - requests: - storage: 20Gi ---- -apiVersion: v1 kind: Service metadata: name: stalwart-smtp @@ -114,12 +101,6 @@ spec: readOnly: true - name: config mountPath: /etc/stalwart - - name: fix-permissions - image: busybox:latest - command: ["sh", "-c", "chown -R 2000:2000 /var/lib/stalwart && chmod -R 755 /var/lib/stalwart"] - volumeMounts: - - name: data - mountPath: /var/lib/stalwart containers: - name: stalwart image: stalwartlabs/stalwart:v0.16.18 @@ -159,6 +140,9 @@ spec: - name: tls-certs mountPath: /etc/stalwart/certs readOnly: true + # DEV-474: /var/lib/stalwart is no longer RocksDB storage (state now lives in + # the stalwart-postgres StatefulSet). Kept as emptyDir so stalwart still has + # a writable path if it needs one at runtime. resources: requests: memory: "512Mi" @@ -203,9 +187,10 @@ spec: timeoutSeconds: 3 failureThreshold: 3 volumes: + # DEV-474: was persistentVolumeClaim.claimName=stalwart-data (RocksDB). + # Postgres-backed now; replaced with emptyDir. - name: data - persistentVolumeClaim: - claimName: stalwart-data + emptyDir: {} # emptyDir for config files (populated by copy-config init-container) - name: config emptyDir: {} @@ -246,188 +231,3 @@ spec: name: stalwart-http port: number: 8080 ---- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: stalwart-backup - namespace: stalwart ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: stalwart-backup - namespace: stalwart -rules: -- apiGroups: ["apps"] - resources: ["statefulsets"] - verbs: ["get", "list", "patch"] -- apiGroups: ["apps"] - resources: ["statefulsets/scale"] - verbs: ["get", "update", "patch"] -- apiGroups: [""] - resources: ["pods"] - verbs: ["get", "list", "watch"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: stalwart-backup - namespace: stalwart -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: stalwart-backup -subjects: -- kind: ServiceAccount - name: stalwart-backup - namespace: stalwart ---- -apiVersion: batch/v1 -kind: CronJob -metadata: - name: stalwart-backup - namespace: stalwart -spec: - schedule: "0 3 * * *" # 3 AM daily - successfulJobsHistoryLimit: 3 - failedJobsHistoryLimit: 3 - jobTemplate: - spec: - backoffLimit: 2 - activeDeadlineSeconds: 1800 # 30 minute timeout (was 600s; retries + affinity settling can push past 10 min) - template: - metadata: - labels: - app: stalwart-backup - spec: - serviceAccountName: stalwart-backup - restartPolicy: OnFailure - # Co-locate with stalwart-0 so both pods share the same hcloud block volume - # attachment (RWO). Without this the backup pod can be scheduled on a - # different node and hits FailedAttachVolume/Multi-Attach (DEV-468). - affinity: - podAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchLabels: - app: stalwart - statefulset.kubernetes.io/pod-name: stalwart-0 - topologyKey: kubernetes.io/hostname - containers: - - name: backup - image: alpine:3.24 - command: - - /bin/sh - - -c - - | - set -e - BACKUP_START=$(date +%s) - echo "=== Starting Stalwart backup at $(date) ===" - - # Install required tools - echo "Installing kubectl and restic..." - apk add --no-cache kubectl restic curl bash - - # Build restic repository URL from env vars - export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/stalwart" - echo "Using restic repository: $RESTIC_REPOSITORY" - - # Function to scale StatefulSet - scale_statefulset() { - local replicas=$1 - echo "Scaling stalwart StatefulSet to $replicas replicas..." - kubectl scale statefulset stalwart -n stalwart --replicas=$replicas - - if [ "$replicas" -eq 0 ]; then - echo "Waiting for pod to terminate..." - kubectl wait --for=delete pod/stalwart-0 -n stalwart --timeout=120s || true - sleep 5 - else - echo "Waiting for pod to be ready..." - kubectl wait --for=condition=ready pod/stalwart-0 -n stalwart --timeout=120s || echo "Warning: Pod not ready after 120s" - fi - } - - # Trap to ensure we scale back up on any exit - cleanup() { - EXIT_CODE=$? - echo "Cleanup: scaling stalwart back to 1 replica..." - scale_statefulset 1 || echo "ERROR: Failed to scale back up!" - - BACKUP_END=$(date +%s) - DURATION=$((BACKUP_END - BACKUP_START)) - - if [ $EXIT_CODE -eq 0 ]; then - echo "=== Backup completed successfully in ${DURATION}s at $(date) ===" - else - echo "=== Backup FAILED after ${DURATION}s at $(date) ===" - fi - - exit $EXIT_CODE - } - trap cleanup EXIT INT TERM - - # Scale down Stalwart - scale_statefulset 0 - - # Build restic repository URL from env vars (K8s doesn't expand $(VAR) in value fields) - export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/stalwart" - echo "Using repository: $RESTIC_REPOSITORY" - - # Initialize restic repo if needed - echo "Initializing restic repository..." - restic snapshots || restic init - - # Backup the data directory - echo "Running backup..." - restic backup /var/lib/stalwart \ - --tag stalwart \ - --tag daily \ - --host stalwart-k8s \ - --verbose - - # Prune old backups (keep 7 daily, 4 weekly, 6 monthly) - echo "Pruning old backups..." - restic forget \ - --tag stalwart \ - --keep-daily 7 \ - --keep-weekly 4 \ - --keep-monthly 6 \ - --prune - - echo "Backup operations completed" - env: - - name: RESTIC_PASSWORD - valueFrom: - secretKeyRef: - name: stalwart-s3-backup - key: restic-password - - name: AWS_ACCESS_KEY_ID - valueFrom: - secretKeyRef: - name: stalwart-s3-backup - key: access-key - - name: AWS_SECRET_ACCESS_KEY - valueFrom: - secretKeyRef: - name: stalwart-s3-backup - key: secret-key - - name: S3_ENDPOINT - valueFrom: - secretKeyRef: - name: stalwart-s3-backup - key: endpoint - - name: S3_BUCKET - valueFrom: - secretKeyRef: - name: stalwart-s3-backup - key: bucket - volumeMounts: - - name: data - mountPath: /var/lib/stalwart - readOnly: true - volumes: - - name: data - persistentVolumeClaim: - claimName: stalwart-data