chore(stalwart): retire RocksDB PVC + backup — Postgres backend is authoritative (DEV-474)

Post-cutover 14-day observation window on DEV-474 closed today with 14/14
healthy days. This removes every RocksDB-era artefact from the manifest and
updates the README to reflect the PostgreSQL backend as the default:

- Delete PVC `stalwart-data` (was 20Gi hcloud-volumes-encrypted, RWO).
- Drop the `fix-permissions` init-container (existed only to chown the
  RocksDB PVC mount).
- Replace the StatefulSet's `data` volume with an `emptyDir` and keep the
  `/var/lib/stalwart` mount as a plain scratch path (Stalwart's state is
  now in the `stalwart-postgres` StatefulSet).
- Delete the `stalwart-backup` CronJob and its `stalwart-backup`
  ServiceAccount / Role / RoleBinding — RocksDB snapshotting is obsolete
  now that state lives in Postgres. Daily backup is via
  `stalwart-postgres-backup` (restic tag `stalwart-postgres`).
- README: update version, backend, storage, backup, files, and deployment
  sections; correct exposed-ports language to match SMTP 25/587 + IMAPS 993.

Executed alongside the merge:

- `restic forget --prune` removed all 11 snapshots on tag=stalwart (Job
  `stalwart-restic-forget-rocksdb` on 2026-08-28 15:09 UTC, log preserved
  on DEV-474). 16.165 MiB pruned. `restic snapshots --tag stalwart` now
  returns empty; only the 14 tag=`stalwart-postgres` snapshots remain.
- `stalwart-backup` CronJob was pre-suspended in-cluster to prevent
  tonight's 03:00 UTC fire from re-creating a stalwart-tagged snapshot
  before this PR merges.

Rollback (if needed before merge): revert this commit; the old CronJob
would then re-create RocksDB snapshots from live PVC contents. After
merge the rollback path is the Postgres `pg_dump` restic chain only.

Refs: DEV-467, DEV-473 (cutover), DEV-474 (this observation + teardown),
DEV-548 (day-14 execution issue).

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/stalwart/README.md

@@ -1,21 +1,21 @@
-# Stalwart Mail Server v0.16.11
+# Stalwart Mail Server v0.16.18
 
-Clean deployment of Stalwart mail server with username/password authentication only.
+Deployment of Stalwart mail server on a PostgreSQL backend (RocksDB was retired in DEV-467/DEV-473; teardown completed in DEV-474).
 
 ## Architecture
 
-- **Version**: v0.16.11
+- **Version**: v0.16.18
-- **Authentication**: Username/password only (NO OAuth/OIDC)
+- **Authentication**: OIDC via Pocket-ID (backed by Postgres accounts store)
-- **Configuration**: API-based (stored in RocksDB)
+- **Configuration + state**: PostgreSQL (`stalwart-postgres` StatefulSet in-namespace)
-- **Storage**: Encrypted hcloud-volumes (20Gi)
+- **Storage**: Encrypted hcloud-volumes attached to `stalwart-postgres-0` (10Gi). Stalwart itself uses an emptyDir at `/var/lib/stalwart` — no persistent block volume on `stalwart-0`
-- **Backup**: Daily restic backup to S3 at 3 AM
+- **Backup**: Daily Postgres logical dump to S3 via `stalwart-postgres-backup` CronJob (restic tag `stalwart-postgres`) at 02:30 UTC
 - **Web UI**: https://mail.basicstack.de
 
 ## Network Architecture
 
 Stalwart uses a two-tier load balancing setup:
 
-1. **Kubernetes LoadBalancer Services**: Four separate LoadBalancer services expose SMTP (25, 587, 465) and IMAP (143, 993) ports. The service type `LoadBalancer` is required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type.
+1. **Kubernetes LoadBalancer Services**: Two LoadBalancer services expose SMTP (25, 587) and IMAPS (993) ports (SMTPS/465 removed per DEV-359 approval, plaintext IMAP/143 disabled). The service type `LoadBalancer` is required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type.
 
 2. **Hetzner Load Balancer**: Automatically managed by the Hetzner CCM. The load balancer distributes traffic across:
    - k3s-cp-1 (control plane node)
@@ -37,8 +37,13 @@ The Hetzner CCM watches for LoadBalancer-type services and automatically:
 
 ## Files
 
-- `stalwart-fresh-deployment.yaml` - Main deployment manifest
+- `stalwart-fresh-deployment.yaml` - Main deployment manifest (StatefulSet, services, ingress)
+- `stalwart-postgres.yaml` - Postgres backend StatefulSet + service (state store)
+- `stalwart-postgres-backup.yaml` - Daily `pg_dump` → restic CronJob (tag `stalwart-postgres`)
+- `stalwart-config.yaml` - Stalwart configmap (`stalwart.toml`)
+- `stalwart-bootstrap-config.yaml` - Bootstrap configmap pointing Stalwart at Postgres
 - `stalwart-admin-credentials-sealed.yaml` - Sealed secret for admin password
+- `stalwart-postgres-credentials-sealed.yaml` - Sealed secret for Postgres role
 - `stalwart-s3-backup-sealed.yaml` - Sealed secret for S3 backup credentials
 
 ## Deployment
@@ -46,12 +51,16 @@ The Hetzner CCM watches for LoadBalancer-type services and automatically:
 ```bash
 # Apply sealed secrets first
 kubectl apply -f stalwart-admin-credentials-sealed.yaml
+kubectl apply -f stalwart-postgres-credentials-sealed.yaml
 kubectl apply -f stalwart-s3-backup-sealed.yaml
 
-# Create bootstrap config
+# Configmaps: bootstrap points Stalwart at Postgres; stalwart-config carries stalwart.toml
-kubectl create configmap stalwart-bootstrap-config \
+kubectl apply -f stalwart-bootstrap-config.yaml
-  --from-literal=config.json='{"@type":"RocksDb","path":"/var/lib/stalwart"}' \
+kubectl apply -f stalwart-config.yaml
-  -n stalwart
+
+# Postgres backend + backup
+kubectl apply -f stalwart-postgres.yaml
+kubectl apply -f stalwart-postgres-backup.yaml
 
 # Deploy Stalwart
 kubectl apply -f stalwart-fresh-deployment.yaml
@@ -65,9 +74,11 @@ After deployment, log in at https://mail.basicstack.de with:
 
 ## Configuration
 
-> **IMPORTANT:** The configuration is stored in the RocksDB and cannot be overwritten by a configuration file!
+> **IMPORTANT:** Runtime configuration and account/queue state now live in **PostgreSQL** (`stalwart-postgres-0`), not RocksDB.
 
-All configuration is done via the web UI or API. The bootstrap config only points to the RocksDB database location. NO config.toml files are used.
+The `stalwart-bootstrap-config` configmap points Stalwart at the Postgres store; runtime edits made in the web UI or API are persisted to Postgres. The `stalwart-config` configmap carries the boot-time `stalwart.toml`.
+
+The historical RocksDB backend (PVC `stalwart-data`, restic tag `stalwart`, `stalwart-backup` CronJob, and the `fix-permissions` init-container) was retired on 2026-08-28 after 14 healthy post-cutover days (DEV-474). No RocksDB restic chain remains; rollback is now via the Postgres `pg_dump` restic chain (`tag=stalwart-postgres`).
 
 Check https://stalw.art/docs/ref/ for configuration possibilities.
 The API access via /jmap seems to be too complex for the agent and the configruation has been done manually.
@@ -126,11 +137,13 @@ The Kubernetes services MUST remain type `LoadBalancer` because:
 
 ## Storage
 
-Data is stored in `/var/lib/stalwart` using the RocksDB database format. This includes:
+All persistent state lives in **PostgreSQL** on `stalwart-postgres-0` (PVC `postgres-data-stalwart-postgres-0`, 10Gi, `hcloud-volumes-encrypted`). Stalwart connects to Postgres via the bootstrap config and stores:
-- Email messages
+- Email messages (blob + metadata)
 - User accounts
 - Server configuration
-- TLS certificates configuration
+- Queue / spool state
+
+`/var/lib/stalwart` on `stalwart-0` is an emptyDir mount kept only as a writable path if Stalwart needs one at runtime; no data of value lives there.
 
 ## Certificate Renewal
 

apps/stalwart/stalwart-fresh-deployment.yaml

@@ -5,19 +5,6 @@ metadata:
   name: stalwart
 ---
 apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: stalwart-data
-  namespace: stalwart
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: hcloud-volumes-encrypted
-  resources:
-    requests:
-      storage: 20Gi
----
-apiVersion: v1
 kind: Service
 metadata:
   name: stalwart-smtp
@@ -114,12 +101,6 @@ spec:
           readOnly: true
         - name: config
           mountPath: /etc/stalwart
-      - name: fix-permissions
-        image: busybox:latest
-        command: ["sh", "-c", "chown -R 2000:2000 /var/lib/stalwart && chmod -R 755 /var/lib/stalwart"]
-        volumeMounts:
-        - name: data
-          mountPath: /var/lib/stalwart
       containers:
       - name: stalwart
         image: stalwartlabs/stalwart:v0.16.18
@@ -159,6 +140,9 @@ spec:
         - name: tls-certs
           mountPath: /etc/stalwart/certs
           readOnly: true
+        # DEV-474: /var/lib/stalwart is no longer RocksDB storage (state now lives in
+        # the stalwart-postgres StatefulSet). Kept as emptyDir so stalwart still has
+        # a writable path if it needs one at runtime.
         resources:
           requests:
             memory: "512Mi"
@@ -203,9 +187,10 @@ spec:
           timeoutSeconds: 3
           failureThreshold: 3
       volumes:
+      # DEV-474: was persistentVolumeClaim.claimName=stalwart-data (RocksDB).
+      # Postgres-backed now; replaced with emptyDir.
       - name: data
-        persistentVolumeClaim:
+        emptyDir: {}
-          claimName: stalwart-data
       # emptyDir for config files (populated by copy-config init-container)
       - name: config
         emptyDir: {}
@@ -246,188 +231,3 @@ spec:
             name: stalwart-http
             port:
               number: 8080
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: stalwart-backup
-  namespace: stalwart
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: stalwart-backup
-  namespace: stalwart
-rules:
-- apiGroups: ["apps"]
-  resources: ["statefulsets"]
-  verbs: ["get", "list", "patch"]
-- apiGroups: ["apps"]
-  resources: ["statefulsets/scale"]
-  verbs: ["get", "update", "patch"]
-- apiGroups: [""]
-  resources: ["pods"]
-  verbs: ["get", "list", "watch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: stalwart-backup
-  namespace: stalwart
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: stalwart-backup
-subjects:
-- kind: ServiceAccount
-  name: stalwart-backup
-  namespace: stalwart
----
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: stalwart-backup
-  namespace: stalwart
-spec:
-  schedule: "0 3 * * *"  # 3 AM daily
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    spec:
-      backoffLimit: 2
-      activeDeadlineSeconds: 1800  # 30 minute timeout (was 600s; retries + affinity settling can push past 10 min)
-      template:
-        metadata:
-          labels:
-            app: stalwart-backup
-        spec:
-          serviceAccountName: stalwart-backup
-          restartPolicy: OnFailure
-          # Co-locate with stalwart-0 so both pods share the same hcloud block volume
-          # attachment (RWO). Without this the backup pod can be scheduled on a
-          # different node and hits FailedAttachVolume/Multi-Attach (DEV-468).
-          affinity:
-            podAffinity:
-              requiredDuringSchedulingIgnoredDuringExecution:
-              - labelSelector:
-                  matchLabels:
-                    app: stalwart
-                    statefulset.kubernetes.io/pod-name: stalwart-0
-                topologyKey: kubernetes.io/hostname
-          containers:
-          - name: backup
-            image: alpine:3.24
-            command:
-            - /bin/sh
-            - -c
-            - |
-              set -e
-              BACKUP_START=$(date +%s)
-              echo "=== Starting Stalwart backup at $(date) ==="
-
-              # Install required tools
-              echo "Installing kubectl and restic..."
-              apk add --no-cache kubectl restic curl bash
-
-              # Build restic repository URL from env vars
-              export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/stalwart"
-              echo "Using restic repository: $RESTIC_REPOSITORY"
-
-              # Function to scale StatefulSet
-              scale_statefulset() {
-                local replicas=$1
-                echo "Scaling stalwart StatefulSet to $replicas replicas..."
-                kubectl scale statefulset stalwart -n stalwart --replicas=$replicas
-
-                if [ "$replicas" -eq 0 ]; then
-                  echo "Waiting for pod to terminate..."
-                  kubectl wait --for=delete pod/stalwart-0 -n stalwart --timeout=120s || true
-                  sleep 5
-                else
-                  echo "Waiting for pod to be ready..."
-                  kubectl wait --for=condition=ready pod/stalwart-0 -n stalwart --timeout=120s || echo "Warning: Pod not ready after 120s"
-                fi
-              }
-
-              # Trap to ensure we scale back up on any exit
-              cleanup() {
-                EXIT_CODE=$?
-                echo "Cleanup: scaling stalwart back to 1 replica..."
-                scale_statefulset 1 || echo "ERROR: Failed to scale back up!"
-
-                BACKUP_END=$(date +%s)
-                DURATION=$((BACKUP_END - BACKUP_START))
-
-                if [ $EXIT_CODE -eq 0 ]; then
-                  echo "=== Backup completed successfully in ${DURATION}s at $(date) ==="
-                else
-                  echo "=== Backup FAILED after ${DURATION}s at $(date) ==="
-                fi
-
-                exit $EXIT_CODE
-              }
-              trap cleanup EXIT INT TERM
-
-              # Scale down Stalwart
-              scale_statefulset 0
-
-              # Build restic repository URL from env vars (K8s doesn't expand $(VAR) in value fields)
-              export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/stalwart"
-              echo "Using repository: $RESTIC_REPOSITORY"
-
-              # Initialize restic repo if needed
-              echo "Initializing restic repository..."
-              restic snapshots || restic init
-
-              # Backup the data directory
-              echo "Running backup..."
-              restic backup /var/lib/stalwart \
-                --tag stalwart \
-                --tag daily \
-                --host stalwart-k8s \
-                --verbose
-
-              # Prune old backups (keep 7 daily, 4 weekly, 6 monthly)
-              echo "Pruning old backups..."
-              restic forget \
-                --tag stalwart \
-                --keep-daily 7 \
-                --keep-weekly 4 \
-                --keep-monthly 6 \
-                --prune
-
-              echo "Backup operations completed"
-            env:
-            - name: RESTIC_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: stalwart-s3-backup
-                  key: restic-password
-            - name: AWS_ACCESS_KEY_ID
-              valueFrom:
-                secretKeyRef:
-                  name: stalwart-s3-backup
-                  key: access-key
-            - name: AWS_SECRET_ACCESS_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: stalwart-s3-backup
-                  key: secret-key
-            - name: S3_ENDPOINT
-              valueFrom:
-                secretKeyRef:
-                  name: stalwart-s3-backup
-                  key: endpoint
-            - name: S3_BUCKET
-              valueFrom:
-                secretKeyRef:
-                  name: stalwart-s3-backup
-                  key: bucket
-            volumeMounts:
-            - name: data
-              mountPath: /var/lib/stalwart
-              readOnly: true
-          volumes:
-          - name: data
-            persistentVolumeClaim:
-              claimName: stalwart-data