Post-cutover 14-day observation window on DEV-474 closed today with 14/14
healthy days. This removes every RocksDB-era artefact from the manifest and
updates the README to reflect the PostgreSQL backend as the default:
- Delete PVC `stalwart-data` (was 20Gi hcloud-volumes-encrypted, RWO).
- Drop the `fix-permissions` init-container (existed only to chown the
RocksDB PVC mount).
- Replace the StatefulSet's `data` volume with an `emptyDir` and keep the
`/var/lib/stalwart` mount as a plain scratch path (Stalwart's state is
now in the `stalwart-postgres` StatefulSet).
- Delete the `stalwart-backup` CronJob and its `stalwart-backup`
ServiceAccount / Role / RoleBinding — RocksDB snapshotting is obsolete
now that state lives in Postgres. Daily backup is via
`stalwart-postgres-backup` (restic tag `stalwart-postgres`).
- README: update version, backend, storage, backup, files, and deployment
sections; correct exposed-ports language to match SMTP 25/587 + IMAPS 993.
Executed alongside the merge:
- `restic forget --prune` removed all 11 snapshots on tag=stalwart (Job
`stalwart-restic-forget-rocksdb` on 2026-08-28 15:09 UTC, log preserved
on DEV-474). 16.165 MiB pruned. `restic snapshots --tag stalwart` now
returns empty; only the 14 tag=`stalwart-postgres` snapshots remain.
- `stalwart-backup` CronJob was pre-suspended in-cluster to prevent
tonight's 03:00 UTC fire from re-creating a stalwart-tagged snapshot
before this PR merges.
Rollback (if needed before merge): revert this commit; the old CronJob
would then re-create RocksDB snapshots from live PVC contents. After
merge the rollback path is the Postgres `pg_dump` restic chain only.
Refs: DEV-467, DEV-473 (cutover), DEV-474 (this observation + teardown),
DEV-548 (day-14 execution issue).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added comprehensive documentation of the two-tier load balancing setup:
- Hetzner Cloud Load Balancer (external layer, managed by Hetzner CCM)
- Kubernetes LoadBalancer services (internal layer, k3s ServiceLB)
Key points documented:
- Traffic flow from external client through both LB layers to pod
- Why LoadBalancer service type is required (CCM integration)
- Historical context of the migration from hostPort to Hetzner LB
- Service definitions and port configurations
Updated:
- apps/stalwart/README.md: Added Network Architecture section
- infrastructure/networking/NETWORK_ARCHITECTURE.md: Enhanced Stalwart
section with two-tier architecture details and updated traffic flows
Resolves documentation gap identified in DEV-439.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add comprehensive Network Architecture section explaining:
- Two-tier load balancing (K8s LoadBalancer + Hetzner LB)
- Why LoadBalancer service type is required (Hetzner CCM auto-management)
- Traffic flow from external clients to Stalwart pods
- Warning against changing to NodePort (breaks CCM integration)
This documents the investigation from DEV-439, confirming that the current
LoadBalancer service setup is correct and necessary for our infrastructure.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Extends commit 291feb9 by adding stalwart-config (stalwart.toml) to the
copy-config initContainer. The previous fix only handled bootstrap-config
but missed stalwart-config, which was the root cause of CrashLoopBackOff
identified in DEV-426 stability testing.
Root cause: commit 693fcd3 introduced stalwart-config ConfigMap with
subPath mount. During pod restarts, Kubernetes subPath ConfigMap mounting
race leaves stalwart.toml empty/unparseable, causing:
"Failed to parse data store settings at /etc/stalwart/stalwart.toml:
expected value at line 1 column 1"
This completes the hardening by ensuring BOTH config files (config.json
and stalwart.toml) are atomically copied before Stalwart starts.
Fixes: DEV-433, DEV-431
Ref: stack.basicstack.de/apps/stalwart/STABILITY-VERIFICATION-2026-08-01.md
Co-Authored-By: Paperclip <noreply@paperclip.ing>
