Following the DEV-473 rocksdb-to-postgres migration, point Stalwart's bootstrap
config at the PostgreSQL store and expose PGPASSWORD to the container so the
`authSecret: EnvironmentVariable` lookup resolves.
- stalwart-bootstrap-config.yaml: config.json switches @type RocksDb -> PostgreSql
(host=stalwart-postgres, db=stalwart, authUsername=stalwart,
authSecret=EnvironmentVariable/PGPASSWORD, useTls=false).
- stalwart-fresh-deployment.yaml: adds PGPASSWORD env from
secret/stalwart-postgres-credentials.POSTGRES_PASSWORD on the stalwart container.
- stalwart-config.yaml: mirrors the switch in stalwart.toml as documentation
(the running pod uses `--config /etc/stalwart/config.json`, not the toml).
Board (DEV-476) explicitly asked for the manifest-level switch after the web UI
save failed with a `Permission denied` on the read-only bootstrap emptyDir.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The stalwart-backup CronJob has failed nightly since 2026-08-13, all with
FailureTarget=DeadlineExceeded. Root cause: the backup pod had no
scheduling constraint and got placed on a node different from stalwart-0.
The hcloud CSI block volume is RWO and can only be attached to one node,
so the backup pod stayed in ContainerCreating with FailedAttachVolume /
Multi-Attach until the 600s active deadline killed it.
- Add podAffinity requiredDuringScheduling on
app=stalwart,statefulset.kubernetes.io/pod-name=stalwart-0 with topology
key kubernetes.io/hostname so the backup pod always lands on the same
node. Same-node co-location lets both pods share the already-attached
block volume; the in-container script then scales stalwart-0 down,
backs up, and scales it back up as before.
- Raise activeDeadlineSeconds from 600s to 1800s as safety headroom
(successful runs are ~86s; the extra budget covers prune growth).
Verified: manual run of the patched CronJob completed in 86s and wrote
restic snapshot f76f534e (2026-08-15 12:07:30) to
s3://basicstack-backup/stalwart. stalwart-0 is back to Ready.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Extends commit 291feb9 by adding stalwart-config (stalwart.toml) to the
copy-config initContainer. The previous fix only handled bootstrap-config
but missed stalwart-config, which was the root cause of CrashLoopBackOff
identified in DEV-426 stability testing.
Root cause: commit 693fcd3 introduced stalwart-config ConfigMap with
subPath mount. During pod restarts, Kubernetes subPath ConfigMap mounting
race leaves stalwart.toml empty/unparseable, causing:
"Failed to parse data store settings at /etc/stalwart/stalwart.toml:
expected value at line 1 column 1"
This completes the hardening by ensuring BOTH config files (config.json
and stalwart.toml) are atomically copied before Stalwart starts.
Fixes: DEV-433, DEV-431
Ref: stack.basicstack.de/apps/stalwart/STABILITY-VERIFICATION-2026-08-01.md
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Issue: During stability testing (DEV-426), pod restart hit a ConfigMap
subPath mounting race, leaving config.json empty/unparseable and causing
CrashLoopBackOff. This undermines the HA work.
Root cause: Kubernetes subPath ConfigMap mounts can race during pod
restart, resulting in empty or incomplete files before the container starts.
Changes:
- Add copy-config init-container that copies ConfigMap files to emptyDir
- Replace subPath mount with directory mount from emptyDir
- Config files are now guaranteed to be complete before Stalwart starts
- Eliminates the ConfigMap subPath mounting race entirely
This is a critical hardening fix for production mail server stability.
Resolves DEV-431.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This reverts commit c060c83. The TOML configuration approach caused
Stalwart to fail to start due to configuration parsing errors.
Will implement a different approach that doesn't require a full
configuration file rewrite.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Stalwart was blocking the HTTP port (8080) from Traefik's internal IP
(10.244.2.227), causing 502 errors when accessing mail.basicstack.de.
Changes:
- Added complete Stalwart TOML configuration (stalwart-config.yaml)
- Configured HTTP listener security to allow internal pod network (10.244.0.0/16)
- Updated StatefulSet to use the new configuration file
- This allows Traefik ingress to reach the Stalwart web UI backend
The fix is non-destructive:
- PVC data is preserved
- Rolling update will restart the pod with new config
- Only security setting is changed (adding allowed IPs)
Fixes: DEV-422
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The pod was crash-looping because Stalwart's security configuration
blocks the kubelet's IP (10.244.4.1) from accessing the HTTP health
endpoints. The kubelet's health checks were failing, causing the
startup probe to fail after 6 attempts, leading to pod restarts.
Changed all three health probes (startup, liveness, readiness) from
httpGet to exec with curl localhost. This bypasses the IP blocking
since the health check runs from inside the container using localhost,
which is not subject to Stalwart's external IP blocking rules.
This fix is non-destructive to Stalwart's configuration and state.
The pod will restart once with the new probe configuration, but no
data or configuration will be lost.
Root cause: Stalwart logs showed "Blocked IP address (security.ip-blocked)
listenerId=http, remoteIp=10.244.4.1" followed by "Shutting down Stalwart
Server (server.shutdown) causedBy=SIGTERM" in a repeating pattern.
Fixes: DEV-420
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Changed both stalwart-smtp and stalwart-imap services from
externalTrafficPolicy: Local to externalTrafficPolicy: Cluster.
This enables all 4 k3s nodes to appear healthy in Hetzner Load
Balancer health checks, instead of only k3s-worker-1 (where the
single Stalwart pod runs).
With Cluster policy, traffic arriving at any node will be
forwarded via kube-proxy to k3s-worker-1 where the pod is running.
Verification:
- All 4 nodes now show in service EXTERNAL-IP field
- Mail ports (25, 587, 993) accessible from all nodes
- Connectivity tested from k3s-worker-1 and k3s-worker-2
Fixes: DEV-368
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Drop legacy insecure mail ports per DEV-359 approval:
- Remove port 465 (SMTPS) from SMTP service - enforce STARTTLS on 587
- Remove port 143 (IMAP) from IMAP service - enforce TLS on 993
This reduces attack surface and enforces secure mail protocols.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: ArgoCD continuously reverts the LoadBalancer services to use
externalTrafficPolicy: Cluster (the k8s default), causing k3s to assign
internal flannel VXLAN IPs (10.42.1.x) instead of the node public IPs.
With externalTrafficPolicy: Cluster, traffic can be routed to any node,
and k3s's service controller assigns the flannel overlay IPs. This breaks
external connectivity because those IPs are not routable from outside.
With externalTrafficPolicy: Local, traffic is only routed to pods on the
same node, and k3s assigns the node's actual public IP to the LoadBalancer.
This was the missing piece from the reliability hardening in commit b0f2acf.
Without this in git, any manual kubectl patch is reverted by ArgoCD sync.
Evidence: stalwart-smtp and stalwart-imap both showing LoadBalancer IPs:
10.42.1.1, 10.42.1.2, 10.42.1.3, 10.42.1.5 (internal flannel IPs)
Related: DEV-230, DEV-231, DEV-233, DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Implement scale-down/backup/scale-up pattern to work around PVC access mode limitation.
Changes:
- Add RBAC (ServiceAccount, Role, RoleBinding) with statefulsets/scale and pods permissions
- Switch to alpine:3.19 base image with kubectl and restic
- Scale down StatefulSet to 0 replicas before backup
- Run restic backup while pod is stopped
- Scale back up to 1 replica with error handling
- Add cleanup trap to ensure scale-up even on failure
- Set 10-minute timeout and backoff limit
Tested successfully: backup completes in ~32 seconds with minimal downtime.
Resolves DEV-236.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.
Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures
Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service
Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.
Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added Stakater Reloader to automatically restart Stalwart pods when
TLS certificates are renewed by cert-manager. This ensures seamless
certificate rotation without manual intervention.
Changes:
- Deploy Stakater Reloader in infrastructure/networking/
- Add Reloader annotation to Stalwart StatefulSet to watch stalwart-tls secret
- Document certificate renewal process and troubleshooting
The certificate is managed by cert-manager with Let's Encrypt and will
automatically renew 30 days before expiration (renewal date: 2026-08-20).
Reloader detects secret updates and triggers a rolling restart of the
Stalwart StatefulSet to load the new certificate.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Removed stalwart-config ConfigMap (config.toml not used in v0.16.11)
- All configuration is done via Stalwart API and stored in RocksDB
- Added TLS certificate mount from stalwart-tls secret
- Using stalwartlabs/stalwart:v0.16.11
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Fixed init container to set ownership to 2000:2000
- Stalwart container runs as user 2000 by default
- Resolves permission denied error during web UI setup
Refs: DEV-206
Co-Authored-By: Paperclip <noreply@paperclip.ing>
