The stalwart-backup CronJob has failed nightly since 2026-08-13, all with
FailureTarget=DeadlineExceeded. Root cause: the backup pod had no
scheduling constraint and got placed on a node different from stalwart-0.
The hcloud CSI block volume is RWO and can only be attached to one node,
so the backup pod stayed in ContainerCreating with FailedAttachVolume /
Multi-Attach until the 600s active deadline killed it.
- Add podAffinity requiredDuringScheduling on
app=stalwart,statefulset.kubernetes.io/pod-name=stalwart-0 with topology
key kubernetes.io/hostname so the backup pod always lands on the same
node. Same-node co-location lets both pods share the already-attached
block volume; the in-container script then scales stalwart-0 down,
backs up, and scales it back up as before.
- Raise activeDeadlineSeconds from 600s to 1800s as safety headroom
(successful runs are ~86s; the extra budget covers prune growth).
Verified: manual run of the patched CronJob completed in 86s and wrote
restic snapshot f76f534e (2026-08-15 12:07:30) to
s3://basicstack-backup/stalwart. stalwart-0 is back to Ready.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Phase 2 of the Stalwart RocksDB -> PostgreSQL migration (DEV-467).
Purely additive - does not touch the running stalwart-0 pod, its
ConfigMap, PVC, or bootstrap-config.
- stalwart-postgres.yaml: single-replica StatefulSet with a 10Gi
hcloud-volumes-encrypted PVC (subPath pgdata), ClusterIP Service
on 5432, nodeAffinity csi.hetzner.cloud/location=fsn1 so the DB
co-locates with stalwart-0 (which is fsn1-pinned by its PVC).
Resources match forgejo-postgres (250m/512Mi req, 500m/1Gi lim).
- stalwart-postgres-credentials-sealed.yaml: sealed secret with
POSTGRES_USER=stalwart, POSTGRES_DB=stalwart, POSTGRES_PASSWORD,
plus a copy of the password under stalwart-db-password for
Stalwart's [store.postgres] config in the Phase 4 cutover.
- stalwart-postgres-backup.yaml: daily CronJob at 02:30 UTC that
streams pg_dump | gzip into the existing stalwart-s3-backup restic
repo tagged stalwart-postgres, with independent retention keys
(14d/8w/6m) so it doesn't collide with the RocksDB snapshots.
activeDeadlineSeconds=1800 mirrors the DEV-464 sibling fix.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Investigation confirmed all SMTP/IMAP ports working correctly as of 21:00 UTC.
SMTP issue at 20:31 UTC coincided with LoadBalancer IP updates (10.42.1.1 → 178.105.17.239).
External port tests verify correct protocol greetings on all ports.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added comprehensive documentation of the two-tier load balancing setup:
- Hetzner Cloud Load Balancer (external layer, managed by Hetzner CCM)
- Kubernetes LoadBalancer services (internal layer, k3s ServiceLB)
Key points documented:
- Traffic flow from external client through both LB layers to pod
- Why LoadBalancer service type is required (CCM integration)
- Historical context of the migration from hostPort to Hetzner LB
- Service definitions and port configurations
Updated:
- apps/stalwart/README.md: Added Network Architecture section
- infrastructure/networking/NETWORK_ARCHITECTURE.md: Enhanced Stalwart
section with two-tier architecture details and updated traffic flows
Resolves documentation gap identified in DEV-439.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add comprehensive Network Architecture section explaining:
- Two-tier load balancing (K8s LoadBalancer + Hetzner LB)
- Why LoadBalancer service type is required (Hetzner CCM auto-management)
- Traffic flow from external clients to Stalwart pods
- Warning against changing to NodePort (breaks CCM integration)
This documents the investigation from DEV-439, confirming that the current
LoadBalancer service setup is correct and necessary for our infrastructure.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Test documentation for the ConfigMap subPath mount hardening deployed
in commit 291feb9.
Results:
- Pod deletion test: 3/3 passed with 0 restarts (69s, ~80s, 93s recovery)
- Alert test: Completed but found Alertmanager sends to null receiver
- Mail service health: Verified healthy after all tests
DEV-431.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Extends commit 291feb9 by adding stalwart-config (stalwart.toml) to the
copy-config initContainer. The previous fix only handled bootstrap-config
but missed stalwart-config, which was the root cause of CrashLoopBackOff
identified in DEV-426 stability testing.
Root cause: commit 693fcd3 introduced stalwart-config ConfigMap with
subPath mount. During pod restarts, Kubernetes subPath ConfigMap mounting
race leaves stalwart.toml empty/unparseable, causing:
"Failed to parse data store settings at /etc/stalwart/stalwart.toml:
expected value at line 1 column 1"
This completes the hardening by ensuring BOTH config files (config.json
and stalwart.toml) are atomically copied before Stalwart starts.
Fixes: DEV-433, DEV-431
Ref: stack.basicstack.de/apps/stalwart/STABILITY-VERIFICATION-2026-08-01.md
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Issue: During stability testing (DEV-426), pod restart hit a ConfigMap
subPath mounting race, leaving config.json empty/unparseable and causing
CrashLoopBackOff. This undermines the HA work.
Root cause: Kubernetes subPath ConfigMap mounts can race during pod
restart, resulting in empty or incomplete files before the container starts.
Changes:
- Add copy-config init-container that copies ConfigMap files to emptyDir
- Replace subPath mount with directory mount from emptyDir
- Config files are now guaranteed to be complete before Stalwart starts
- Eliminates the ConfigMap subPath mounting race entirely
This is a critical hardening fix for production mail server stability.
Resolves DEV-431.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This reverts commit c060c83. The TOML configuration approach caused
Stalwart to fail to start due to configuration parsing errors.
Will implement a different approach that doesn't require a full
configuration file rewrite.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Stalwart was blocking the HTTP port (8080) from Traefik's internal IP
(10.244.2.227), causing 502 errors when accessing mail.basicstack.de.
Changes:
- Added complete Stalwart TOML configuration (stalwart-config.yaml)
- Configured HTTP listener security to allow internal pod network (10.244.0.0/16)
- Updated StatefulSet to use the new configuration file
- This allows Traefik ingress to reach the Stalwart web UI backend
The fix is non-destructive:
- PVC data is preserved
- Rolling update will restart the pod with new config
- Only security setting is changed (adding allowed IPs)
Fixes: DEV-422
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The pod was crash-looping because Stalwart's security configuration
blocks the kubelet's IP (10.244.4.1) from accessing the HTTP health
endpoints. The kubelet's health checks were failing, causing the
startup probe to fail after 6 attempts, leading to pod restarts.
Changed all three health probes (startup, liveness, readiness) from
httpGet to exec with curl localhost. This bypasses the IP blocking
since the health check runs from inside the container using localhost,
which is not subject to Stalwart's external IP blocking rules.
This fix is non-destructive to Stalwart's configuration and state.
The pod will restart once with the new probe configuration, but no
data or configuration will be lost.
Root cause: Stalwart logs showed "Blocked IP address (security.ip-blocked)
listenerId=http, remoteIp=10.244.4.1" followed by "Shutting down Stalwart
Server (server.shutdown) causedBy=SIGTERM" in a repeating pattern.
Fixes: DEV-420
Co-Authored-By: Paperclip <noreply@paperclip.ing>
CONTEXT: DEV-403 - Stalwart pod not starting after ConfigMap revert
ROOT CAUSE:
- The stalwart-bootstrap-config ConfigMap was manually created on July 5
- It was added to GitOps at commit 191614f (12:27 today)
- It was reverted at commit fcc5042 (12:56 today)
- ArgoCD pruned the ConfigMap from the cluster after the revert
- Stalwart deployment requires this ConfigMap to start
RECOVERY:
This commit restores the ConfigMap to Git so ArgoCD can recreate it.
The ConfigMap content is identical to the working version from before.
This is a non-destructive recovery - no data or accounts are affected.
The actual Stalwart data on PVC stalwart-data remains intact.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The stalwart-bootstrap-config ConfigMap was previously created manually
via kubectl and not tracked in Git. This adds it to the repository for
proper GitOps management via Argo CD.
Related to DEV-403 - Stalwart sync hanging issue.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add argocd.argoproj.io/compare-options: IgnoreExtraneous annotation to
both Stalwart sealed secret templates (stalwart-admin-credentials and
stalwart-s3-backup) to prevent ArgoCD from seeing the unsealed secrets
(created by sealed-secrets controller) as extraneous.
This is the same fix applied in DEV-377 (Directus) and DEV-378 (Harbor).
Resolves: DEV-379
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Changed both stalwart-smtp and stalwart-imap services from
externalTrafficPolicy: Local to externalTrafficPolicy: Cluster.
This enables all 4 k3s nodes to appear healthy in Hetzner Load
Balancer health checks, instead of only k3s-worker-1 (where the
single Stalwart pod runs).
With Cluster policy, traffic arriving at any node will be
forwarded via kube-proxy to k3s-worker-1 where the pod is running.
Verification:
- All 4 nodes now show in service EXTERNAL-IP field
- Mail ports (25, 587, 993) accessible from all nodes
- Connectivity tested from k3s-worker-1 and k3s-worker-2
Fixes: DEV-368
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Drop legacy insecure mail ports per DEV-359 approval:
- Remove port 465 (SMTPS) from SMTP service - enforce STARTTLS on 587
- Remove port 143 (IMAP) from IMAP service - enforce TLS on 993
This reduces attack surface and enforces secure mail protocols.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Start new 7-day verification window: 2026-07-13 21:07 UTC → 2026-07-20 21:07 UTC
- CEO confirmed Web-UI SMTP and IMAP working
- Restarting verification per Option A (clean 7-day window)
- Baseline status: all services verified working
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Verified all infrastructure working correctly (SMTP/IMAP/LoadBalancer)
- Identified user SMTP issue as client configuration (port 143 vs 587)
- Documented proposed verification window restart (2026-07-13 to 2026-07-20)
- Original window had 2+ days with broken LoadBalancer (10.42.1.1 vs 178.105.17.239)
Related: DEV-300
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Investigation revealed that k3s ServiceLB uses hostPort bindings on nodes'
public IPs, not the misleading EXTERNAL-IP VIP (10.42.1.1) shown in kubectl.
External connectivity was working correctly the entire time. The confusion
was caused by misinterpreting the flannel pod network IP in the EXTERNAL-IP
field as meaning external traffic couldn't reach the service.
Verified all SMTP/IMAP ports responding correctly with proper Stalwart banners.
Related: DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: ArgoCD continuously reverts the LoadBalancer services to use
externalTrafficPolicy: Cluster (the k8s default), causing k3s to assign
internal flannel VXLAN IPs (10.42.1.x) instead of the node public IPs.
With externalTrafficPolicy: Cluster, traffic can be routed to any node,
and k3s's service controller assigns the flannel overlay IPs. This breaks
external connectivity because those IPs are not routable from outside.
With externalTrafficPolicy: Local, traffic is only routed to pods on the
same node, and k3s assigns the node's actual public IP to the LoadBalancer.
This was the missing piece from the reliability hardening in commit b0f2acf.
Without this in git, any manual kubectl patch is reverted by ArgoCD sync.
Evidence: stalwart-smtp and stalwart-imap both showing LoadBalancer IPs:
10.42.1.1, 10.42.1.2, 10.42.1.3, 10.42.1.5 (internal flannel IPs)
Related: DEV-230, DEV-231, DEV-233, DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: k3s ServiceLB assigns internal VIPs (10.42.1.x) that are not
publicly routable. External traffic to mail.basicstack.de cannot reach the
SMTP/IMAP services.
Investigation shows:
- Web-UI works (goes through Traefik IngressRoute)
- SMTP/IMAP ports have correct firewall rules
- iptables DNAT rules exist but don't help external traffic
- Internal connectivity works correctly
- LoadBalancer 'external IPs' are actually pod network IPs
Three solution options documented:
A. NodePort + iptables REDIRECT (recommended)
B. Deploy MetalLB for true LoadBalancer IPs
C. Hetzner Cloud LoadBalancers (not recommended, cost)
Awaiting approval on approach before implementing fix.
Related: DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This document establishes operational rules and diagnostic playbooks for
Stalwart stability monitoring and incident response.
Key sections:
- Critical operational rules (repository as single point of truth)
- Requirement to read git history before any investigation
- Current 7-day stability check procedures (DEV-235)
- Known issues and their fixes (k3s networking, commit b0f2acf)
- Diagnostic playbook for SMTP/IMAP/Web-UI issues
- Issue documentation template
- Evidence requirements for verification completion
This ensures all future investigations follow documented procedures and
avoid repeated debugging cycles.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move all Application manifests (app-*.yaml) out of apps/argocd/ to apps/
to avoid chicken-and-egg issue where Applications couldn't update themselves.
Architecture:
- apps/app-stack-basicstack-de.yaml: manages apps/** excluding argocd/**
- apps/app-argocd.yaml: manages apps/argocd/** via kustomize
- apps/app-basicstack-org.yaml: manages basicstack.org repo
This enables full self-management: all Applications can sync their own
configurations from git.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Implement scale-down/backup/scale-up pattern to work around PVC access mode limitation.
Changes:
- Add RBAC (ServiceAccount, Role, RoleBinding) with statefulsets/scale and pods permissions
- Switch to alpine:3.19 base image with kubectl and restic
- Scale down StatefulSet to 0 replicas before backup
- Run restic backup while pod is stopped
- Scale back up to 1 replica with error handling
- Add cleanup trap to ensure scale-up even on failure
- Set 10-minute timeout and backoff limit
Tested successfully: backup completes in ~32 seconds with minimal downtime.
Resolves DEV-236.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.
Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures
Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service
Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.
Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added Stakater Reloader to automatically restart Stalwart pods when
TLS certificates are renewed by cert-manager. This ensures seamless
certificate rotation without manual intervention.
Changes:
- Deploy Stakater Reloader in infrastructure/networking/
- Add Reloader annotation to Stalwart StatefulSet to watch stalwart-tls secret
- Document certificate renewal process and troubleshooting
The certificate is managed by cert-manager with Let's Encrypt and will
automatically renew 30 days before expiration (renewal date: 2026-08-20).
Reloader detects secret updates and triggers a rolling restart of the
Stalwart StatefulSet to load the new certificate.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Removed stalwart-config ConfigMap (config.toml not used in v0.16.11)
- All configuration is done via Stalwart API and stored in RocksDB
- Added TLS certificate mount from stalwart-tls secret
- Using stalwartlabs/stalwart:v0.16.11
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Fixed init container to set ownership to 2000:2000
- Stalwart container runs as user 2000 by default
- Resolves permission denied error during web UI setup
Refs: DEV-206
Co-Authored-By: Paperclip <noreply@paperclip.ing>
