Root cause: ArgoCD continuously reverts the LoadBalancer services to use
externalTrafficPolicy: Cluster (the k8s default), causing k3s to assign
internal flannel VXLAN IPs (10.42.1.x) instead of the node public IPs.
With externalTrafficPolicy: Cluster, traffic can be routed to any node,
and k3s's service controller assigns the flannel overlay IPs. This breaks
external connectivity because those IPs are not routable from outside.
With externalTrafficPolicy: Local, traffic is only routed to pods on the
same node, and k3s assigns the node's actual public IP to the LoadBalancer.
This was the missing piece from the reliability hardening in commit b0f2acf.
Without this in git, any manual kubectl patch is reverted by ArgoCD sync.
Evidence: stalwart-smtp and stalwart-imap both showing LoadBalancer IPs:
10.42.1.1, 10.42.1.2, 10.42.1.3, 10.42.1.5 (internal flannel IPs)
Related: DEV-230, DEV-231, DEV-233, DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Implement scale-down/backup/scale-up pattern to work around PVC access mode limitation.
Changes:
- Add RBAC (ServiceAccount, Role, RoleBinding) with statefulsets/scale and pods permissions
- Switch to alpine:3.19 base image with kubectl and restic
- Scale down StatefulSet to 0 replicas before backup
- Run restic backup while pod is stopped
- Scale back up to 1 replica with error handling
- Add cleanup trap to ensure scale-up even on failure
- Set 10-minute timeout and backoff limit
Tested successfully: backup completes in ~32 seconds with minimal downtime.
Resolves DEV-236.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.
Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures
Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service
Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.
Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added Stakater Reloader to automatically restart Stalwart pods when
TLS certificates are renewed by cert-manager. This ensures seamless
certificate rotation without manual intervention.
Changes:
- Deploy Stakater Reloader in infrastructure/networking/
- Add Reloader annotation to Stalwart StatefulSet to watch stalwart-tls secret
- Document certificate renewal process and troubleshooting
The certificate is managed by cert-manager with Let's Encrypt and will
automatically renew 30 days before expiration (renewal date: 2026-08-20).
Reloader detects secret updates and triggers a rolling restart of the
Stalwart StatefulSet to load the new certificate.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Removed stalwart-config ConfigMap (config.toml not used in v0.16.11)
- All configuration is done via Stalwart API and stored in RocksDB
- Added TLS certificate mount from stalwart-tls secret
- Using stalwartlabs/stalwart:v0.16.11
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Fixed init container to set ownership to 2000:2000
- Stalwart container runs as user 2000 by default
- Resolves permission denied error during web UI setup
Refs: DEV-206
Co-Authored-By: Paperclip <noreply@paperclip.ing>
