basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/apps/stalwart
History
CTO Agent b0f2acf5f3 Stalwart reliability hardening: fix k3s service networking issues 
Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.

Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures

Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service

Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.

Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-07-11 11:06:36 +00:00
..
CERTIFICATE-RENEWAL.md Implement automatic TLS certificate renewal for Stalwart 2026-07-04 16:48:48 +00:00
manual_config_steps.md apps/stalwart/manual_config_steps.md aktualisiert 2026-07-04 16:39:11 +00:00
README.md apps/stalwart/README.md aktualisiert 2026-07-04 17:19:55 +00:00
stalwart-admin-credentials-sealed.yaml Convert all secrets to SealedSecrets for enhanced security 2026-07-01 18:38:27 +00:00
stalwart-fresh-deployment.yaml Stalwart reliability hardening: fix k3s service networking issues 2026-07-11 11:06:36 +00:00
stalwart-monitoring.yaml Stalwart reliability hardening: fix k3s service networking issues 2026-07-11 11:06:36 +00:00
stalwart-s3-backup-sealed.yaml Convert all secrets to SealedSecrets for enhanced security 2026-07-01 18:38:27 +00:00

README.md

Stalwart Mail Server v0.16.11

Clean deployment of Stalwart mail server with username/password authentication only.

Architecture

  • Version: v0.16.11
  • Authentication: Username/password only (NO OAuth/OIDC)
  • Configuration: API-based (stored in RocksDB)
  • Storage: Encrypted hcloud-volumes (20Gi)
  • Backup: Daily restic backup to S3 at 3 AM
  • Web UI: https://mail.basicstack.de

Files

  • stalwart-fresh-deployment.yaml - Main deployment manifest
  • stalwart-admin-credentials-sealed.yaml - Sealed secret for admin password
  • stalwart-s3-backup-sealed.yaml - Sealed secret for S3 backup credentials

Deployment

# Apply sealed secrets first
kubectl apply -f stalwart-admin-credentials-sealed.yaml
kubectl apply -f stalwart-s3-backup-sealed.yaml

# Create bootstrap config
kubectl create configmap stalwart-bootstrap-config \
  --from-literal=config.json='{"@type":"RocksDb","path":"/var/lib/stalwart"}' \
  -n stalwart

# Deploy Stalwart
kubectl apply -f stalwart-fresh-deployment.yaml

Initial Admin Login

After deployment, log in at https://mail.basicstack.de with:

  • Username: admin
  • Password: (from stalwart-admin-credentials secret)

Configuration

All configuration is done via the web UI or API. The bootstrap config only points to the RocksDB database location. NO config.toml files are used.

Check https://stalw.art/docs/ref/ for configuration possibilities. The API access via /jmap seems to be too complex for the agent and the configruation has been done manually.

Refer to manual configuration steps

Ports

  • SMTP: 25, 587, 465
  • IMAP: 143, 993
  • HTTP: 8080 (web UI)

Storage

Data is stored in /var/lib/stalwart using the RocksDB database format. This includes:

  • Email messages
  • User accounts
  • Server configuration
  • TLS certificates configuration

Certificate Renewal

Refer to Automatic Renewal TLS Certificate