Commit graph

3 commits

Author SHA1 Message Date
CTO Agent
b0f2acf5f3 Stalwart reliability hardening: fix k3s service networking issues
Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.

Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures

Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service

Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.

Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-07-11 11:06:36 +00:00
CTO Agent
afe05cc772 Clean up Stalwart deployment - remove all old configs and OIDC attempts
Removed all experimental files, patches, OIDC configs, and Helm values.
Keeping only the clean v0.16.11 deployment with username/password auth.

Files kept:
- stalwart-fresh-deployment.yaml (main manifest)
- stalwart-admin-credentials-sealed.yaml (admin password)
- stalwart-s3-backup-sealed.yaml (backup credentials)
- README.md (updated documentation)

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-07-04 13:22:36 +00:00
CTO Agent
c64f9ab0d0 Initialize CD/CI repository structure with Stalwart example
Set up the repository structure following GitOps principles:
- apps/ for application deployments (Stalwart as example)
- infrastructure/ for cluster-wide configs (networking, monitoring)
- docs/ for general documentation

Migrated complete Stalwart deployment configuration including:
- Multiple deployment variants (basic, OIDC-enabled)
- Helm values files
- Monitoring and dashboard configurations
- Operational documentation (backup/restore, bootstrap)
- Configuration patches and fixes

Added comprehensive README files at each level to guide future use.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-07-01 18:08:31 +00:00