Root cause: k3s ServiceLB assigns internal VIPs (10.42.1.x) that are not
publicly routable. External traffic to mail.basicstack.de cannot reach the
SMTP/IMAP services.
Investigation shows:
- Web-UI works (goes through Traefik IngressRoute)
- SMTP/IMAP ports have correct firewall rules
- iptables DNAT rules exist but don't help external traffic
- Internal connectivity works correctly
- LoadBalancer 'external IPs' are actually pod network IPs
Three solution options documented:
A. NodePort + iptables REDIRECT (recommended)
B. Deploy MetalLB for true LoadBalancer IPs
C. Hetzner Cloud LoadBalancers (not recommended, cost)
Awaiting approval on approach before implementing fix.
Related: DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This document establishes operational rules and diagnostic playbooks for
Stalwart stability monitoring and incident response.
Key sections:
- Critical operational rules (repository as single point of truth)
- Requirement to read git history before any investigation
- Current 7-day stability check procedures (DEV-235)
- Known issues and their fixes (k3s networking, commit b0f2acf)
- Diagnostic playbook for SMTP/IMAP/Web-UI issues
- Issue documentation template
- Evidence requirements for verification completion
This ensures all future investigations follow documented procedures and
avoid repeated debugging cycles.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move all Application manifests (app-*.yaml) out of apps/argocd/ to apps/
to avoid chicken-and-egg issue where Applications couldn't update themselves.
Architecture:
- apps/app-stack-basicstack-de.yaml: manages apps/** excluding argocd/**
- apps/app-argocd.yaml: manages apps/argocd/** via kustomize
- apps/app-basicstack-org.yaml: manages basicstack.org repo
This enables full self-management: all Applications can sync their own
configurations from git.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Implement scale-down/backup/scale-up pattern to work around PVC access mode limitation.
Changes:
- Add RBAC (ServiceAccount, Role, RoleBinding) with statefulsets/scale and pods permissions
- Switch to alpine:3.19 base image with kubectl and restic
- Scale down StatefulSet to 0 replicas before backup
- Run restic backup while pod is stopped
- Scale back up to 1 replica with error handling
- Add cleanup trap to ensure scale-up even on failure
- Set 10-minute timeout and backoff limit
Tested successfully: backup completes in ~32 seconds with minimal downtime.
Resolves DEV-236.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.
Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures
Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service
Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.
Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added Stakater Reloader to automatically restart Stalwart pods when
TLS certificates are renewed by cert-manager. This ensures seamless
certificate rotation without manual intervention.
Changes:
- Deploy Stakater Reloader in infrastructure/networking/
- Add Reloader annotation to Stalwart StatefulSet to watch stalwart-tls secret
- Document certificate renewal process and troubleshooting
The certificate is managed by cert-manager with Let's Encrypt and will
automatically renew 30 days before expiration (renewal date: 2026-08-20).
Reloader detects secret updates and triggers a rolling restart of the
Stalwart StatefulSet to load the new certificate.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Removed stalwart-config ConfigMap (config.toml not used in v0.16.11)
- All configuration is done via Stalwart API and stored in RocksDB
- Added TLS certificate mount from stalwart-tls secret
- Using stalwartlabs/stalwart:v0.16.11
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Fixed init container to set ownership to 2000:2000
- Stalwart container runs as user 2000 by default
- Resolves permission denied error during web UI setup
Refs: DEV-206
Co-Authored-By: Paperclip <noreply@paperclip.ing>
