# Stalwart Stability Check Procedure ## CRITICAL OPERATIONAL RULES **READ THIS FIRST BEFORE ANY INVESTIGATION OR FIX** ### Single Point of Truth This repository (`https://forgejo.basicstack.de/basicstack/stack.basicstack.de`) is the **single point of truth** for all Stalwart configuration. - ✅ **DO**: Make all changes in this repository - ✅ **DO**: Sync changes via Argo CD - ❌ **NEVER**: Make direct kubectl changes without updating this repository first - ❌ **NEVER**: Reset to recovery/default admin state as a "fix" (destroys working OIDC config) ### Before Any Investigation 1. **Read the git history**: ```bash git log --oneline --all -- apps/stalwart/ | head -20 git show # for details on relevant fixes ``` 2. **Check recent changes**: ```bash git log --since="7 days ago" -- apps/stalwart/ ``` 3. **Read this file** to understand what's been tried and what's known 4. **Document your findings** in this repository under issues, not just in task comments ### Scope Discipline - **Web-UI working but SMTP/IMAP broken** = Network/port issue, NOT auth/config issue - **Web-UI login broken** = Auth/config issue, NOT network issue - Do NOT "fix" the auth layer when the problem is transport/network, and vice-versa ### Known Issues and History #### Issue #1: k3s Service Networking Instability (Root Cause - FIXED 2026-07-11) **Commit**: `b0f2acf` - "Stalwart reliability hardening: fix k3s service networking issues" **Symptom**: Pod healthy but service ClusterIP routing breaks intermittently **Root Cause**: k3s uses iptables-legacy, not iptables-nft (see memory file: k3s-iptables-backend.md) - Service ClusterIP routing fails despite correct iptables rules and healthy endpoints - Direct pod IP works, service ClusterIP fails with "Connection reset by peer" - Required pod restart + Traefik restart to restore service **Fix Applied**: - Added startup probe (60s delay) to prevent premature service registration - Added comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes) - External health checks test full service path, not just pod health - Alerting for: service failures, high latency, pod restarts, backup failures **Verification**: 7-day stability window to confirm fix (2026-07-11 11:04 UTC → 2026-07-18 11:04 UTC) --- ## Current Stability Check (DEV-300) **Original Window (DEV-235)**: 2026-07-11 11:04 UTC → 2026-07-18 11:04 UTC **Issue**: LoadBalancer used incorrect IP (10.42.1.1 instead of 178.105.17.239) until 2026-07-13 19:10 UTC **Status**: Window restart pending board decision (as of 2026-07-13 20:37 UTC) **Proposed New Window**: 2026-07-13 20:37 UTC → 2026-07-20 20:37 UTC **Rationale**: Clean 7-day verification with working external access ### Investigation Results (2026-07-13) **Infrastructure Status**: ✅ ALL VERIFIED WORKING - SMTP (ports 25, 587, 465): Responding with correct ESMTP greeting - IMAP (ports 143, 993): Responding with correct IMAP greeting - LoadBalancer external IP: 178.105.17.239 (correct) - Pod status: Running, 0 restarts - Monitoring: PrometheusRules and ServiceMonitors in place **User-Reported SMTP Issue**: Client configuration problem (Thunderbird connecting to port 143 instead of 587), not infrastructure issue ### Architecture Overview ``` External Traffic ↓ Hetzner LoadBalancer (SMTP: 25,587,465 / IMAP: 143,993) ↓ Kubernetes Service (LoadBalancer type) ↓ stalwart-0 Pod (StatefulSet) ↓ RocksDB Storage (PVC: hcloud-volumes-encrypted) Web Traffic ↓ Traefik IngressRoute (mail.basicstack.de) ↓ stalwart-http Service (ClusterIP) ↓ stalwart-0 Pod (port 8080) ``` ### Passive Monitoring Checks Run these after the 7-day window completes (2026-07-18 11:04 UTC): 1. **No service routing failures**: Verify no manual restarts were needed 2. **Low pod restart count**: ```bash ssh root@178.105.17.239 'kubectl get pods -n stalwart' # RESTARTS column should be 0 or very low ``` 3. **No Prometheus alerts fired**: Check alert history for: - `StalwartDown` - `StalwartExternalProbeFailure` - `StalwartHighResponseTime` - `StalwartPodRestarting` - `StalwartBackupFailing` ### Active Tests Run these before closing the verification: #### 1. Pod Restart Test ```bash # Delete the pod ssh root@178.105.17.239 'kubectl delete pod stalwart-0 -n stalwart' # Wait and verify it recovers within 3 minutes ssh root@178.105.17.239 'kubectl get pods -n stalwart -w' ``` #### 2. Node Drain Test ```bash # Drain the worker node ssh root@178.105.17.239 'kubectl drain k3s-worker-1 --ignore-daemonsets --delete-emptydir-data' # Verify StatefulSet reschedules ssh root@178.105.17.239 'kubectl get pods -n stalwart -o wide' # Service should come back on another node # Uncordon after test ssh root@178.105.17.239 'kubectl uncordon k3s-worker-1' ``` #### 3. Alert Delivery Test ```bash # Temporarily stop Stalwart ssh root@178.105.17.239 'kubectl scale statefulset stalwart -n stalwart --replicas=0' # Wait 3-5 minutes for alert to fire # Verify alert was delivered to the monitoring channel # Scale back up ssh root@178.105.17.239 'kubectl scale statefulset stalwart -n stalwart --replicas=1' ``` --- ## Diagnostic Playbook ### Check Pod Health ```bash ssh root@178.105.17.239 'kubectl get pods -n stalwart' ssh root@178.105.17.239 'kubectl describe pod stalwart-0 -n stalwart' ssh root@178.105.17.239 'kubectl logs stalwart-0 -n stalwart --tail=100' ``` ### Check Service and Endpoints ```bash ssh root@178.105.17.239 'kubectl get svc -n stalwart' ssh root@178.105.17.239 'kubectl get endpoints -n stalwart' ssh root@178.105.17.239 'kubectl describe svc stalwart-smtp -n stalwart' ssh root@178.105.17.239 'kubectl describe svc stalwart-imap -n stalwart' ``` ### Check LoadBalancer External IPs ```bash # SMTP LoadBalancer ssh root@178.105.17.239 'kubectl get svc stalwart-smtp -n stalwart -o jsonpath="{.status.loadBalancer.ingress[0].ip}"' # IMAP LoadBalancer ssh root@178.105.17.239 'kubectl get svc stalwart-imap -n stalwart -o jsonpath="{.status.loadBalancer.ingress[0].ip}"' ``` ### Test SMTP/IMAP Connectivity #### From outside the cluster ```bash # SMTP telnet 25 telnet 587 # IMAP telnet 143 telnet 993 ``` #### From inside the cluster ```bash ssh root@178.105.17.239 'kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- bash' # Then inside the pod: telnet stalwart-smtp.stalwart.svc.cluster.local 25 telnet stalwart-imap.stalwart.svc.cluster.local 143 ``` ### Check Firewall Rules (Hetzner Cloud) ```bash # List firewall rules hcloud firewall list hcloud firewall describe # Verify ports 25, 587, 465, 143, 993 are open to the LoadBalancer nodes ``` ### Check iptables (k3s nodes) ```bash # On the node running the stalwart pod ssh root@ 'iptables-legacy -L -n -v | grep -A 20 stalwart' ssh root@ 'iptables-legacy -t nat -L -n -v | grep -A 20 stalwart' ``` ### Check Network Policies ```bash ssh root@178.105.17.239 'kubectl get networkpolicies -n stalwart' ssh root@178.105.17.239 'kubectl describe networkpolicy -n stalwart' ``` --- ## Issue Documentation Template When you find an issue, create a file in this directory: **Filename**: `ISSUE-YYYY-MM-DD-.md` **Content**: ```markdown # Issue: **Date**: YYYY-MM-DD HH:MM UTC **Discovered by**: **Related Task**: DEV-XXX ## Symptom What users are experiencing ## Investigation Steps taken to diagnose ## Root Cause What was actually broken ## Fix Applied Changes made (commit hash if available) ## Verification How to verify the fix worked ## Follow-up Any remaining concerns or monitoring needed ``` --- ## Evidence Required for Completion Before marking DEV-235 as done, document: 1. Screenshot or log excerpt showing 7 days of uptime (pod age, restart count) 2. Pod restart test: before/after pod status with timestamps 3. Node drain test: before/after with timestamps showing pod migration 4. Alert delivery: confirmation that alert fired and was received 5. Summary: Any issues found during the 7-day window and how they were resolved --- ## Related Documentation - [README.md](README.md) - Main Stalwart deployment documentation - [CERTIFICATE-RENEWAL.md](CERTIFICATE-RENEWAL.md) - TLS certificate automation - [manual_config_steps.md](manual_config_steps.md) - Manual configuration procedures - Memory files: - `cluster-infrastructure.md` - Cluster layout and node IPs - `k3s-iptables-backend.md` - k3s uses iptables-legacy (root cause of past outages) --- ## Emergency Contacts **DO NOT** take destructive actions without approval: - Resetting to recovery admin - Dropping the RocksDB database - Force-deleting the PVC **Instead**: Post findings to the issue, request approval, and wait for explicit confirmation before proceeding.