From c84411d99f6ae91b4fd48803a6b2e93dc19050e5 Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Mon, 13 Jul 2026 19:56:35 +0000 Subject: [PATCH] Add comprehensive Stalwart stability check procedure This document establishes operational rules and diagnostic playbooks for Stalwart stability monitoring and incident response. Key sections: - Critical operational rules (repository as single point of truth) - Requirement to read git history before any investigation - Current 7-day stability check procedures (DEV-235) - Known issues and their fixes (k3s networking, commit b0f2acf) - Diagnostic playbook for SMTP/IMAP/Web-UI issues - Issue documentation template - Evidence requirements for verification completion This ensures all future investigations follow documented procedures and avoid repeated debugging cycles. Co-Authored-By: Paperclip --- apps/stalwart/STABILITY-CHECK.md | 283 +++++++++++++++++++++++++++++++ 1 file changed, 283 insertions(+) create mode 100644 apps/stalwart/STABILITY-CHECK.md diff --git a/apps/stalwart/STABILITY-CHECK.md b/apps/stalwart/STABILITY-CHECK.md new file mode 100644 index 0000000..bcbfe2a --- /dev/null +++ b/apps/stalwart/STABILITY-CHECK.md @@ -0,0 +1,283 @@ +# Stalwart Stability Check Procedure + +## CRITICAL OPERATIONAL RULES + +**READ THIS FIRST BEFORE ANY INVESTIGATION OR FIX** + +### Single Point of Truth + +This repository (`https://forgejo.basicstack.de/basicstack/stack.basicstack.de`) is the **single point of truth** for all Stalwart configuration. + +- ✅ **DO**: Make all changes in this repository +- ✅ **DO**: Sync changes via Argo CD +- ❌ **NEVER**: Make direct kubectl changes without updating this repository first +- ❌ **NEVER**: Reset to recovery/default admin state as a "fix" (destroys working OIDC config) + +### Before Any Investigation + +1. **Read the git history**: + ```bash + git log --oneline --all -- apps/stalwart/ | head -20 + git show # for details on relevant fixes + ``` + +2. **Check recent changes**: + ```bash + git log --since="7 days ago" -- apps/stalwart/ + ``` + +3. **Read this file** to understand what's been tried and what's known + +4. **Document your findings** in this repository under issues, not just in task comments + +### Scope Discipline + +- **Web-UI working but SMTP/IMAP broken** = Network/port issue, NOT auth/config issue +- **Web-UI login broken** = Auth/config issue, NOT network issue +- Do NOT "fix" the auth layer when the problem is transport/network, and vice-versa + +### Known Issues and History + +#### Issue #1: k3s Service Networking Instability (Root Cause - FIXED 2026-07-11) + +**Commit**: `b0f2acf` - "Stalwart reliability hardening: fix k3s service networking issues" + +**Symptom**: Pod healthy but service ClusterIP routing breaks intermittently + +**Root Cause**: k3s uses iptables-legacy, not iptables-nft (see memory file: k3s-iptables-backend.md) +- Service ClusterIP routing fails despite correct iptables rules and healthy endpoints +- Direct pod IP works, service ClusterIP fails with "Connection reset by peer" +- Required pod restart + Traefik restart to restore service + +**Fix Applied**: +- Added startup probe (60s delay) to prevent premature service registration +- Added comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes) +- External health checks test full service path, not just pod health +- Alerting for: service failures, high latency, pod restarts, backup failures + +**Verification**: 7-day stability window to confirm fix (2026-07-11 11:04 UTC → 2026-07-18 11:04 UTC) + +--- + +## Current Stability Check (DEV-235) + +**Start**: 2026-07-11 11:04 UTC +**End**: 2026-07-18 11:04 UTC +**Status**: In progress (as of 2026-07-13) + +### Architecture Overview + +``` +External Traffic + ↓ +Hetzner LoadBalancer (SMTP: 25,587,465 / IMAP: 143,993) + ↓ +Kubernetes Service (LoadBalancer type) + ↓ +stalwart-0 Pod (StatefulSet) + ↓ +RocksDB Storage (PVC: hcloud-volumes-encrypted) + +Web Traffic + ↓ +Traefik IngressRoute (mail.basicstack.de) + ↓ +stalwart-http Service (ClusterIP) + ↓ +stalwart-0 Pod (port 8080) +``` + +### Passive Monitoring Checks + +Run these after the 7-day window completes (2026-07-18 11:04 UTC): + +1. **No service routing failures**: Verify no manual restarts were needed +2. **Low pod restart count**: + ```bash + ssh root@178.105.17.239 'kubectl get pods -n stalwart' + # RESTARTS column should be 0 or very low + ``` +3. **No Prometheus alerts fired**: Check alert history for: + - `StalwartDown` + - `StalwartExternalProbeFailure` + - `StalwartHighResponseTime` + - `StalwartPodRestarting` + - `StalwartBackupFailing` + +### Active Tests + +Run these before closing the verification: + +#### 1. Pod Restart Test +```bash +# Delete the pod +ssh root@178.105.17.239 'kubectl delete pod stalwart-0 -n stalwart' + +# Wait and verify it recovers within 3 minutes +ssh root@178.105.17.239 'kubectl get pods -n stalwart -w' +``` + +#### 2. Node Drain Test +```bash +# Drain the worker node +ssh root@178.105.17.239 'kubectl drain k3s-worker-1 --ignore-daemonsets --delete-emptydir-data' + +# Verify StatefulSet reschedules +ssh root@178.105.17.239 'kubectl get pods -n stalwart -o wide' + +# Service should come back on another node +# Uncordon after test +ssh root@178.105.17.239 'kubectl uncordon k3s-worker-1' +``` + +#### 3. Alert Delivery Test +```bash +# Temporarily stop Stalwart +ssh root@178.105.17.239 'kubectl scale statefulset stalwart -n stalwart --replicas=0' + +# Wait 3-5 minutes for alert to fire +# Verify alert was delivered to the monitoring channel + +# Scale back up +ssh root@178.105.17.239 'kubectl scale statefulset stalwart -n stalwart --replicas=1' +``` + +--- + +## Diagnostic Playbook + +### Check Pod Health +```bash +ssh root@178.105.17.239 'kubectl get pods -n stalwart' +ssh root@178.105.17.239 'kubectl describe pod stalwart-0 -n stalwart' +ssh root@178.105.17.239 'kubectl logs stalwart-0 -n stalwart --tail=100' +``` + +### Check Service and Endpoints +```bash +ssh root@178.105.17.239 'kubectl get svc -n stalwart' +ssh root@178.105.17.239 'kubectl get endpoints -n stalwart' +ssh root@178.105.17.239 'kubectl describe svc stalwart-smtp -n stalwart' +ssh root@178.105.17.239 'kubectl describe svc stalwart-imap -n stalwart' +``` + +### Check LoadBalancer External IPs +```bash +# SMTP LoadBalancer +ssh root@178.105.17.239 'kubectl get svc stalwart-smtp -n stalwart -o jsonpath="{.status.loadBalancer.ingress[0].ip}"' + +# IMAP LoadBalancer +ssh root@178.105.17.239 'kubectl get svc stalwart-imap -n stalwart -o jsonpath="{.status.loadBalancer.ingress[0].ip}"' +``` + +### Test SMTP/IMAP Connectivity + +#### From outside the cluster +```bash +# SMTP +telnet 25 +telnet 587 + +# IMAP +telnet 143 +telnet 993 +``` + +#### From inside the cluster +```bash +ssh root@178.105.17.239 'kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- bash' +# Then inside the pod: +telnet stalwart-smtp.stalwart.svc.cluster.local 25 +telnet stalwart-imap.stalwart.svc.cluster.local 143 +``` + +### Check Firewall Rules (Hetzner Cloud) +```bash +# List firewall rules +hcloud firewall list +hcloud firewall describe + +# Verify ports 25, 587, 465, 143, 993 are open to the LoadBalancer nodes +``` + +### Check iptables (k3s nodes) +```bash +# On the node running the stalwart pod +ssh root@ 'iptables-legacy -L -n -v | grep -A 20 stalwart' +ssh root@ 'iptables-legacy -t nat -L -n -v | grep -A 20 stalwart' +``` + +### Check Network Policies +```bash +ssh root@178.105.17.239 'kubectl get networkpolicies -n stalwart' +ssh root@178.105.17.239 'kubectl describe networkpolicy -n stalwart' +``` + +--- + +## Issue Documentation Template + +When you find an issue, create a file in this directory: + +**Filename**: `ISSUE-YYYY-MM-DD-.md` + +**Content**: +```markdown +# Issue: + +**Date**: YYYY-MM-DD HH:MM UTC +**Discovered by**: +**Related Task**: DEV-XXX + +## Symptom +What users are experiencing + +## Investigation +Steps taken to diagnose + +## Root Cause +What was actually broken + +## Fix Applied +Changes made (commit hash if available) + +## Verification +How to verify the fix worked + +## Follow-up +Any remaining concerns or monitoring needed +``` + +--- + +## Evidence Required for Completion + +Before marking DEV-235 as done, document: + +1. Screenshot or log excerpt showing 7 days of uptime (pod age, restart count) +2. Pod restart test: before/after pod status with timestamps +3. Node drain test: before/after with timestamps showing pod migration +4. Alert delivery: confirmation that alert fired and was received +5. Summary: Any issues found during the 7-day window and how they were resolved + +--- + +## Related Documentation + +- [README.md](README.md) - Main Stalwart deployment documentation +- [CERTIFICATE-RENEWAL.md](CERTIFICATE-RENEWAL.md) - TLS certificate automation +- [manual_config_steps.md](manual_config_steps.md) - Manual configuration procedures +- Memory files: + - `cluster-infrastructure.md` - Cluster layout and node IPs + - `k3s-iptables-backend.md` - k3s uses iptables-legacy (root cause of past outages) + +--- + +## Emergency Contacts + +**DO NOT** take destructive actions without approval: +- Resetting to recovery admin +- Dropping the RocksDB database +- Force-deleting the PVC + +**Instead**: Post findings to the issue, request approval, and wait for explicit confirmation before proceeding.