basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Add comprehensive Stalwart stability check procedure

Browse source 
This document establishes operational rules and diagnostic playbooks for
Stalwart stability monitoring and incident response.

Key sections:
- Critical operational rules (repository as single point of truth)
- Requirement to read git history before any investigation
- Current 7-day stability check procedures (DEV-235)
- Known issues and their fixes (k3s networking, commit b0f2acf)
- Diagnostic playbook for SMTP/IMAP/Web-UI issues
- Issue documentation template
- Evidence requirements for verification completion

This ensures all future investigations follow documented procedures and
avoid repeated debugging cycles.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-07-13 19:56:35 +00:00
parent c429df27ea
commit c84411d99f
1 changed files with 283 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

283
apps/stalwart/STABILITY-CHECK.md Normal file
View file

@ -0,0 +1,283 @@
# Stalwart Stability Check Procedure
## CRITICAL OPERATIONAL RULES
**READ THIS FIRST BEFORE ANY INVESTIGATION OR FIX**
### Single Point of Truth
This repository (`https://forgejo.basicstack.de/basicstack/stack.basicstack.de`) is the **single point of truth** for all Stalwart configuration.
- ✅ **DO**: Make all changes in this repository
- ✅ **DO**: Sync changes via Argo CD
- ❌ **NEVER**: Make direct kubectl changes without updating this repository first
- ❌ **NEVER**: Reset to recovery/default admin state as a "fix" (destroys working OIDC config)
### Before Any Investigation
1. **Read the git history**:
```bash
git log --oneline --all -- apps/stalwart/ | head -20
git show <commit-hash> # for details on relevant fixes
```
2. **Check recent changes**:
```bash
git log --since="7 days ago" -- apps/stalwart/
```
3. **Read this file** to understand what's been tried and what's known
4. **Document your findings** in this repository under issues, not just in task comments
### Scope Discipline
- **Web-UI working but SMTP/IMAP broken** = Network/port issue, NOT auth/config issue
- **Web-UI login broken** = Auth/config issue, NOT network issue
- Do NOT "fix" the auth layer when the problem is transport/network, and vice-versa
### Known Issues and History
#### Issue #1: k3s Service Networking Instability (Root Cause - FIXED 2026-07-11)
**Commit**: `b0f2acf` - "Stalwart reliability hardening: fix k3s service networking issues"
**Symptom**: Pod healthy but service ClusterIP routing breaks intermittently
**Root Cause**: k3s uses iptables-legacy, not iptables-nft (see memory file: k3s-iptables-backend.md)
- Service ClusterIP routing fails despite correct iptables rules and healthy endpoints
- Direct pod IP works, service ClusterIP fails with "Connection reset by peer"
- Required pod restart + Traefik restart to restore service
**Fix Applied**:
- Added startup probe (60s delay) to prevent premature service registration
- Added comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- External health checks test full service path, not just pod health
- Alerting for: service failures, high latency, pod restarts, backup failures
**Verification**: 7-day stability window to confirm fix (2026-07-11 11:04 UTC → 2026-07-18 11:04 UTC)
---
## Current Stability Check (DEV-235)
**Start**: 2026-07-11 11:04 UTC
**End**: 2026-07-18 11:04 UTC
**Status**: In progress (as of 2026-07-13)
### Architecture Overview
```
External Traffic
↓
Hetzner LoadBalancer (SMTP: 25,587,465 / IMAP: 143,993)
↓
Kubernetes Service (LoadBalancer type)
↓
stalwart-0 Pod (StatefulSet)
↓
RocksDB Storage (PVC: hcloud-volumes-encrypted)
Web Traffic
↓
Traefik IngressRoute (mail.basicstack.de)
↓
stalwart-http Service (ClusterIP)
↓
stalwart-0 Pod (port 8080)
```
### Passive Monitoring Checks
Run these after the 7-day window completes (2026-07-18 11:04 UTC):
1. **No service routing failures**: Verify no manual restarts were needed
2. **Low pod restart count**:
```bash
ssh root@178.105.17.239 'kubectl get pods -n stalwart'
# RESTARTS column should be 0 or very low
```
3. **No Prometheus alerts fired**: Check alert history for:
- `StalwartDown`
- `StalwartExternalProbeFailure`
- `StalwartHighResponseTime`
- `StalwartPodRestarting`
- `StalwartBackupFailing`
### Active Tests
Run these before closing the verification:
#### 1. Pod Restart Test
```bash
# Delete the pod
ssh root@178.105.17.239 'kubectl delete pod stalwart-0 -n stalwart'
# Wait and verify it recovers within 3 minutes
ssh root@178.105.17.239 'kubectl get pods -n stalwart -w'
```
#### 2. Node Drain Test
```bash
# Drain the worker node
ssh root@178.105.17.239 'kubectl drain k3s-worker-1 --ignore-daemonsets --delete-emptydir-data'
# Verify StatefulSet reschedules
ssh root@178.105.17.239 'kubectl get pods -n stalwart -o wide'
# Service should come back on another node
# Uncordon after test
ssh root@178.105.17.239 'kubectl uncordon k3s-worker-1'
```
#### 3. Alert Delivery Test
```bash
# Temporarily stop Stalwart
ssh root@178.105.17.239 'kubectl scale statefulset stalwart -n stalwart --replicas=0'
# Wait 3-5 minutes for alert to fire
# Verify alert was delivered to the monitoring channel
# Scale back up
ssh root@178.105.17.239 'kubectl scale statefulset stalwart -n stalwart --replicas=1'
```
---
## Diagnostic Playbook
### Check Pod Health
```bash
ssh root@178.105.17.239 'kubectl get pods -n stalwart'
ssh root@178.105.17.239 'kubectl describe pod stalwart-0 -n stalwart'
ssh root@178.105.17.239 'kubectl logs stalwart-0 -n stalwart --tail=100'
```
### Check Service and Endpoints
```bash
ssh root@178.105.17.239 'kubectl get svc -n stalwart'
ssh root@178.105.17.239 'kubectl get endpoints -n stalwart'
ssh root@178.105.17.239 'kubectl describe svc stalwart-smtp -n stalwart'
ssh root@178.105.17.239 'kubectl describe svc stalwart-imap -n stalwart'
```
### Check LoadBalancer External IPs
```bash
# SMTP LoadBalancer
ssh root@178.105.17.239 'kubectl get svc stalwart-smtp -n stalwart -o jsonpath="{.status.loadBalancer.ingress[0].ip}"'
# IMAP LoadBalancer
ssh root@178.105.17.239 'kubectl get svc stalwart-imap -n stalwart -o jsonpath="{.status.loadBalancer.ingress[0].ip}"'
```
### Test SMTP/IMAP Connectivity
#### From outside the cluster
```bash
# SMTP
telnet <smtp-lb-ip> 25
telnet <smtp-lb-ip> 587
# IMAP
telnet <imap-lb-ip> 143
telnet <imap-lb-ip> 993
```
#### From inside the cluster
```bash
ssh root@178.105.17.239 'kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- bash'
# Then inside the pod:
telnet stalwart-smtp.stalwart.svc.cluster.local 25
telnet stalwart-imap.stalwart.svc.cluster.local 143
```
### Check Firewall Rules (Hetzner Cloud)
```bash
# List firewall rules
hcloud firewall list
hcloud firewall describe <firewall-id>
# Verify ports 25, 587, 465, 143, 993 are open to the LoadBalancer nodes
```
### Check iptables (k3s nodes)
```bash
# On the node running the stalwart pod
ssh root@<node-ip> 'iptables-legacy -L -n -v | grep -A 20 stalwart'
ssh root@<node-ip> 'iptables-legacy -t nat -L -n -v | grep -A 20 stalwart'
```
### Check Network Policies
```bash
ssh root@178.105.17.239 'kubectl get networkpolicies -n stalwart'
ssh root@178.105.17.239 'kubectl describe networkpolicy -n stalwart'
```
---
## Issue Documentation Template
When you find an issue, create a file in this directory:
**Filename**: `ISSUE-YYYY-MM-DD-<short-description>.md`
**Content**:
```markdown
# Issue: <Short Description>
**Date**: YYYY-MM-DD HH:MM UTC
**Discovered by**: <Name/Agent>
**Related Task**: DEV-XXX
## Symptom
What users are experiencing
## Investigation
Steps taken to diagnose
## Root Cause
What was actually broken
## Fix Applied
Changes made (commit hash if available)
## Verification
How to verify the fix worked
## Follow-up
Any remaining concerns or monitoring needed
```
---
## Evidence Required for Completion
Before marking DEV-235 as done, document:
1. Screenshot or log excerpt showing 7 days of uptime (pod age, restart count)
2. Pod restart test: before/after pod status with timestamps
3. Node drain test: before/after with timestamps showing pod migration
4. Alert delivery: confirmation that alert fired and was received
5. Summary: Any issues found during the 7-day window and how they were resolved
---
## Related Documentation
- [README.md](README.md) - Main Stalwart deployment documentation
- [CERTIFICATE-RENEWAL.md](CERTIFICATE-RENEWAL.md) - TLS certificate automation
- [manual_config_steps.md](manual_config_steps.md) - Manual configuration procedures
- Memory files:
- `cluster-infrastructure.md` - Cluster layout and node IPs
- `k3s-iptables-backend.md` - k3s uses iptables-legacy (root cause of past outages)
---
## Emergency Contacts
**DO NOT** take destructive actions without approval:
- Resetting to recovery admin
- Dropping the RocksDB database
- Force-deleting the PVC
**Instead**: Post findings to the issue, request approval, and wait for explicit confirmation before proceeding.