stack.basicstack.de/apps/stalwart/STABILITY-CHECK.md

# Stalwart Stability Check Procedure

## CRITICAL OPERATIONAL RULES

**READ THIS FIRST BEFORE ANY INVESTIGATION OR FIX**

### Single Point of Truth

This repository (`https://forgejo.basicstack.de/basicstack/stack.basicstack.de`) is the **single point of truth** for all Stalwart configuration.

- ✅ **DO**: Make all changes in this repository
- ✅ **DO**: Sync changes via Argo CD
- ❌ **NEVER**: Make direct kubectl changes without updating this repository first
- ❌ **NEVER**: Reset to recovery/default admin state as a "fix" (destroys working OIDC config)

### Before Any Investigation

1. **Read the git history**:
   ```bash
   git log --oneline --all -- apps/stalwart/ | head -20
   git show <commit-hash>  # for details on relevant fixes
   ```

2. **Check recent changes**:
   ```bash
   git log --since="7 days ago" -- apps/stalwart/
   ```

3. **Read this file** to understand what's been tried and what's known

4. **Document your findings** in this repository under issues, not just in task comments

### Scope Discipline

- **Web-UI working but SMTP/IMAP broken** = Network/port issue, NOT auth/config issue
- **Web-UI login broken** = Auth/config issue, NOT network issue
- Do NOT "fix" the auth layer when the problem is transport/network, and vice-versa

### Known Issues and History

#### Issue #1: k3s Service Networking Instability (Root Cause - FIXED 2026-07-11)

**Commit**: `b0f2acf` - "Stalwart reliability hardening: fix k3s service networking issues"

**Symptom**: Pod healthy but service ClusterIP routing breaks intermittently

**Root Cause**: k3s uses iptables-legacy, not iptables-nft (see memory file: k3s-iptables-backend.md)
- Service ClusterIP routing fails despite correct iptables rules and healthy endpoints
- Direct pod IP works, service ClusterIP fails with "Connection reset by peer"
- Required pod restart + Traefik restart to restore service

**Fix Applied**:
- Added startup probe (60s delay) to prevent premature service registration
- Added comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- External health checks test full service path, not just pod health
- Alerting for: service failures, high latency, pod restarts, backup failures

**Verification**: 7-day stability window to confirm fix (2026-07-11 11:04 UTC → 2026-07-18 11:04 UTC)

---

## Current Stability Check (DEV-300)

**Original Window (DEV-235)**: 2026-07-11 11:04 UTC → 2026-07-18 11:04 UTC  
**Issue**: LoadBalancer used incorrect IP (10.42.1.1 instead of 178.105.17.239) until 2026-07-13 19:10 UTC  
**Status**: Window restart pending board decision (as of 2026-07-13 20:37 UTC)

**Proposed New Window**: 2026-07-13 20:37 UTC → 2026-07-20 20:37 UTC  
**Rationale**: Clean 7-day verification with working external access

### Investigation Results (2026-07-13)

**Infrastructure Status**: ✅ ALL VERIFIED WORKING
- SMTP (ports 25, 587, 465): Responding with correct ESMTP greeting
- IMAP (ports 143, 993): Responding with correct IMAP greeting  
- LoadBalancer external IP: 178.105.17.239 (correct)
- Pod status: Running, 0 restarts
- Monitoring: PrometheusRules and ServiceMonitors in place

**User-Reported SMTP Issue**: Client configuration problem (Thunderbird connecting to port 143 instead of 587), not infrastructure issue

### Architecture Overview

```
External Traffic
    ↓
Hetzner LoadBalancer (SMTP: 25,587,465 / IMAP: 143,993)
    ↓
Kubernetes Service (LoadBalancer type)
    ↓
stalwart-0 Pod (StatefulSet)
    ↓
RocksDB Storage (PVC: hcloud-volumes-encrypted)

Web Traffic
    ↓
Traefik IngressRoute (mail.basicstack.de)
    ↓
stalwart-http Service (ClusterIP)
    ↓
stalwart-0 Pod (port 8080)
```

### Passive Monitoring Checks

Run these after the 7-day window completes (2026-07-18 11:04 UTC):

1. **No service routing failures**: Verify no manual restarts were needed
2. **Low pod restart count**:
   ```bash
   ssh root@178.105.17.239 'kubectl get pods -n stalwart'
   # RESTARTS column should be 0 or very low
   ```
3. **No Prometheus alerts fired**: Check alert history for:
   - `StalwartDown`
   - `StalwartExternalProbeFailure`
   - `StalwartHighResponseTime`
   - `StalwartPodRestarting`
   - `StalwartBackupFailing`

### Active Tests

Run these before closing the verification:

#### 1. Pod Restart Test
```bash
# Delete the pod
ssh root@178.105.17.239 'kubectl delete pod stalwart-0 -n stalwart'

# Wait and verify it recovers within 3 minutes
ssh root@178.105.17.239 'kubectl get pods -n stalwart -w'
```

#### 2. Node Drain Test
```bash
# Drain the worker node
ssh root@178.105.17.239 'kubectl drain k3s-worker-1 --ignore-daemonsets --delete-emptydir-data'

# Verify StatefulSet reschedules
ssh root@178.105.17.239 'kubectl get pods -n stalwart -o wide'

# Service should come back on another node
# Uncordon after test
ssh root@178.105.17.239 'kubectl uncordon k3s-worker-1'
```

#### 3. Alert Delivery Test
```bash
# Temporarily stop Stalwart
ssh root@178.105.17.239 'kubectl scale statefulset stalwart -n stalwart --replicas=0'

# Wait 3-5 minutes for alert to fire
# Verify alert was delivered to the monitoring channel

# Scale back up
ssh root@178.105.17.239 'kubectl scale statefulset stalwart -n stalwart --replicas=1'
```

---

## Diagnostic Playbook

### Check Pod Health
```bash
ssh root@178.105.17.239 'kubectl get pods -n stalwart'
ssh root@178.105.17.239 'kubectl describe pod stalwart-0 -n stalwart'
ssh root@178.105.17.239 'kubectl logs stalwart-0 -n stalwart --tail=100'
```

### Check Service and Endpoints
```bash
ssh root@178.105.17.239 'kubectl get svc -n stalwart'
ssh root@178.105.17.239 'kubectl get endpoints -n stalwart'
ssh root@178.105.17.239 'kubectl describe svc stalwart-smtp -n stalwart'
ssh root@178.105.17.239 'kubectl describe svc stalwart-imap -n stalwart'
```

### Check LoadBalancer External IPs
```bash
# SMTP LoadBalancer
ssh root@178.105.17.239 'kubectl get svc stalwart-smtp -n stalwart -o jsonpath="{.status.loadBalancer.ingress[0].ip}"'

# IMAP LoadBalancer
ssh root@178.105.17.239 'kubectl get svc stalwart-imap -n stalwart -o jsonpath="{.status.loadBalancer.ingress[0].ip}"'
```

### Test SMTP/IMAP Connectivity

#### From outside the cluster
```bash
# SMTP
telnet <smtp-lb-ip> 25
telnet <smtp-lb-ip> 587

# IMAP
telnet <imap-lb-ip> 143
telnet <imap-lb-ip> 993
```

#### From inside the cluster
```bash
ssh root@178.105.17.239 'kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- bash'
# Then inside the pod:
telnet stalwart-smtp.stalwart.svc.cluster.local 25
telnet stalwart-imap.stalwart.svc.cluster.local 143
```

### Check Firewall Rules (Hetzner Cloud)
```bash
# List firewall rules
hcloud firewall list
hcloud firewall describe <firewall-id>

# Verify ports 25, 587, 465, 143, 993 are open to the LoadBalancer nodes
```

### Check iptables (k3s nodes)
```bash
# On the node running the stalwart pod
ssh root@<node-ip> 'iptables-legacy -L -n -v | grep -A 20 stalwart'
ssh root@<node-ip> 'iptables-legacy -t nat -L -n -v | grep -A 20 stalwart'
```

### Check Network Policies
```bash
ssh root@178.105.17.239 'kubectl get networkpolicies -n stalwart'
ssh root@178.105.17.239 'kubectl describe networkpolicy -n stalwart'
```

---

## Issue Documentation Template

When you find an issue, create a file in this directory:

**Filename**: `ISSUE-YYYY-MM-DD-<short-description>.md`

**Content**:
```markdown
# Issue: <Short Description>

**Date**: YYYY-MM-DD HH:MM UTC
**Discovered by**: <Name/Agent>
**Related Task**: DEV-XXX

## Symptom
What users are experiencing

## Investigation
Steps taken to diagnose

## Root Cause
What was actually broken

## Fix Applied
Changes made (commit hash if available)

## Verification
How to verify the fix worked

## Follow-up
Any remaining concerns or monitoring needed
```

---

## Evidence Required for Completion

Before marking DEV-235 as done, document:

1. Screenshot or log excerpt showing 7 days of uptime (pod age, restart count)
2. Pod restart test: before/after pod status with timestamps
3. Node drain test: before/after with timestamps showing pod migration
4. Alert delivery: confirmation that alert fired and was received
5. Summary: Any issues found during the 7-day window and how they were resolved

---

## Related Documentation

- [README.md](README.md) - Main Stalwart deployment documentation
- [CERTIFICATE-RENEWAL.md](CERTIFICATE-RENEWAL.md) - TLS certificate automation
- [manual_config_steps.md](manual_config_steps.md) - Manual configuration procedures
- Memory files:
  - `cluster-infrastructure.md` - Cluster layout and node IPs
  - `k3s-iptables-backend.md` - k3s uses iptables-legacy (root cause of past outages)

---

## Emergency Contacts

**DO NOT** take destructive actions without approval:
- Resetting to recovery admin
- Dropping the RocksDB database
- Force-deleting the PVC

**Instead**: Post findings to the issue, request approval, and wait for explicit confirmation before proceeding.