Stalwart reliability hardening: fix k3s service networking issues

Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.

Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures

Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service

Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.

Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/stalwart/stalwart-fresh-deployment.yaml

@@ -144,6 +144,14 @@ spec:
           limits:
             memory: "2Gi"
             cpu: "2000m"
+        startupProbe:
+          httpGet:
+            path: /
+            port: 8080
+          initialDelaySeconds: 60
+          periodSeconds: 10
+          timeoutSeconds: 5
+          failureThreshold: 6
         livenessProbe:
           httpGet:
             path: /
@@ -225,6 +233,10 @@ spec:
               set -e
               echo "Starting Stalwart backup at $(date)"
 
+              # Build restic repository URL from env vars (K8s doesn't expand $(VAR) in value fields)
+              export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/stalwart"
+              echo "Using repository: $RESTIC_REPOSITORY"
+
               # Initialize restic repo if needed
               restic snapshots || restic init
 
@@ -243,8 +255,6 @@ spec:
 
               echo "Backup completed successfully at $(date)"
             env:
-            - name: RESTIC_REPOSITORY
-              value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/stalwart"
             - name: RESTIC_PASSWORD
               valueFrom:
                 secretKeyRef:

apps/stalwart/stalwart-monitoring.yaml

@@ -0,0 +1,132 @@
+---
+# ServiceMonitor for Stalwart application metrics
+# Requires prometheus-operator to be installed
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: stalwart-metrics
+  namespace: stalwart
+  labels:
+    app: stalwart
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: stalwart
+  endpoints:
+  - port: http
+    interval: 30s
+    path: /metrics
+    scheme: http
+---
+# Blackbox exporter probe for external health check
+# Tests the full service path from outside the cluster
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-external-probe
+  namespace: stalwart
+  labels:
+    app: stalwart-probe
+spec:
+  type: ClusterIP
+  clusterIP: None  # Headless service for probe
+  selector:
+    app: stalwart-probe-dummy  # No pods, just for ServiceMonitor
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: stalwart-blackbox-external
+  namespace: stalwart
+  labels:
+    app: stalwart
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: stalwart-probe
+  endpoints:
+  - port: http
+    interval: 60s
+    scrapeTimeout: 30s
+    path: /probe
+    params:
+      module: [http_2xx]
+      target: ['https://mail.basicstack.de/']
+    relabelings:
+    - sourceLabels: [__address__]
+      targetLabel: __param_target
+    - sourceLabels: [__param_target]
+      targetLabel: instance
+    - targetLabel: __address__
+      replacement: prometheus-blackbox-exporter.observability.svc.cluster.local:9115
+---
+# PrometheusRule for alerting on Stalwart failures
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: stalwart-alerts
+  namespace: stalwart
+  labels:
+    app: stalwart
+    release: kube-prometheus-stack
+spec:
+  groups:
+  - name: stalwart.rules
+    interval: 30s
+    rules:
+    - alert: StalwartDown
+      expr: up{job="stalwart-metrics"} == 0
+      for: 2m
+      labels:
+        severity: critical
+        service: stalwart
+        component: mail
+      annotations:
+        summary: "Stalwart mail service is down"
+        description: "Stalwart pod in namespace {{ $labels.namespace }} has been unreachable for more than 2 minutes."
+
+    - alert: StalwartExternalProbeFailure
+      expr: probe_success{job="stalwart-blackbox-external"} == 0
+      for: 5m
+      labels:
+        severity: critical
+        service: stalwart
+        component: external-access
+      annotations:
+        summary: "Stalwart external endpoint unreachable"
+        description: "External probe to {{ $labels.instance }} has failed for more than 5 minutes. Users cannot access mail service."
+
+    - alert: StalwartHighResponseTime
+      expr: probe_http_duration_seconds{job="stalwart-blackbox-external"} > 5
+      for: 5m
+      labels:
+        severity: warning
+        service: stalwart
+        component: performance
+      annotations:
+        summary: "Stalwart responding slowly"
+        description: "Stalwart response time to {{ $labels.instance }} is {{ $value }}s (threshold: 5s)."
+
+    - alert: StalwartPodRestarting
+      expr: rate(kube_pod_container_status_restarts_total{namespace="stalwart",pod=~"stalwart-.*"}[15m]) > 0
+      for: 5m
+      labels:
+        severity: warning
+        service: stalwart
+        component: stability
+      annotations:
+        summary: "Stalwart pod restarting"
+        description: "Stalwart pod {{ $labels.pod }} has restarted {{ $value }} times in the last 15 minutes."
+
+    - alert: StalwartBackupFailing
+      expr: kube_job_status_failed{namespace="stalwart",job_name=~"stalwart-backup-.*"} > 0
+      for: 1h
+      labels:
+        severity: warning
+        service: stalwart
+        component: backup
+      annotations:
+        summary: "Stalwart backup job failing"
+        description: "Backup job {{ $labels.job_name }} has failed. Investigate backup configuration."