stack.basicstack.de/apps/stalwart/CERTIFICATE-RENEWAL.md

# Stalwart TLS Certificate Automatic Renewal

## Overview

The Stalwart mail server uses TLS certificates for secure email communications (SMTP, IMAP, and HTTPS). These certificates are automatically managed and renewed by cert-manager and reloaded by Stakater Reloader.

## Components

### 1. cert-manager

cert-manager is a Kubernetes add-on that automates the management and issuance of TLS certificates from various issuing sources, including Let's Encrypt.

**Configuration:**
- **ClusterIssuer:** `letsencrypt-prod` (configured in `infrastructure/networking/`)
- **Email:** admin@basicstack.de
- **ACME Server:** Let's Encrypt production (https://acme-v02.api.letsencrypt.org/directory)
- **Challenge Type:** HTTP-01 (via Traefik ingress)

**Certificate Resource:**
```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: stalwart-tls
  namespace: stalwart
spec:
  dnsNames:
  - mail.basicstack.de
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: stalwart-tls
```

The Certificate resource is automatically created by cert-manager when it sees the Ingress annotation:
```yaml
annotations:
  cert-manager.io/cluster-issuer: "letsencrypt-prod"
```

### 2. Automatic Renewal Schedule

cert-manager automatically renews certificates **30 days before expiration**. For a Let's Encrypt certificate valid for 90 days:

- **Issue Date:** 2026-06-21
- **Expiration Date:** 2026-09-19 (90 days)
- **Renewal Date:** 2026-08-20 (30 days before expiration)

You can verify the renewal schedule with:
```bash
kubectl get certificate stalwart-tls -n stalwart -o yaml | grep renewalTime
```

### 3. Stakater Reloader

When cert-manager renews a certificate, it updates the `stalwart-tls` Kubernetes Secret with the new certificate and private key. However, the Stalwart pod doesn't automatically detect this change and continues using the old certificate loaded at startup.

**Stakater Reloader** solves this by watching Secrets and ConfigMaps for changes and automatically triggering a rolling restart of the associated pods.

**Deployment:** `infrastructure/networking/reloader.yaml`

**How it works:**
1. Reloader watches the `stalwart-tls` secret for changes
2. When cert-manager updates the secret with a renewed certificate, Reloader detects the change
3. Reloader triggers a rolling restart of the Stalwart StatefulSet
4. The new pod loads the updated certificate from the secret

**Annotation in stalwart-fresh-deployment.yaml:**
```yaml
metadata:
  annotations:
    secret.reloader.stakater.com/reload: "stalwart-tls"
```

## Verification

### Check Certificate Status
```bash
# View certificate details
kubectl get certificate stalwart-tls -n stalwart

# Check expiration and renewal time
kubectl describe certificate stalwart-tls -n stalwart

# View the actual certificate from the secret
kubectl get secret stalwart-tls -n stalwart -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text
```

### Check Reloader Status
```bash
# Check if Reloader is running
kubectl get deployment reloader -n reloader

# View Reloader logs
kubectl logs -n reloader -l app=reloader

# Check for reload events
kubectl get events -n stalwart --sort-by='.lastTimestamp' | grep -i reload
```

### Monitor Renewal Process

cert-manager creates temporary Pods and Ingress resources during certificate renewal. You can monitor this process:

```bash
# Watch cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager -f

# Check for ACME challenge resources
kubectl get challenges -n stalwart

# Check certificate renewal history
kubectl describe certificate stalwart-tls -n stalwart | tail -20
```

## Troubleshooting

### Certificate Not Renewing

1. **Check cert-manager is running:**
   ```bash
   kubectl get pods -n cert-manager
   ```

2. **Check ClusterIssuer status:**
   ```bash
   kubectl get clusterissuer letsencrypt-prod
   ```

3. **Check Certificate resource events:**
   ```bash
   kubectl describe certificate stalwart-tls -n stalwart
   ```

4. **Check cert-manager logs:**
   ```bash
   kubectl logs -n cert-manager -l app=cert-manager --tail=100
   ```

### Pod Not Reloading After Renewal

1. **Check Reloader is running:**
   ```bash
   kubectl get deployment reloader -n reloader
   ```

2. **Verify the annotation is present:**
   ```bash
   kubectl get statefulset stalwart -n stalwart -o yaml | grep reload
   ```

3. **Check Reloader logs:**
   ```bash
   kubectl logs -n reloader -l app=reloader --tail=100
   ```

4. **Manually trigger a reload (if needed):**
   ```bash
   kubectl rollout restart statefulset/stalwart -n stalwart
   ```

### Certificate Validation Issues

If cert-manager can't complete the ACME challenge:

1. **Ensure DNS is correct:**
   ```bash
   dig mail.basicstack.de
   ```

2. **Check Ingress is accessible:**
   ```bash
   curl -I http://mail.basicstack.de/.well-known/acme-challenge/test
   ```

3. **Verify Traefik is routing correctly:**
   ```bash
   kubectl logs -n kube-system -l app.kubernetes.io/name=traefik
   ```

## Manual Operations

### Force Certificate Renewal

If you need to force a renewal (e.g., certificate was compromised):

```bash
# Delete the Certificate resource (cert-manager will recreate it)
kubectl delete certificate stalwart-tls -n stalwart

# Or, trigger renewal by deleting the secret
kubectl delete secret stalwart-tls -n stalwart
```

cert-manager will automatically request a new certificate.

### Temporarily Disable Auto-Reload

If you need to prevent automatic restarts during maintenance:

```bash
# Scale down Reloader
kubectl scale deployment reloader -n reloader --replicas=0

# Later, scale it back up
kubectl scale deployment reloader -n reloader --replicas=1
```

## Security Considerations

1. **Private Key Protection:** The certificate private key is stored in the `stalwart-tls` Kubernetes Secret with restricted access
2. **ACME Account Key:** The Let's Encrypt account key is stored in `letsencrypt-prod-account-key` secret
3. **Rate Limits:** Let's Encrypt has rate limits (50 certificates per domain per week). Avoid unnecessary deletions.
4. **Certificate Revocation:** If a certificate is compromised, revoke it through Let's Encrypt and request a new one

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────┐
│                      Certificate Lifecycle                   │
└─────────────────────────────────────────────────────────────┘

1. Initial Request:
   Ingress → cert-manager → Let's Encrypt (ACME HTTP-01) → Secret

2. Automatic Renewal (30 days before expiry):
   cert-manager timer → Let's Encrypt → Updated Secret

3. Automatic Pod Reload:
   Reloader (watching Secret) → Rolling restart → Stalwart loads new cert

┌──────────────┐      ┌──────────────┐      ┌──────────────┐
│              │      │              │      │              │
│ cert-manager │─────>│ stalwart-tls │<─────│   Reloader   │
│              │      │    Secret    │      │              │
└──────────────┘      └──────────────┘      └──────────────┘
       │                      │                      │
       │                      │                      │
       v                      v                      v
  Let's Encrypt          Mounted in           Triggers restart
                         Stalwart Pod         when secret changes
```

## References

- [cert-manager documentation](https://cert-manager.io/docs/)
- [Let's Encrypt ACME documentation](https://letsencrypt.org/docs/)
- [Stakater Reloader](https://github.com/stakater/Reloader)
- [Stalwart configuration](https://stalw.art/docs/configuration)