stack.basicstack.de/apps/stalwart/CERTIFICATE-RENEWAL.md

Stalwart TLS Certificate Automatic Renewal

Overview

The Stalwart mail server uses TLS certificates for secure email communications (SMTP, IMAP, and HTTPS). These certificates are automatically managed and renewed by cert-manager and reloaded by Stakater Reloader.

Components

1. cert-manager

cert-manager is a Kubernetes add-on that automates the management and issuance of TLS certificates from various issuing sources, including Let's Encrypt.

Configuration:

• ClusterIssuer: letsencrypt-prod (configured in infrastructure/networking/)
• Email: admin@basicstack.de
• ACME Server: Let's Encrypt production (https://acme-v02.api.letsencrypt.org/directory)
• Challenge Type: HTTP-01 (via Traefik ingress)

Certificate Resource:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: stalwart-tls
  namespace: stalwart
spec:
  dnsNames:
  - mail.basicstack.de
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: stalwart-tls

The Certificate resource is automatically created by cert-manager when it sees the Ingress annotation:

annotations:
  cert-manager.io/cluster-issuer: "letsencrypt-prod"

2. Automatic Renewal Schedule

cert-manager automatically renews certificates 30 days before expiration. For a Let's Encrypt certificate valid for 90 days:

• Issue Date: 2026-06-21
• Expiration Date: 2026-09-19 (90 days)
• Renewal Date: 2026-08-20 (30 days before expiration)

You can verify the renewal schedule with:

kubectl get certificate stalwart-tls -n stalwart -o yaml | grep renewalTime

3. Stakater Reloader

When cert-manager renews a certificate, it updates the stalwart-tls Kubernetes Secret with the new certificate and private key. However, the Stalwart pod doesn't automatically detect this change and continues using the old certificate loaded at startup.

Stakater Reloader solves this by watching Secrets and ConfigMaps for changes and automatically triggering a rolling restart of the associated pods.

Deployment: infrastructure/networking/reloader.yaml

How it works:

1. Reloader watches the stalwart-tls secret for changes
2. When cert-manager updates the secret with a renewed certificate, Reloader detects the change
3. Reloader triggers a rolling restart of the Stalwart StatefulSet
4. The new pod loads the updated certificate from the secret

Annotation in stalwart-fresh-deployment.yaml:

metadata:
  annotations:
    secret.reloader.stakater.com/reload: "stalwart-tls"

Verification

Check Certificate Status

# View certificate details
kubectl get certificate stalwart-tls -n stalwart

# Check expiration and renewal time
kubectl describe certificate stalwart-tls -n stalwart

# View the actual certificate from the secret
kubectl get secret stalwart-tls -n stalwart -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text

Check Reloader Status

# Check if Reloader is running
kubectl get deployment reloader -n reloader

# View Reloader logs
kubectl logs -n reloader -l app=reloader

# Check for reload events
kubectl get events -n stalwart --sort-by='.lastTimestamp' | grep -i reload

Monitor Renewal Process

cert-manager creates temporary Pods and Ingress resources during certificate renewal. You can monitor this process:

# Watch cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager -f

# Check for ACME challenge resources
kubectl get challenges -n stalwart

# Check certificate renewal history
kubectl describe certificate stalwart-tls -n stalwart | tail -20

Troubleshooting

Certificate Not Renewing

1. Check cert-manager is running:

   kubectl get pods -n cert-manager
   

2. Check ClusterIssuer status:

   kubectl get clusterissuer letsencrypt-prod
   

3. Check Certificate resource events:

   kubectl describe certificate stalwart-tls -n stalwart
   

4. Check cert-manager logs:

   kubectl logs -n cert-manager -l app=cert-manager --tail=100
   

Pod Not Reloading After Renewal

1. Check Reloader is running:

   kubectl get deployment reloader -n reloader
   

2. Verify the annotation is present:

   kubectl get statefulset stalwart -n stalwart -o yaml | grep reload
   

3. Check Reloader logs:

   kubectl logs -n reloader -l app=reloader --tail=100
   

4. Manually trigger a reload (if needed):

   kubectl rollout restart statefulset/stalwart -n stalwart
   

Certificate Validation Issues

If cert-manager can't complete the ACME challenge:

1. Ensure DNS is correct:

   dig mail.basicstack.de
   

2. Check Ingress is accessible:

   curl -I http://mail.basicstack.de/.well-known/acme-challenge/test
   

3. Verify Traefik is routing correctly:

   kubectl logs -n kube-system -l app.kubernetes.io/name=traefik
   

Manual Operations

Force Certificate Renewal

If you need to force a renewal (e.g., certificate was compromised):

# Delete the Certificate resource (cert-manager will recreate it)
kubectl delete certificate stalwart-tls -n stalwart

# Or, trigger renewal by deleting the secret
kubectl delete secret stalwart-tls -n stalwart

cert-manager will automatically request a new certificate.

Temporarily Disable Auto-Reload

If you need to prevent automatic restarts during maintenance:

# Scale down Reloader
kubectl scale deployment reloader -n reloader --replicas=0

# Later, scale it back up
kubectl scale deployment reloader -n reloader --replicas=1

Security Considerations

1. Private Key Protection: The certificate private key is stored in the stalwart-tls Kubernetes Secret with restricted access
2. ACME Account Key: The Let's Encrypt account key is stored in letsencrypt-prod-account-key secret
3. Rate Limits: Let's Encrypt has rate limits (50 certificates per domain per week). Avoid unnecessary deletions.
4. Certificate Revocation: If a certificate is compromised, revoke it through Let's Encrypt and request a new one

Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                      Certificate Lifecycle                   │
└─────────────────────────────────────────────────────────────┘

1. Initial Request:
   Ingress → cert-manager → Let's Encrypt (ACME HTTP-01) → Secret

2. Automatic Renewal (30 days before expiry):
   cert-manager timer → Let's Encrypt → Updated Secret

3. Automatic Pod Reload:
   Reloader (watching Secret) → Rolling restart → Stalwart loads new cert

┌──────────────┐      ┌──────────────┐      ┌──────────────┐
│              │      │              │      │              │
│ cert-manager │─────>│ stalwart-tls │<─────│   Reloader   │
│              │      │    Secret    │      │              │
└──────────────┘      └──────────────┘      └──────────────┘
       │                      │                      │
       │                      │                      │
       v                      v                      v
  Let's Encrypt          Mounted in           Triggers restart
                         Stalwart Pod         when secret changes

References

• cert-manager documentation
• Let's Encrypt ACME documentation
• Stakater Reloader
• Stalwart configuration