diff --git a/apps/stalwart/CERTIFICATE-RENEWAL.md b/apps/stalwart/CERTIFICATE-RENEWAL.md new file mode 100644 index 0000000..77dbca2 --- /dev/null +++ b/apps/stalwart/CERTIFICATE-RENEWAL.md @@ -0,0 +1,249 @@ +# Stalwart TLS Certificate Automatic Renewal + +## Overview + +The Stalwart mail server uses TLS certificates for secure email communications (SMTP, IMAP, and HTTPS). These certificates are automatically managed and renewed by cert-manager and reloaded by Stakater Reloader. + +## Components + +### 1. cert-manager + +cert-manager is a Kubernetes add-on that automates the management and issuance of TLS certificates from various issuing sources, including Let's Encrypt. + +**Configuration:** +- **ClusterIssuer:** `letsencrypt-prod` (configured in `infrastructure/networking/`) +- **Email:** admin@basicstack.de +- **ACME Server:** Let's Encrypt production (https://acme-v02.api.letsencrypt.org/directory) +- **Challenge Type:** HTTP-01 (via Traefik ingress) + +**Certificate Resource:** +```yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: stalwart-tls + namespace: stalwart +spec: + dnsNames: + - mail.basicstack.de + issuerRef: + kind: ClusterIssuer + name: letsencrypt-prod + secretName: stalwart-tls +``` + +The Certificate resource is automatically created by cert-manager when it sees the Ingress annotation: +```yaml +annotations: + cert-manager.io/cluster-issuer: "letsencrypt-prod" +``` + +### 2. Automatic Renewal Schedule + +cert-manager automatically renews certificates **30 days before expiration**. For a Let's Encrypt certificate valid for 90 days: + +- **Issue Date:** 2026-06-21 +- **Expiration Date:** 2026-09-19 (90 days) +- **Renewal Date:** 2026-08-20 (30 days before expiration) + +You can verify the renewal schedule with: +```bash +kubectl get certificate stalwart-tls -n stalwart -o yaml | grep renewalTime +``` + +### 3. Stakater Reloader + +When cert-manager renews a certificate, it updates the `stalwart-tls` Kubernetes Secret with the new certificate and private key. However, the Stalwart pod doesn't automatically detect this change and continues using the old certificate loaded at startup. + +**Stakater Reloader** solves this by watching Secrets and ConfigMaps for changes and automatically triggering a rolling restart of the associated pods. + +**Deployment:** `infrastructure/networking/reloader.yaml` + +**How it works:** +1. Reloader watches the `stalwart-tls` secret for changes +2. When cert-manager updates the secret with a renewed certificate, Reloader detects the change +3. Reloader triggers a rolling restart of the Stalwart StatefulSet +4. The new pod loads the updated certificate from the secret + +**Annotation in stalwart-fresh-deployment.yaml:** +```yaml +metadata: + annotations: + secret.reloader.stakater.com/reload: "stalwart-tls" +``` + +## Verification + +### Check Certificate Status +```bash +# View certificate details +kubectl get certificate stalwart-tls -n stalwart + +# Check expiration and renewal time +kubectl describe certificate stalwart-tls -n stalwart + +# View the actual certificate from the secret +kubectl get secret stalwart-tls -n stalwart -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text +``` + +### Check Reloader Status +```bash +# Check if Reloader is running +kubectl get deployment reloader -n reloader + +# View Reloader logs +kubectl logs -n reloader -l app=reloader + +# Check for reload events +kubectl get events -n stalwart --sort-by='.lastTimestamp' | grep -i reload +``` + +### Monitor Renewal Process + +cert-manager creates temporary Pods and Ingress resources during certificate renewal. You can monitor this process: + +```bash +# Watch cert-manager logs +kubectl logs -n cert-manager -l app=cert-manager -f + +# Check for ACME challenge resources +kubectl get challenges -n stalwart + +# Check certificate renewal history +kubectl describe certificate stalwart-tls -n stalwart | tail -20 +``` + +## Troubleshooting + +### Certificate Not Renewing + +1. **Check cert-manager is running:** + ```bash + kubectl get pods -n cert-manager + ``` + +2. **Check ClusterIssuer status:** + ```bash + kubectl get clusterissuer letsencrypt-prod + ``` + +3. **Check Certificate resource events:** + ```bash + kubectl describe certificate stalwart-tls -n stalwart + ``` + +4. **Check cert-manager logs:** + ```bash + kubectl logs -n cert-manager -l app=cert-manager --tail=100 + ``` + +### Pod Not Reloading After Renewal + +1. **Check Reloader is running:** + ```bash + kubectl get deployment reloader -n reloader + ``` + +2. **Verify the annotation is present:** + ```bash + kubectl get statefulset stalwart -n stalwart -o yaml | grep reload + ``` + +3. **Check Reloader logs:** + ```bash + kubectl logs -n reloader -l app=reloader --tail=100 + ``` + +4. **Manually trigger a reload (if needed):** + ```bash + kubectl rollout restart statefulset/stalwart -n stalwart + ``` + +### Certificate Validation Issues + +If cert-manager can't complete the ACME challenge: + +1. **Ensure DNS is correct:** + ```bash + dig mail.basicstack.de + ``` + +2. **Check Ingress is accessible:** + ```bash + curl -I http://mail.basicstack.de/.well-known/acme-challenge/test + ``` + +3. **Verify Traefik is routing correctly:** + ```bash + kubectl logs -n kube-system -l app.kubernetes.io/name=traefik + ``` + +## Manual Operations + +### Force Certificate Renewal + +If you need to force a renewal (e.g., certificate was compromised): + +```bash +# Delete the Certificate resource (cert-manager will recreate it) +kubectl delete certificate stalwart-tls -n stalwart + +# Or, trigger renewal by deleting the secret +kubectl delete secret stalwart-tls -n stalwart +``` + +cert-manager will automatically request a new certificate. + +### Temporarily Disable Auto-Reload + +If you need to prevent automatic restarts during maintenance: + +```bash +# Scale down Reloader +kubectl scale deployment reloader -n reloader --replicas=0 + +# Later, scale it back up +kubectl scale deployment reloader -n reloader --replicas=1 +``` + +## Security Considerations + +1. **Private Key Protection:** The certificate private key is stored in the `stalwart-tls` Kubernetes Secret with restricted access +2. **ACME Account Key:** The Let's Encrypt account key is stored in `letsencrypt-prod-account-key` secret +3. **Rate Limits:** Let's Encrypt has rate limits (50 certificates per domain per week). Avoid unnecessary deletions. +4. **Certificate Revocation:** If a certificate is compromised, revoke it through Let's Encrypt and request a new one + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Certificate Lifecycle │ +└─────────────────────────────────────────────────────────────┘ + +1. Initial Request: + Ingress → cert-manager → Let's Encrypt (ACME HTTP-01) → Secret + +2. Automatic Renewal (30 days before expiry): + cert-manager timer → Let's Encrypt → Updated Secret + +3. Automatic Pod Reload: + Reloader (watching Secret) → Rolling restart → Stalwart loads new cert + +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ │ │ │ │ │ +│ cert-manager │─────>│ stalwart-tls │<─────│ Reloader │ +│ │ │ Secret │ │ │ +└──────────────┘ └──────────────┘ └──────────────┘ + │ │ │ + │ │ │ + v v v + Let's Encrypt Mounted in Triggers restart + Stalwart Pod when secret changes +``` + +## References + +- [cert-manager documentation](https://cert-manager.io/docs/) +- [Let's Encrypt ACME documentation](https://letsencrypt.org/docs/) +- [Stakater Reloader](https://github.com/stakater/Reloader) +- [Stalwart configuration](https://stalw.art/docs/configuration) diff --git a/apps/stalwart/stalwart-fresh-deployment.yaml b/apps/stalwart/stalwart-fresh-deployment.yaml index 630fa2f..e4a5f69 100644 --- a/apps/stalwart/stalwart-fresh-deployment.yaml +++ b/apps/stalwart/stalwart-fresh-deployment.yaml @@ -79,6 +79,10 @@ kind: StatefulSet metadata: name: stalwart namespace: stalwart + annotations: + # Automatically restart this StatefulSet when the TLS certificate secret is updated + # This ensures the pod reloads new certificates after cert-manager renews them + secret.reloader.stakater.com/reload: "stalwart-tls" spec: serviceName: stalwart-http replicas: 1 diff --git a/infrastructure/networking/reloader.yaml b/infrastructure/networking/reloader.yaml new file mode 100644 index 0000000..454847b --- /dev/null +++ b/infrastructure/networking/reloader.yaml @@ -0,0 +1,122 @@ +--- +# Stakater Reloader - Automatically restarts pods when their ConfigMaps or Secrets change +# This is essential for certificate renewal, as it ensures pods reload updated TLS certificates +# without manual intervention. +# +# Docs: https://github.com/stakater/Reloader + +apiVersion: v1 +kind: Namespace +metadata: + name: reloader +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: reloader + namespace: reloader +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: reloader +rules: + - apiGroups: + - "" + resources: + - secrets + - configmaps + verbs: + - list + - get + - watch + - apiGroups: + - apps + resources: + - deployments + - daemonsets + - statefulsets + verbs: + - list + - get + - update + - patch + - apiGroups: + - "" + resources: + - events + verbs: + - create + - patch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: reloader +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: reloader +subjects: + - kind: ServiceAccount + name: reloader + namespace: reloader +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: reloader + namespace: reloader + labels: + app: reloader +spec: + replicas: 1 + selector: + matchLabels: + app: reloader + template: + metadata: + labels: + app: reloader + spec: + serviceAccountName: reloader + containers: + - name: reloader + image: stakater/reloader:v1.0.79 + imagePullPolicy: IfNotPresent + ports: + - name: http + containerPort: 9090 + livenessProbe: + httpGet: + path: /live + port: http + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /metrics + port: http + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + resources: + limits: + cpu: 100m + memory: 128Mi + requests: + cpu: 10m + memory: 32Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 65534 + securityContext: + fsGroup: 65534