Implement automatic TLS certificate renewal for Stalwart
Added Stakater Reloader to automatically restart Stalwart pods when TLS certificates are renewed by cert-manager. This ensures seamless certificate rotation without manual intervention. Changes: - Deploy Stakater Reloader in infrastructure/networking/ - Add Reloader annotation to Stalwart StatefulSet to watch stalwart-tls secret - Document certificate renewal process and troubleshooting The certificate is managed by cert-manager with Let's Encrypt and will automatically renew 30 days before expiration (renewal date: 2026-08-20). Reloader detects secret updates and triggers a rolling restart of the Stalwart StatefulSet to load the new certificate. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
a24420ac3e
commit
8054ef4018
3 changed files with 375 additions and 0 deletions
249
apps/stalwart/CERTIFICATE-RENEWAL.md
Normal file
249
apps/stalwart/CERTIFICATE-RENEWAL.md
Normal file
|
|
@ -0,0 +1,249 @@
|
|||
# Stalwart TLS Certificate Automatic Renewal
|
||||
|
||||
## Overview
|
||||
|
||||
The Stalwart mail server uses TLS certificates for secure email communications (SMTP, IMAP, and HTTPS). These certificates are automatically managed and renewed by cert-manager and reloaded by Stakater Reloader.
|
||||
|
||||
## Components
|
||||
|
||||
### 1. cert-manager
|
||||
|
||||
cert-manager is a Kubernetes add-on that automates the management and issuance of TLS certificates from various issuing sources, including Let's Encrypt.
|
||||
|
||||
**Configuration:**
|
||||
- **ClusterIssuer:** `letsencrypt-prod` (configured in `infrastructure/networking/`)
|
||||
- **Email:** admin@basicstack.de
|
||||
- **ACME Server:** Let's Encrypt production (https://acme-v02.api.letsencrypt.org/directory)
|
||||
- **Challenge Type:** HTTP-01 (via Traefik ingress)
|
||||
|
||||
**Certificate Resource:**
|
||||
```yaml
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: stalwart-tls
|
||||
namespace: stalwart
|
||||
spec:
|
||||
dnsNames:
|
||||
- mail.basicstack.de
|
||||
issuerRef:
|
||||
kind: ClusterIssuer
|
||||
name: letsencrypt-prod
|
||||
secretName: stalwart-tls
|
||||
```
|
||||
|
||||
The Certificate resource is automatically created by cert-manager when it sees the Ingress annotation:
|
||||
```yaml
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: "letsencrypt-prod"
|
||||
```
|
||||
|
||||
### 2. Automatic Renewal Schedule
|
||||
|
||||
cert-manager automatically renews certificates **30 days before expiration**. For a Let's Encrypt certificate valid for 90 days:
|
||||
|
||||
- **Issue Date:** 2026-06-21
|
||||
- **Expiration Date:** 2026-09-19 (90 days)
|
||||
- **Renewal Date:** 2026-08-20 (30 days before expiration)
|
||||
|
||||
You can verify the renewal schedule with:
|
||||
```bash
|
||||
kubectl get certificate stalwart-tls -n stalwart -o yaml | grep renewalTime
|
||||
```
|
||||
|
||||
### 3. Stakater Reloader
|
||||
|
||||
When cert-manager renews a certificate, it updates the `stalwart-tls` Kubernetes Secret with the new certificate and private key. However, the Stalwart pod doesn't automatically detect this change and continues using the old certificate loaded at startup.
|
||||
|
||||
**Stakater Reloader** solves this by watching Secrets and ConfigMaps for changes and automatically triggering a rolling restart of the associated pods.
|
||||
|
||||
**Deployment:** `infrastructure/networking/reloader.yaml`
|
||||
|
||||
**How it works:**
|
||||
1. Reloader watches the `stalwart-tls` secret for changes
|
||||
2. When cert-manager updates the secret with a renewed certificate, Reloader detects the change
|
||||
3. Reloader triggers a rolling restart of the Stalwart StatefulSet
|
||||
4. The new pod loads the updated certificate from the secret
|
||||
|
||||
**Annotation in stalwart-fresh-deployment.yaml:**
|
||||
```yaml
|
||||
metadata:
|
||||
annotations:
|
||||
secret.reloader.stakater.com/reload: "stalwart-tls"
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
### Check Certificate Status
|
||||
```bash
|
||||
# View certificate details
|
||||
kubectl get certificate stalwart-tls -n stalwart
|
||||
|
||||
# Check expiration and renewal time
|
||||
kubectl describe certificate stalwart-tls -n stalwart
|
||||
|
||||
# View the actual certificate from the secret
|
||||
kubectl get secret stalwart-tls -n stalwart -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text
|
||||
```
|
||||
|
||||
### Check Reloader Status
|
||||
```bash
|
||||
# Check if Reloader is running
|
||||
kubectl get deployment reloader -n reloader
|
||||
|
||||
# View Reloader logs
|
||||
kubectl logs -n reloader -l app=reloader
|
||||
|
||||
# Check for reload events
|
||||
kubectl get events -n stalwart --sort-by='.lastTimestamp' | grep -i reload
|
||||
```
|
||||
|
||||
### Monitor Renewal Process
|
||||
|
||||
cert-manager creates temporary Pods and Ingress resources during certificate renewal. You can monitor this process:
|
||||
|
||||
```bash
|
||||
# Watch cert-manager logs
|
||||
kubectl logs -n cert-manager -l app=cert-manager -f
|
||||
|
||||
# Check for ACME challenge resources
|
||||
kubectl get challenges -n stalwart
|
||||
|
||||
# Check certificate renewal history
|
||||
kubectl describe certificate stalwart-tls -n stalwart | tail -20
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Certificate Not Renewing
|
||||
|
||||
1. **Check cert-manager is running:**
|
||||
```bash
|
||||
kubectl get pods -n cert-manager
|
||||
```
|
||||
|
||||
2. **Check ClusterIssuer status:**
|
||||
```bash
|
||||
kubectl get clusterissuer letsencrypt-prod
|
||||
```
|
||||
|
||||
3. **Check Certificate resource events:**
|
||||
```bash
|
||||
kubectl describe certificate stalwart-tls -n stalwart
|
||||
```
|
||||
|
||||
4. **Check cert-manager logs:**
|
||||
```bash
|
||||
kubectl logs -n cert-manager -l app=cert-manager --tail=100
|
||||
```
|
||||
|
||||
### Pod Not Reloading After Renewal
|
||||
|
||||
1. **Check Reloader is running:**
|
||||
```bash
|
||||
kubectl get deployment reloader -n reloader
|
||||
```
|
||||
|
||||
2. **Verify the annotation is present:**
|
||||
```bash
|
||||
kubectl get statefulset stalwart -n stalwart -o yaml | grep reload
|
||||
```
|
||||
|
||||
3. **Check Reloader logs:**
|
||||
```bash
|
||||
kubectl logs -n reloader -l app=reloader --tail=100
|
||||
```
|
||||
|
||||
4. **Manually trigger a reload (if needed):**
|
||||
```bash
|
||||
kubectl rollout restart statefulset/stalwart -n stalwart
|
||||
```
|
||||
|
||||
### Certificate Validation Issues
|
||||
|
||||
If cert-manager can't complete the ACME challenge:
|
||||
|
||||
1. **Ensure DNS is correct:**
|
||||
```bash
|
||||
dig mail.basicstack.de
|
||||
```
|
||||
|
||||
2. **Check Ingress is accessible:**
|
||||
```bash
|
||||
curl -I http://mail.basicstack.de/.well-known/acme-challenge/test
|
||||
```
|
||||
|
||||
3. **Verify Traefik is routing correctly:**
|
||||
```bash
|
||||
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik
|
||||
```
|
||||
|
||||
## Manual Operations
|
||||
|
||||
### Force Certificate Renewal
|
||||
|
||||
If you need to force a renewal (e.g., certificate was compromised):
|
||||
|
||||
```bash
|
||||
# Delete the Certificate resource (cert-manager will recreate it)
|
||||
kubectl delete certificate stalwart-tls -n stalwart
|
||||
|
||||
# Or, trigger renewal by deleting the secret
|
||||
kubectl delete secret stalwart-tls -n stalwart
|
||||
```
|
||||
|
||||
cert-manager will automatically request a new certificate.
|
||||
|
||||
### Temporarily Disable Auto-Reload
|
||||
|
||||
If you need to prevent automatic restarts during maintenance:
|
||||
|
||||
```bash
|
||||
# Scale down Reloader
|
||||
kubectl scale deployment reloader -n reloader --replicas=0
|
||||
|
||||
# Later, scale it back up
|
||||
kubectl scale deployment reloader -n reloader --replicas=1
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Private Key Protection:** The certificate private key is stored in the `stalwart-tls` Kubernetes Secret with restricted access
|
||||
2. **ACME Account Key:** The Let's Encrypt account key is stored in `letsencrypt-prod-account-key` secret
|
||||
3. **Rate Limits:** Let's Encrypt has rate limits (50 certificates per domain per week). Avoid unnecessary deletions.
|
||||
4. **Certificate Revocation:** If a certificate is compromised, revoke it through Let's Encrypt and request a new one
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Certificate Lifecycle │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
1. Initial Request:
|
||||
Ingress → cert-manager → Let's Encrypt (ACME HTTP-01) → Secret
|
||||
|
||||
2. Automatic Renewal (30 days before expiry):
|
||||
cert-manager timer → Let's Encrypt → Updated Secret
|
||||
|
||||
3. Automatic Pod Reload:
|
||||
Reloader (watching Secret) → Rolling restart → Stalwart loads new cert
|
||||
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ │ │ │ │ │
|
||||
│ cert-manager │─────>│ stalwart-tls │<─────│ Reloader │
|
||||
│ │ │ Secret │ │ │
|
||||
└──────────────┘ └──────────────┘ └──────────────┘
|
||||
│ │ │
|
||||
│ │ │
|
||||
v v v
|
||||
Let's Encrypt Mounted in Triggers restart
|
||||
Stalwart Pod when secret changes
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [cert-manager documentation](https://cert-manager.io/docs/)
|
||||
- [Let's Encrypt ACME documentation](https://letsencrypt.org/docs/)
|
||||
- [Stakater Reloader](https://github.com/stakater/Reloader)
|
||||
- [Stalwart configuration](https://stalw.art/docs/configuration)
|
||||
|
|
@ -79,6 +79,10 @@ kind: StatefulSet
|
|||
metadata:
|
||||
name: stalwart
|
||||
namespace: stalwart
|
||||
annotations:
|
||||
# Automatically restart this StatefulSet when the TLS certificate secret is updated
|
||||
# This ensures the pod reloads new certificates after cert-manager renews them
|
||||
secret.reloader.stakater.com/reload: "stalwart-tls"
|
||||
spec:
|
||||
serviceName: stalwart-http
|
||||
replicas: 1
|
||||
|
|
|
|||
122
infrastructure/networking/reloader.yaml
Normal file
122
infrastructure/networking/reloader.yaml
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
---
|
||||
# Stakater Reloader - Automatically restarts pods when their ConfigMaps or Secrets change
|
||||
# This is essential for certificate renewal, as it ensures pods reload updated TLS certificates
|
||||
# without manual intervention.
|
||||
#
|
||||
# Docs: https://github.com/stakater/Reloader
|
||||
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: reloader
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: reloader
|
||||
namespace: reloader
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: reloader
|
||||
rules:
|
||||
- apiGroups:
|
||||
- ""
|
||||
resources:
|
||||
- secrets
|
||||
- configmaps
|
||||
verbs:
|
||||
- list
|
||||
- get
|
||||
- watch
|
||||
- apiGroups:
|
||||
- apps
|
||||
resources:
|
||||
- deployments
|
||||
- daemonsets
|
||||
- statefulsets
|
||||
verbs:
|
||||
- list
|
||||
- get
|
||||
- update
|
||||
- patch
|
||||
- apiGroups:
|
||||
- ""
|
||||
resources:
|
||||
- events
|
||||
verbs:
|
||||
- create
|
||||
- patch
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: reloader
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: reloader
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: reloader
|
||||
namespace: reloader
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: reloader
|
||||
namespace: reloader
|
||||
labels:
|
||||
app: reloader
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: reloader
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: reloader
|
||||
spec:
|
||||
serviceAccountName: reloader
|
||||
containers:
|
||||
- name: reloader
|
||||
image: stakater/reloader:v1.0.79
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 9090
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /live
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /metrics
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
resources:
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65534
|
||||
securityContext:
|
||||
fsGroup: 65534
|
||||
Loading…
Add table
Reference in a new issue