basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Implement automatic TLS certificate renewal for Stalwart

Browse source 
Added Stakater Reloader to automatically restart Stalwart pods when
TLS certificates are renewed by cert-manager. This ensures seamless
certificate rotation without manual intervention.

Changes:
- Deploy Stakater Reloader in infrastructure/networking/
- Add Reloader annotation to Stalwart StatefulSet to watch stalwart-tls secret
- Document certificate renewal process and troubleshooting

The certificate is managed by cert-manager with Let's Encrypt and will
automatically renew 30 days before expiration (renewal date: 2026-08-20).
Reloader detects secret updates and triggers a rolling restart of the
Stalwart StatefulSet to load the new certificate.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-07-04 16:48:48 +00:00
parent a24420ac3e
commit 8054ef4018
3 changed files with 375 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

249
apps/stalwart/CERTIFICATE-RENEWAL.md Normal file
View file

@ -0,0 +1,249 @@
# Stalwart TLS Certificate Automatic Renewal
## Overview
The Stalwart mail server uses TLS certificates for secure email communications (SMTP, IMAP, and HTTPS). These certificates are automatically managed and renewed by cert-manager and reloaded by Stakater Reloader.
## Components
### 1. cert-manager
cert-manager is a Kubernetes add-on that automates the management and issuance of TLS certificates from various issuing sources, including Let's Encrypt.
**Configuration:**
- **ClusterIssuer:** `letsencrypt-prod` (configured in `infrastructure/networking/`)
- **Email:** admin@basicstack.de
- **ACME Server:** Let's Encrypt production (https://acme-v02.api.letsencrypt.org/directory)
- **Challenge Type:** HTTP-01 (via Traefik ingress)
**Certificate Resource:**
```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: stalwart-tls
namespace: stalwart
spec:
dnsNames:
- mail.basicstack.de
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod
secretName: stalwart-tls
```
The Certificate resource is automatically created by cert-manager when it sees the Ingress annotation:
```yaml
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
```
### 2. Automatic Renewal Schedule
cert-manager automatically renews certificates **30 days before expiration**. For a Let's Encrypt certificate valid for 90 days:
- **Issue Date:** 2026-06-21
- **Expiration Date:** 2026-09-19 (90 days)
- **Renewal Date:** 2026-08-20 (30 days before expiration)
You can verify the renewal schedule with:
```bash
kubectl get certificate stalwart-tls -n stalwart -o yaml | grep renewalTime
```
### 3. Stakater Reloader
When cert-manager renews a certificate, it updates the `stalwart-tls` Kubernetes Secret with the new certificate and private key. However, the Stalwart pod doesn't automatically detect this change and continues using the old certificate loaded at startup.
**Stakater Reloader** solves this by watching Secrets and ConfigMaps for changes and automatically triggering a rolling restart of the associated pods.
**Deployment:** `infrastructure/networking/reloader.yaml`
**How it works:**
1. Reloader watches the `stalwart-tls` secret for changes
2. When cert-manager updates the secret with a renewed certificate, Reloader detects the change
3. Reloader triggers a rolling restart of the Stalwart StatefulSet
4. The new pod loads the updated certificate from the secret
**Annotation in stalwart-fresh-deployment.yaml:**
```yaml
metadata:
annotations:
secret.reloader.stakater.com/reload: "stalwart-tls"
```
## Verification
### Check Certificate Status
```bash
# View certificate details
kubectl get certificate stalwart-tls -n stalwart
# Check expiration and renewal time
kubectl describe certificate stalwart-tls -n stalwart
# View the actual certificate from the secret
kubectl get secret stalwart-tls -n stalwart -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text
```
### Check Reloader Status
```bash
# Check if Reloader is running
kubectl get deployment reloader -n reloader
# View Reloader logs
kubectl logs -n reloader -l app=reloader
# Check for reload events
kubectl get events -n stalwart --sort-by='.lastTimestamp' | grep -i reload
```
### Monitor Renewal Process
cert-manager creates temporary Pods and Ingress resources during certificate renewal. You can monitor this process:
```bash
# Watch cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager -f
# Check for ACME challenge resources
kubectl get challenges -n stalwart
# Check certificate renewal history
kubectl describe certificate stalwart-tls -n stalwart | tail -20
```
## Troubleshooting
### Certificate Not Renewing
1. **Check cert-manager is running:**
```bash
kubectl get pods -n cert-manager
```
2. **Check ClusterIssuer status:**
```bash
kubectl get clusterissuer letsencrypt-prod
```
3. **Check Certificate resource events:**
```bash
kubectl describe certificate stalwart-tls -n stalwart
```
4. **Check cert-manager logs:**
```bash
kubectl logs -n cert-manager -l app=cert-manager --tail=100
```
### Pod Not Reloading After Renewal
1. **Check Reloader is running:**
```bash
kubectl get deployment reloader -n reloader
```
2. **Verify the annotation is present:**
```bash
kubectl get statefulset stalwart -n stalwart -o yaml | grep reload
```
3. **Check Reloader logs:**
```bash
kubectl logs -n reloader -l app=reloader --tail=100
```
4. **Manually trigger a reload (if needed):**
```bash
kubectl rollout restart statefulset/stalwart -n stalwart
```
### Certificate Validation Issues
If cert-manager can't complete the ACME challenge:
1. **Ensure DNS is correct:**
```bash
dig mail.basicstack.de
```
2. **Check Ingress is accessible:**
```bash
curl -I http://mail.basicstack.de/.well-known/acme-challenge/test
```
3. **Verify Traefik is routing correctly:**
```bash
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik
```
## Manual Operations
### Force Certificate Renewal
If you need to force a renewal (e.g., certificate was compromised):
```bash
# Delete the Certificate resource (cert-manager will recreate it)
kubectl delete certificate stalwart-tls -n stalwart
# Or, trigger renewal by deleting the secret
kubectl delete secret stalwart-tls -n stalwart
```
cert-manager will automatically request a new certificate.
### Temporarily Disable Auto-Reload
If you need to prevent automatic restarts during maintenance:
```bash
# Scale down Reloader
kubectl scale deployment reloader -n reloader --replicas=0
# Later, scale it back up
kubectl scale deployment reloader -n reloader --replicas=1
```
## Security Considerations
1. **Private Key Protection:** The certificate private key is stored in the `stalwart-tls` Kubernetes Secret with restricted access
2. **ACME Account Key:** The Let's Encrypt account key is stored in `letsencrypt-prod-account-key` secret
3. **Rate Limits:** Let's Encrypt has rate limits (50 certificates per domain per week). Avoid unnecessary deletions.
4. **Certificate Revocation:** If a certificate is compromised, revoke it through Let's Encrypt and request a new one
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────────┐
│ Certificate Lifecycle │
└─────────────────────────────────────────────────────────────┘
1. Initial Request:
Ingress → cert-manager → Let's Encrypt (ACME HTTP-01) → Secret
2. Automatic Renewal (30 days before expiry):
cert-manager timer → Let's Encrypt → Updated Secret
3. Automatic Pod Reload:
Reloader (watching Secret) → Rolling restart → Stalwart loads new cert
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ │ │ │ │ │
│ cert-manager │─────>│ stalwart-tls │<─────│ Reloader │
│ │ │ Secret │ │ │
└──────────────┘ └──────────────┘ └──────────────┘
│ │ │
│ │ │
v v v
Let's Encrypt Mounted in Triggers restart
Stalwart Pod when secret changes
```
## References
- [cert-manager documentation](https://cert-manager.io/docs/)
- [Let's Encrypt ACME documentation](https://letsencrypt.org/docs/)
- [Stakater Reloader](https://github.com/stakater/Reloader)
- [Stalwart configuration](https://stalw.art/docs/configuration)

4
apps/stalwart/stalwart-fresh-deployment.yaml
View file

@ -79,6 +79,10 @@ kind: StatefulSet
metadata: metadata:
name: stalwart name: stalwart
namespace: stalwart namespace: stalwart
annotations:
# Automatically restart this StatefulSet when the TLS certificate secret is updated
# This ensures the pod reloads new certificates after cert-manager renews them
secret.reloader.stakater.com/reload: "stalwart-tls"
spec: spec:
serviceName: stalwart-http serviceName: stalwart-http
replicas: 1 replicas: 1

122
infrastructure/networking/reloader.yaml Normal file
View file

@ -0,0 +1,122 @@
---
# Stakater Reloader - Automatically restarts pods when their ConfigMaps or Secrets change
# This is essential for certificate renewal, as it ensures pods reload updated TLS certificates
# without manual intervention.
#
# Docs: https://github.com/stakater/Reloader
apiVersion: v1
kind: Namespace
metadata:
name: reloader
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: reloader
namespace: reloader
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: reloader
rules:
- apiGroups:
- ""
resources:
- secrets
- configmaps
verbs:
- list
- get
- watch
- apiGroups:
- apps
resources:
- deployments
- daemonsets
- statefulsets
verbs:
- list
- get
- update
- patch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: reloader
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: reloader
subjects:
- kind: ServiceAccount
name: reloader
namespace: reloader
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: reloader
namespace: reloader
labels:
app: reloader
spec:
replicas: 1
selector:
matchLabels:
app: reloader
template:
metadata:
labels:
app: reloader
spec:
serviceAccountName: reloader
containers:
- name: reloader
image: stakater/reloader:v1.0.79
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 9090
livenessProbe:
httpGet:
path: /live
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /metrics
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 10m
memory: 32Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65534
securityContext:
fsGroup: 65534