basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

chore(stalwart): retire RocksDB PVC + backup — Postgres backend authoritative (DEV-474) #10

Open
forgejoadmin wants to merge 1 commit from feat/DEV-474-rocksdb-teardown into main
pull from: feat/DEV-474-rocksdb-teardown
merge into: basicstack:main
Conversation 0 Commits 1 Files changed 2 +37 -224
2 changed files with 37 additions and 224 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files
Showing only changes of commit 0dfd6ada6e - Show all commits

49
apps/stalwart/README.md
View file

@ -1,21 +1,21 @@
# Stalwart Mail Server v0.16.11
# Stalwart Mail Server v0.16.18
Clean deployment of Stalwart mail server with username/password authentication only.
Deployment of Stalwart mail server on a PostgreSQL backend (RocksDB was retired in DEV-467/DEV-473; teardown completed in DEV-474).
## Architecture
- **Version**: v0.16.11
- **Authentication**: Username/password only (NO OAuth/OIDC)
- **Configuration**: API-based (stored in RocksDB)
- **Storage**: Encrypted hcloud-volumes (20Gi)
- **Backup**: Daily restic backup to S3 at 3 AM
- **Version**: v0.16.18
- **Authentication**: OIDC via Pocket-ID (backed by Postgres accounts store)
- **Configuration + state**: PostgreSQL (`stalwart-postgres` StatefulSet in-namespace)
- **Storage**: Encrypted hcloud-volumes attached to `stalwart-postgres-0` (10Gi). Stalwart itself uses an emptyDir at `/var/lib/stalwart` — no persistent block volume on `stalwart-0`
- **Backup**: Daily Postgres logical dump to S3 via `stalwart-postgres-backup` CronJob (restic tag `stalwart-postgres`) at 02:30 UTC
- **Web UI**: https://mail.basicstack.de
## Network Architecture
Stalwart uses a two-tier load balancing setup:
1. **Kubernetes LoadBalancer Services**: Four separate LoadBalancer services expose SMTP (25, 587, 465) and IMAP (143, 993) ports. The service type `LoadBalancer` is required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type.
1. **Kubernetes LoadBalancer Services**: Two LoadBalancer services expose SMTP (25, 587) and IMAPS (993) ports (SMTPS/465 removed per DEV-359 approval, plaintext IMAP/143 disabled). The service type `LoadBalancer` is required because the Hetzner Cloud Controller Manager (CCM) automatically provisions and manages the Hetzner Load Balancer when it detects this service type.
2. **Hetzner Load Balancer**: Automatically managed by the Hetzner CCM. The load balancer distributes traffic across:
- k3s-cp-1 (control plane node)
@ -37,8 +37,13 @@ The Hetzner CCM watches for LoadBalancer-type services and automatically:
## Files
- `stalwart-fresh-deployment.yaml` - Main deployment manifest
- `stalwart-fresh-deployment.yaml` - Main deployment manifest (StatefulSet, services, ingress)
- `stalwart-postgres.yaml` - Postgres backend StatefulSet + service (state store)
- `stalwart-postgres-backup.yaml` - Daily `pg_dump` → restic CronJob (tag `stalwart-postgres`)
- `stalwart-config.yaml` - Stalwart configmap (`stalwart.toml`)
- `stalwart-bootstrap-config.yaml` - Bootstrap configmap pointing Stalwart at Postgres
- `stalwart-admin-credentials-sealed.yaml` - Sealed secret for admin password
- `stalwart-postgres-credentials-sealed.yaml` - Sealed secret for Postgres role
- `stalwart-s3-backup-sealed.yaml` - Sealed secret for S3 backup credentials
## Deployment
@ -46,12 +51,16 @@ The Hetzner CCM watches for LoadBalancer-type services and automatically:
```bash
# Apply sealed secrets first
kubectl apply -f stalwart-admin-credentials-sealed.yaml
kubectl apply -f stalwart-postgres-credentials-sealed.yaml
kubectl apply -f stalwart-s3-backup-sealed.yaml
# Create bootstrap config
kubectl create configmap stalwart-bootstrap-config \
--from-literal=config.json='{"@type":"RocksDb","path":"/var/lib/stalwart"}' \
-n stalwart
# Configmaps: bootstrap points Stalwart at Postgres; stalwart-config carries stalwart.toml
kubectl apply -f stalwart-bootstrap-config.yaml
kubectl apply -f stalwart-config.yaml
# Postgres backend + backup
kubectl apply -f stalwart-postgres.yaml
kubectl apply -f stalwart-postgres-backup.yaml
# Deploy Stalwart
kubectl apply -f stalwart-fresh-deployment.yaml
@ -65,9 +74,11 @@ After deployment, log in at https://mail.basicstack.de with:
## Configuration
> **IMPORTANT:** The configuration is stored in the RocksDB and cannot be overwritten by a configuration file!
> **IMPORTANT:** Runtime configuration and account/queue state now live in **PostgreSQL** (`stalwart-postgres-0`), not RocksDB.
All configuration is done via the web UI or API. The bootstrap config only points to the RocksDB database location. NO config.toml files are used.
The `stalwart-bootstrap-config` configmap points Stalwart at the Postgres store; runtime edits made in the web UI or API are persisted to Postgres. The `stalwart-config` configmap carries the boot-time `stalwart.toml`.
The historical RocksDB backend (PVC `stalwart-data`, restic tag `stalwart`, `stalwart-backup` CronJob, and the `fix-permissions` init-container) was retired on 2026-08-28 after 14 healthy post-cutover days (DEV-474). No RocksDB restic chain remains; rollback is now via the Postgres `pg_dump` restic chain (`tag=stalwart-postgres`).
Check https://stalw.art/docs/ref/ for configuration possibilities.
The API access via /jmap seems to be too complex for the agent and the configruation has been done manually.
@ -126,11 +137,13 @@ The Kubernetes services MUST remain type `LoadBalancer` because:
## Storage
Data is stored in `/var/lib/stalwart` using the RocksDB database format. This includes:
- Email messages
All persistent state lives in **PostgreSQL** on `stalwart-postgres-0` (PVC `postgres-data-stalwart-postgres-0`, 10Gi, `hcloud-volumes-encrypted`). Stalwart connects to Postgres via the bootstrap config and stores:
- Email messages (blob + metadata)
- User accounts
- Server configuration
- TLS certificates configuration
- Queue / spool state
`/var/lib/stalwart` on `stalwart-0` is an emptyDir mount kept only as a writable path if Stalwart needs one at runtime; no data of value lives there.
## Certificate Renewal

212
apps/stalwart/stalwart-fresh-deployment.yaml
View file

@ -5,19 +5,6 @@ metadata:
name: stalwart
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: stalwart-data
namespace: stalwart
spec:
accessModes:
- ReadWriteOnce
storageClassName: hcloud-volumes-encrypted
resources:
requests:
storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
name: stalwart-smtp
@ -114,12 +101,6 @@ spec:
readOnly: true
- name: config
mountPath: /etc/stalwart
- name: fix-permissions
image: busybox:latest
command: ["sh", "-c", "chown -R 2000:2000 /var/lib/stalwart && chmod -R 755 /var/lib/stalwart"]
volumeMounts:
- name: data
mountPath: /var/lib/stalwart
containers:
- name: stalwart
image: stalwartlabs/stalwart:v0.16.18
@ -159,6 +140,9 @@ spec:
- name: tls-certs
mountPath: /etc/stalwart/certs
readOnly: true
# DEV-474: /var/lib/stalwart is no longer RocksDB storage (state now lives in
# the stalwart-postgres StatefulSet). Kept as emptyDir so stalwart still has
# a writable path if it needs one at runtime.
resources:
requests:
memory: "512Mi"
@ -203,9 +187,10 @@ spec:
timeoutSeconds: 3
failureThreshold: 3
volumes:
# DEV-474: was persistentVolumeClaim.claimName=stalwart-data (RocksDB).
# Postgres-backed now; replaced with emptyDir.
- name: data
persistentVolumeClaim:
claimName: stalwart-data
emptyDir: {}
# emptyDir for config files (populated by copy-config init-container)
- name: config
emptyDir: {}
@ -246,188 +231,3 @@ spec:
name: stalwart-http
port:
number: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: stalwart-backup
namespace: stalwart
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: stalwart-backup
namespace: stalwart
rules:
- apiGroups: ["apps"]
resources: ["statefulsets"]
verbs: ["get", "list", "patch"]
- apiGroups: ["apps"]
resources: ["statefulsets/scale"]
verbs: ["get", "update", "patch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: stalwart-backup
namespace: stalwart
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: stalwart-backup
subjects:
- kind: ServiceAccount
name: stalwart-backup
namespace: stalwart
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: stalwart-backup
namespace: stalwart
spec:
schedule: "0 3 * * *" # 3 AM daily
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 1800 # 30 minute timeout (was 600s; retries + affinity settling can push past 10 min)
template:
metadata:
labels:
app: stalwart-backup
spec:
serviceAccountName: stalwart-backup
restartPolicy: OnFailure
# Co-locate with stalwart-0 so both pods share the same hcloud block volume
# attachment (RWO). Without this the backup pod can be scheduled on a
# different node and hits FailedAttachVolume/Multi-Attach (DEV-468).
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: stalwart
statefulset.kubernetes.io/pod-name: stalwart-0
topologyKey: kubernetes.io/hostname
containers:
- name: backup
image: alpine:3.24
command:
- /bin/sh
- -c
- |
set -e
BACKUP_START=$(date +%s)
echo "=== Starting Stalwart backup at $(date) ==="
# Install required tools
echo "Installing kubectl and restic..."
apk add --no-cache kubectl restic curl bash
# Build restic repository URL from env vars
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/stalwart"
echo "Using restic repository: $RESTIC_REPOSITORY"
# Function to scale StatefulSet
scale_statefulset() {
local replicas=$1
echo "Scaling stalwart StatefulSet to $replicas replicas..."
kubectl scale statefulset stalwart -n stalwart --replicas=$replicas
if [ "$replicas" -eq 0 ]; then
echo "Waiting for pod to terminate..."
kubectl wait --for=delete pod/stalwart-0 -n stalwart --timeout=120s || true
sleep 5
else
echo "Waiting for pod to be ready..."
kubectl wait --for=condition=ready pod/stalwart-0 -n stalwart --timeout=120s || echo "Warning: Pod not ready after 120s"
fi
}
# Trap to ensure we scale back up on any exit
cleanup() {
EXIT_CODE=$?
echo "Cleanup: scaling stalwart back to 1 replica..."
scale_statefulset 1 || echo "ERROR: Failed to scale back up!"
BACKUP_END=$(date +%s)
DURATION=$((BACKUP_END - BACKUP_START))
if [ $EXIT_CODE -eq 0 ]; then
echo "=== Backup completed successfully in ${DURATION}s at $(date) ==="
else
echo "=== Backup FAILED after ${DURATION}s at $(date) ==="
fi
exit $EXIT_CODE
}
trap cleanup EXIT INT TERM
# Scale down Stalwart
scale_statefulset 0
# Build restic repository URL from env vars (K8s doesn't expand $(VAR) in value fields)
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/stalwart"
echo "Using repository: $RESTIC_REPOSITORY"
# Initialize restic repo if needed
echo "Initializing restic repository..."
restic snapshots || restic init
# Backup the data directory
echo "Running backup..."
restic backup /var/lib/stalwart \
--tag stalwart \
--tag daily \
--host stalwart-k8s \
--verbose
# Prune old backups (keep 7 daily, 4 weekly, 6 monthly)
echo "Pruning old backups..."
restic forget \
--tag stalwart \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
--prune
echo "Backup operations completed"
env:
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef:
name: stalwart-s3-backup
key: restic-password
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: stalwart-s3-backup
key: access-key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: stalwart-s3-backup
key: secret-key
- name: S3_ENDPOINT
valueFrom:
secretKeyRef:
name: stalwart-s3-backup
key: endpoint
- name: S3_BUCKET
valueFrom:
secretKeyRef:
name: stalwart-s3-backup
key: bucket
volumeMounts:
- name: data
mountPath: /var/lib/stalwart
readOnly: true
volumes:
- name: data
persistentVolumeClaim:
claimName: stalwart-data