stack.basicstack.de/apps/stalwart/stalwart-backup-restore.md

Stalwart Mail Server — Backup & Restore Procedures

Overview

Stalwart stores all data in a RocksDB database mounted at /opt/stalwart-mail on a 20Gi encrypted hcloud volume (hcloud-volumes-encrypted storage class).

Backup Strategy

Option A: Hetzner Volume Snapshots (Recommended)

Hetzner Cloud provides volume snapshots that capture the full encrypted volume state.

Manual Snapshot via hcloud CLI

# Get the volume ID
bin/hcloud volume list | grep stalwart

# Create a snapshot (works even while volume is mounted — RocksDB is crash-safe)
bin/hcloud volume snapshot create <volume-id> --description "stalwart-backup-$(date +%Y%m%d)"

Scheduled Snapshots (Daily)

Create a CronJob in Kubernetes to automate snapshots:

# Requires hcloud CLI and API token in a secret
apiVersion: batch/v1
kind: CronJob
metadata:
  name: stalwart-snapshot
  namespace: mail
spec:
  schedule: "0 3 * * *"  # 3 AM daily
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: hcloud-snapshot
            image: alpine:latest
            command:
            - /bin/sh
            - -c
            - |
              apk add --no-cache curl jq
              VOLUME_ID=$(curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \
                https://api.hetzner.cloud/v1/volumes | \
                jq -r '.volumes[] | select(.name | contains("stalwart")) | .id')
              curl -X POST -H "Authorization: Bearer $HCLOUD_TOKEN" \
                -H "Content-Type: application/json" \
                -d "{\"description\": \"stalwart-auto-$(date +%Y%m%d)\"}" \
                https://api.hetzner.cloud/v1/volumes/$VOLUME_ID/actions/create_snapshot
            env:
            - name: HCLOUD_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hcloud-credentials
                  key: token

Option B: Filesystem-Level Backup

Back up the data directory while Stalwart is paused or using a consistent snapshot.

# Scale down Stalwart (brief downtime)
kubectl scale deployment stalwart -n mail --replicas=0

# Exec into a temporary pod with the same PVC
kubectl run backup-helper --image=alpine --restart=Never \
  -n mail \
  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"stalwart-data"}}],"containers":[{"name":"backup-helper","image":"alpine","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/opt/stalwart-mail"}]}]}}'

# Copy data out
kubectl cp mail/backup-helper:/opt/stalwart-mail ./stalwart-backup-$(date +%Y%m%d)

# Clean up helper pod
kubectl delete pod backup-helper -n mail

# Restore Stalwart
kubectl scale deployment stalwart -n mail --replicas=1

Option C: Stalwart Admin API Backup (Config Only)

Backup the configuration without downtime:

# Backup config via Stalwart admin API
curl -u admin:PASSWORD https://mail.paperclip.cloud/api/store/backup \
  -o stalwart-config-backup-$(date +%Y%m%d).zip

Restore Procedures

Restore from Volume Snapshot

1. Create a new volume from the snapshot:

   bin/hcloud volume create --name stalwart-restore --size 20 \
     --snapshot <snapshot-id> --location fsn1
   

2. Scale down Stalwart:

   kubectl scale deployment stalwart -n mail --replicas=0
   

3. Delete old PVC (after backing up the PV name):

   PV_NAME=$(kubectl get pvc stalwart-data -n mail -o jsonpath='{.spec.volumeName}')
   kubectl delete pvc stalwart-data -n mail
   

4. Create PV pointing to restored volume:

   RESTORED_VOLUME_ID=<new-volume-id>
   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: stalwart-restored
   spec:
     capacity:
       storage: 20Gi
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Retain
     storageClassName: hcloud-volumes-encrypted
     csi:
       driver: csi.hetzner.cloud
       volumeHandle: "$RESTORED_VOLUME_ID"
   EOF
   

5. Create PVC bound to restored PV:

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: stalwart-data
     namespace: mail
   spec:
     accessModes:
       - ReadWriteOnce
     storageClassName: hcloud-volumes-encrypted
     resources:
       requests:
         storage: 20Gi
     volumeName: stalwart-restored
   EOF
   

6. Scale Stalwart back up:

   kubectl scale deployment stalwart -n mail --replicas=1
   kubectl rollout status deployment stalwart -n mail
   

7. Verify restoration:

   kubectl logs -n mail deployment/stalwart --tail=20
   curl -k https://mail.paperclip.cloud/api/principal -u admin:PASSWORD
   

Restore from Filesystem Backup

# Scale down Stalwart
kubectl scale deployment stalwart -n mail --replicas=0

# Create restore helper pod
kubectl run restore-helper --image=alpine --restart=Never \
  -n mail \
  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"stalwart-data"}}],"containers":[{"name":"restore-helper","image":"alpine","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/opt/stalwart-mail"}]}]}}'

# Wait for pod
kubectl wait pod restore-helper -n mail --for=condition=Ready

# Clear existing data and restore
kubectl exec -n mail restore-helper -- rm -rf /opt/stalwart-mail/*
kubectl cp ./stalwart-backup-YYYYMMDD/. mail/restore-helper:/opt/stalwart-mail/

# Clean up helper
kubectl delete pod restore-helper -n mail

# Scale back up
kubectl scale deployment stalwart -n mail --replicas=1

Snapshot Retention Policy

Recommended retention:

• Daily snapshots: Keep 7 days
• Weekly snapshots: Keep 4 weeks
• Monthly snapshots: Keep 6 months

Hetzner snapshots are billed at €0.01/GB/month, so a 20GB volume costs €0.20/month per snapshot.

Testing Backup/Restore

Test the restore procedure quarterly:

1. Create a snapshot
2. Create a new volume from the snapshot in a test namespace
3. Deploy a test Stalwart instance pointing to the restored volume
4. Verify mail data is accessible via admin API
5. Delete test resources

# Verification test command
kubectl run test-restore --image=curlimages/curl --restart=Never \
  -n mail -- curl -k -u admin:PASSWORD \
  https://mail.paperclip.cloud/api/principal
kubectl logs test-restore -n mail
kubectl delete pod test-restore -n mail

Recovery Time Objectives

Scenario	RTO	RPO
Pod crash	~30 seconds	0 (persistent volume)
Node failure	~2 minutes	0 (PVC reattaches)
Volume corruption	30-60 minutes	<24 hours (last snapshot)
Data center failure	1-2 hours	<24 hours (manual restore to new region)