stack.basicstack.de/apps/stalwart/stalwart-backup-restore.md

# Stalwart Mail Server — Backup & Restore Procedures

## Overview

Stalwart stores all data in a RocksDB database mounted at `/opt/stalwart-mail` on a 20Gi encrypted hcloud volume (`hcloud-volumes-encrypted` storage class).

## Backup Strategy

### Option A: Hetzner Volume Snapshots (Recommended)

Hetzner Cloud provides volume snapshots that capture the full encrypted volume state.

#### Manual Snapshot via hcloud CLI

```bash
# Get the volume ID
bin/hcloud volume list | grep stalwart

# Create a snapshot (works even while volume is mounted — RocksDB is crash-safe)
bin/hcloud volume snapshot create <volume-id> --description "stalwart-backup-$(date +%Y%m%d)"
```

#### Scheduled Snapshots (Daily)

Create a CronJob in Kubernetes to automate snapshots:

```yaml
# Requires hcloud CLI and API token in a secret
apiVersion: batch/v1
kind: CronJob
metadata:
  name: stalwart-snapshot
  namespace: mail
spec:
  schedule: "0 3 * * *"  # 3 AM daily
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: hcloud-snapshot
            image: alpine:latest
            command:
            - /bin/sh
            - -c
            - |
              apk add --no-cache curl jq
              VOLUME_ID=$(curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \
                https://api.hetzner.cloud/v1/volumes | \
                jq -r '.volumes[] | select(.name | contains("stalwart")) | .id')
              curl -X POST -H "Authorization: Bearer $HCLOUD_TOKEN" \
                -H "Content-Type: application/json" \
                -d "{\"description\": \"stalwart-auto-$(date +%Y%m%d)\"}" \
                https://api.hetzner.cloud/v1/volumes/$VOLUME_ID/actions/create_snapshot
            env:
            - name: HCLOUD_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hcloud-credentials
                  key: token
```

### Option B: Filesystem-Level Backup

Back up the data directory while Stalwart is paused or using a consistent snapshot.

```bash
# Scale down Stalwart (brief downtime)
kubectl scale deployment stalwart -n mail --replicas=0

# Exec into a temporary pod with the same PVC
kubectl run backup-helper --image=alpine --restart=Never \
  -n mail \
  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"stalwart-data"}}],"containers":[{"name":"backup-helper","image":"alpine","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/opt/stalwart-mail"}]}]}}'

# Copy data out
kubectl cp mail/backup-helper:/opt/stalwart-mail ./stalwart-backup-$(date +%Y%m%d)

# Clean up helper pod
kubectl delete pod backup-helper -n mail

# Restore Stalwart
kubectl scale deployment stalwart -n mail --replicas=1
```

### Option C: Stalwart Admin API Backup (Config Only)

Backup the configuration without downtime:

```bash
# Backup config via Stalwart admin API
curl -u admin:PASSWORD https://mail.paperclip.cloud/api/store/backup \
  -o stalwart-config-backup-$(date +%Y%m%d).zip
```

## Restore Procedures

### Restore from Volume Snapshot

1. **Create a new volume from the snapshot**:
   ```bash
   bin/hcloud volume create --name stalwart-restore --size 20 \
     --snapshot <snapshot-id> --location fsn1
   ```

2. **Scale down Stalwart**:
   ```bash
   kubectl scale deployment stalwart -n mail --replicas=0
   ```

3. **Delete old PVC** (after backing up the PV name):
   ```bash
   PV_NAME=$(kubectl get pvc stalwart-data -n mail -o jsonpath='{.spec.volumeName}')
   kubectl delete pvc stalwart-data -n mail
   ```

4. **Create PV pointing to restored volume**:
   ```bash
   RESTORED_VOLUME_ID=<new-volume-id>
   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: stalwart-restored
   spec:
     capacity:
       storage: 20Gi
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Retain
     storageClassName: hcloud-volumes-encrypted
     csi:
       driver: csi.hetzner.cloud
       volumeHandle: "$RESTORED_VOLUME_ID"
   EOF
   ```

5. **Create PVC bound to restored PV**:
   ```bash
   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: stalwart-data
     namespace: mail
   spec:
     accessModes:
       - ReadWriteOnce
     storageClassName: hcloud-volumes-encrypted
     resources:
       requests:
         storage: 20Gi
     volumeName: stalwart-restored
   EOF
   ```

6. **Scale Stalwart back up**:
   ```bash
   kubectl scale deployment stalwart -n mail --replicas=1
   kubectl rollout status deployment stalwart -n mail
   ```

7. **Verify restoration**:
   ```bash
   kubectl logs -n mail deployment/stalwart --tail=20
   curl -k https://mail.paperclip.cloud/api/principal -u admin:PASSWORD
   ```

### Restore from Filesystem Backup

```bash
# Scale down Stalwart
kubectl scale deployment stalwart -n mail --replicas=0

# Create restore helper pod
kubectl run restore-helper --image=alpine --restart=Never \
  -n mail \
  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"stalwart-data"}}],"containers":[{"name":"restore-helper","image":"alpine","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/opt/stalwart-mail"}]}]}}'

# Wait for pod
kubectl wait pod restore-helper -n mail --for=condition=Ready

# Clear existing data and restore
kubectl exec -n mail restore-helper -- rm -rf /opt/stalwart-mail/*
kubectl cp ./stalwart-backup-YYYYMMDD/. mail/restore-helper:/opt/stalwart-mail/

# Clean up helper
kubectl delete pod restore-helper -n mail

# Scale back up
kubectl scale deployment stalwart -n mail --replicas=1
```

## Snapshot Retention Policy

Recommended retention:
- **Daily snapshots**: Keep 7 days
- **Weekly snapshots**: Keep 4 weeks  
- **Monthly snapshots**: Keep 6 months

Hetzner snapshots are billed at €0.01/GB/month, so a 20GB volume costs €0.20/month per snapshot.

## Testing Backup/Restore

Test the restore procedure quarterly:

1. Create a snapshot
2. Create a new volume from the snapshot in a test namespace
3. Deploy a test Stalwart instance pointing to the restored volume
4. Verify mail data is accessible via admin API
5. Delete test resources

```bash
# Verification test command
kubectl run test-restore --image=curlimages/curl --restart=Never \
  -n mail -- curl -k -u admin:PASSWORD \
  https://mail.paperclip.cloud/api/principal
kubectl logs test-restore -n mail
kubectl delete pod test-restore -n mail
```

## Recovery Time Objectives

| Scenario | RTO | RPO |
|----------|-----|-----|
| Pod crash | ~30 seconds | 0 (persistent volume) |
| Node failure | ~2 minutes | 0 (PVC reattaches) |
| Volume corruption | 30-60 minutes | <24 hours (last snapshot) |
| Data center failure | 1-2 hours | <24 hours (manual restore to new region) |