Add OpenCloud backup configuration to Hetzner Object Storage

Created automated daily backup system using rclone and Kubernetes CronJob.

Features:
- Daily backups at 2 AM UTC
- 7-day retention policy
- Backs up data directory and configuration
- Uses Hetzner S3-compatible Object Storage
- Read-only access to OpenCloud volumes

Files:
- backup-cronjob.yaml: CronJob for automated backups
- BACKUP.md: Complete setup and restore documentation

Requires:
- Hetzner Object Storage bucket credentials (sealed secret)
- S3 access key/secret to be provided

Once credentials are configured, backups will run automatically.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/opencloud/BACKUP.md

@@ -0,0 +1,174 @@
+# OpenCloud Backup Configuration
+
+## Overview
+
+Daily automated backup of OpenCloud data to Hetzner Object Storage using rclone.
+
+## Schedule
+
+- **Frequency:** Daily at 2:00 AM UTC
+- **Retention:** Last 7 days of backups
+- **Method:** rclone sync for data, rclone copy for config
+
+## What Gets Backed Up
+
+1. **Data Directory** (`/var/lib/opencloud/data`)
+   - User files
+   - Shares
+   - All uploaded content
+
+2. **Configuration** (`/etc/opencloud`)
+   - OpenCloud configuration files
+   - Note: Secrets are NOT backed up (stored in Kubernetes secrets)
+
+## Setup Required
+
+### 1. Create Hetzner Object Storage Bucket
+
+```bash
+# Create bucket via Hetzner Cloud Console or CLI
+# Bucket name: opencloud-backup
+# Region: fsn1 (or your preferred region)
+```
+
+### 2. Create S3 Access Credentials
+
+Generate S3-compatible access credentials from Hetzner Object Storage console.
+
+### 3. Create Sealed Secret
+
+```bash
+# Create unsealed secret first
+kubectl create secret generic hetzner-s3-credentials \
+  --from-literal=access-key='YOUR_ACCESS_KEY' \
+  --from-literal=secret-key='YOUR_SECRET_KEY' \
+  --namespace=opencloud \
+  --dry-run=client -o yaml > hetzner-s3-credentials.yaml
+
+# Seal it
+kubeseal --cert /tmp/sealed-secrets-cert.pem \
+  -f hetzner-s3-credentials.yaml \
+  -o yaml > hetzner-s3-credentials-sealed.yaml
+
+# Apply sealed secret
+kubectl apply -f hetzner-s3-credentials-sealed.yaml
+
+# Clean up unsealed secret
+rm hetzner-s3-credentials.yaml
+```
+
+### 4. Update Endpoint in backup-cronjob.yaml
+
+Replace `https://fsn1.your-objectstorage.com` with your actual Hetzner Object Storage endpoint:
+- fsn1: `https://fsn1.your-objectstorage.com`
+- nbg1: `https://nbg1.your-objectstorage.com`
+- hel1: `https://hel1.your-objectstorage.com`
+
+### 5. Deploy Backup CronJob
+
+```bash
+kubectl apply -f backup-cronjob.yaml
+```
+
+## Verification
+
+### Check CronJob Status
+
+```bash
+kubectl get cronjob -n opencloud
+kubectl get jobs -n opencloud
+```
+
+### View Backup Logs
+
+```bash
+# Get the latest backup job
+kubectl get jobs -n opencloud -l job-name=opencloud-backup
+
+# View logs
+kubectl logs -n opencloud job/opencloud-backup-<timestamp>
+```
+
+### Trigger Manual Backup
+
+```bash
+kubectl create job -n opencloud \
+  --from=cronjob/opencloud-backup \
+  opencloud-backup-manual-$(date +%s)
+```
+
+## Restore Procedure
+
+### Restore Data from Backup
+
+1. Scale down OpenCloud deployment:
+```bash
+kubectl scale deployment/opencloud -n opencloud --replicas=0
+```
+
+2. Create restore job:
+```bash
+kubectl run opencloud-restore \
+  --image=rclone/rclone:latest \
+  --restart=Never \
+  --namespace=opencloud \
+  --env="S3_ACCESS_KEY=YOUR_KEY" \
+  --env="S3_SECRET_KEY=YOUR_SECRET" \
+  --overrides='
+{
+  "spec": {
+    "containers": [{
+      "name": "restore",
+      "image": "rclone/rclone:latest",
+      "command": ["/bin/sh", "-c"],
+      "args": ["
+        cat > /root/.config/rclone/rclone.conf <<EOC
+        [hetzner-s3]
+        type = s3
+        provider = Other
+        access_key_id = ${S3_ACCESS_KEY}
+        secret_access_key = ${S3_SECRET_KEY}
+        endpoint = https://fsn1.your-objectstorage.com
+        region = fsn1
+        EOC
+        rclone sync hetzner-s3:opencloud-backup/data-YYYYMMDD-HHMMSS /data
+      "],
+      "volumeMounts": [{
+        "name": "data",
+        "mountPath": "/data"
+      }]
+    }],
+    "volumes": [{
+      "name": "data",
+      "persistentVolumeClaim": {
+        "claimName": "opencloud-data"
+      }
+    }]
+  }
+}'
+```
+
+3. Scale up OpenCloud:
+```bash
+kubectl scale deployment/opencloud -n opencloud --replicas=1
+```
+
+## Monitoring
+
+Monitor backup success/failure via:
+- Kubernetes Job status
+- Backup pod logs
+- Hetzner Object Storage bucket size/contents
+
+## Estimated Costs
+
+- **Storage:** ~€0.0059/GB/month (Hetzner Object Storage)
+- **Egress:** Free for first 1TB/month
+- **Example:** 50GB of data × 7 days retention = ~€2-3/month
+
+## Notes
+
+- Backups are encrypted in transit (HTTPS)
+- Backups at rest encryption depends on Hetzner Object Storage settings
+- Consider enabling versioning on the S3 bucket for additional protection
+- The backup job has read-only access to OpenCloud data

apps/opencloud/backup-cronjob.yaml

@@ -0,0 +1,92 @@
+---
+# OpenCloud Daily Backup to Hetzner Object Storage
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: opencloud-backup
+  namespace: opencloud
+spec:
+  schedule: "0 2 * * *"  # Daily at 2 AM UTC
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          restartPolicy: OnFailure
+          containers:
+          - name: backup
+            image: rclone/rclone:latest
+            command:
+              - /bin/sh
+              - -c
+              - |
+                set -e
+                DATE=$(date +%Y%m%d-%H%M%S)
+                echo "Starting OpenCloud backup at $DATE"
+                
+                # Configure rclone for Hetzner S3
+                cat > /root/.config/rclone/rclone.conf <<EOC
+                [hetzner-s3]
+                type = s3
+                provider = Other
+                access_key_id = ${S3_ACCESS_KEY}
+                secret_access_key = ${S3_SECRET_KEY}
+                endpoint = https://fsn1.your-objectstorage.com
+                region = fsn1
+                acl = private
+                EOC
+                
+                # Backup OpenCloud data directory
+                echo "Backing up data directory..."
+                rclone sync /data hetzner-s3:opencloud-backup/data-${DATE} \
+                  --progress \
+                  --transfers 8 \
+                  --checkers 8 \
+                  --stats 1m
+                
+                # Backup configuration
+                echo "Backing up configuration..."
+                rclone copy /config hetzner-s3:opencloud-backup/config-${DATE} \
+                  --progress
+                
+                # Keep only last 7 days of backups
+                echo "Cleaning old backups (keeping last 7 days)..."
+                rclone delete hetzner-s3:opencloud-backup \
+                  --min-age 7d \
+                  --rmdirs
+                
+                echo "Backup completed successfully at $(date)"
+            env:
+            - name: S3_ACCESS_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: hetzner-s3-credentials
+                  key: access-key
+            - name: S3_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: hetzner-s3-credentials
+                  key: secret-key
+            volumeMounts:
+            - name: opencloud-data
+              mountPath: /data
+              readOnly: true
+            - name: opencloud-config
+              mountPath: /config
+              readOnly: true
+            resources:
+              requests:
+                memory: "256Mi"
+                cpu: "100m"
+              limits:
+                memory: "512Mi"
+                cpu: "500m"
+          volumes:
+          - name: opencloud-data
+            persistentVolumeClaim:
+              claimName: opencloud-data
+          - name: opencloud-config
+            configMap:
+              name: opencloud-config