Initialize CD/CI repository structure with Stalwart example

Set up the repository structure following GitOps principles:
- apps/ for application deployments (Stalwart as example)
- infrastructure/ for cluster-wide configs (networking, monitoring)
- docs/ for general documentation

Migrated complete Stalwart deployment configuration including:
- Multiple deployment variants (basic, OIDC-enabled)
- Helm values files
- Monitoring and dashboard configurations
- Operational documentation (backup/restore, bootstrap)
- Configuration patches and fixes

Added comprehensive README files at each level to guide future use.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

README.md

@@ -1,3 +1,65 @@
 # stack.basicstack.de
 
-CD/CI deployment manifests and configurations for basicstack.de cluster
+CD/CI deployment manifests and configurations for the basicstack.de Kubernetes cluster.
+
+## Repository Structure
+
+```
+stack.basicstack.de/
+├── apps/                    # Application deployments
+│   ├── stalwart/           # Stalwart mail server (example)
+│   └── forgejo/            # Forgejo Git service (placeholder)
+├── infrastructure/          # Infrastructure-level configurations
+│   ├── networking/         # Network policies, ingress, DNS
+│   └── monitoring/         # Monitoring, logging, observability
+└── docs/                   # Documentation and guides
+```
+
+## Purpose
+
+This repository serves as the central source of truth for all deployment configurations targeting the `basicstack.de` Kubernetes cluster. It follows GitOps principles where infrastructure and application state is declaratively defined and version-controlled.
+
+## Directory Details
+
+### `apps/`
+Contains deployment configurations for individual applications and services running on the cluster. Each application should have its own subdirectory with:
+- Kubernetes manifests (Deployments, StatefulSets, Services, etc.)
+- Helm values files
+- Configuration files
+- Application-specific documentation
+
+**Example:** The `stalwart/` directory contains the complete deployment configuration for the Stalwart mail server, including multiple deployment variants, monitoring setup, and operational guides.
+
+### `infrastructure/`
+Contains cluster-wide infrastructure configurations:
+- **networking/**: Ingress controllers, network policies, DNS configurations, load balancers
+- **monitoring/**: Prometheus, Grafana, logging infrastructure, observability tools
+
+### `docs/`
+General documentation including:
+- Deployment procedures
+- Cluster architecture
+- Troubleshooting guides
+- Best practices
+
+## Getting Started
+
+1. Clone this repository
+2. Review the example Stalwart deployment in `apps/stalwart/`
+3. Follow the pattern for new application deployments
+4. Ensure all manifests are tested before committing
+
+## Contributing
+
+All changes should be:
+1. Committed with clear, descriptive messages
+2. Tested in a development environment when possible
+3. Documented appropriately
+4. Reviewed before deployment to production
+
+## Cluster Information
+
+- **Cluster**: basicstack.de
+- **Platform**: K3s on Hetzner Cloud
+- **Namespace Strategy**: One namespace per application (recommended)
+- **Ingress**: Traefik (default K3s ingress controller)

apps/README.md

@@ -0,0 +1,40 @@
+# Applications
+
+This directory contains deployment configurations for all applications running on the basicstack.de cluster.
+
+## Structure
+
+Each application should have its own subdirectory containing:
+
+- **Kubernetes manifests**: Deployment, StatefulSet, Service, ConfigMap, Secret definitions
+- **Helm values**: If using Helm charts, include values.yaml files
+- **Configuration files**: Application-specific configs (TOML, JSON, YAML)
+- **Documentation**: README or guide specific to the application deployment
+- **Patches**: Any kubectl patches or modifications needed
+
+## Example: Stalwart
+
+The `stalwart/` directory serves as a reference implementation, containing:
+- Multiple deployment variants (basic, with OIDC, etc.)
+- Helm values files
+- Monitoring dashboard configurations
+- Backup/restore procedures
+- Operational documentation
+
+## Adding a New Application
+
+1. Create a new directory: `apps/<application-name>/`
+2. Add your Kubernetes manifests
+3. Include a README.md explaining:
+   - What the application does
+   - How to deploy it
+   - Configuration options
+   - Troubleshooting steps
+4. Test the deployment in a dev environment
+5. Commit with a descriptive message
+
+## Naming Conventions
+
+- Directory names: lowercase, hyphen-separated (e.g., `my-app`)
+- Manifest files: descriptive names indicating resource type (e.g., `deployment.yaml`, `service.yaml`)
+- Use consistent naming across applications

apps/stalwart/README.md

@@ -0,0 +1,54 @@
+# Stalwart Mail Server Deployment
+
+This directory contains the complete deployment configuration for the Stalwart mail server on the basicstack.de cluster.
+
+## Files Overview
+
+### Deployment Manifests
+- `stalwart-deployment.yaml` - Basic deployment configuration
+- `stalwart-deployment-new.yaml` - Updated deployment variant
+- `stalwart-deployment-with-oidc.yaml` - Deployment with OIDC integration
+- `stalwart-deployment-oidc-only.yaml` - OIDC-only authentication deployment
+
+### Helm Configuration
+- `stalwart-values.yaml` - Main Helm values file
+- `stalwart-values-fixed-ports.yaml` - Values with corrected port configurations
+- `stalwart-values-correct.yaml` - Verified correct values
+- `stalwart-helm-values-oidc.yaml` - Helm values for OIDC setup
+- `stalwart-helm-fix.yaml` - Helm chart fixes
+
+### Configuration
+- `stalwart-oidc-config.yaml` - OIDC provider configuration
+- `stalwart-config-fix.yaml` - Configuration corrections
+- `stalwart-config-fix-v2.yaml` - Updated configuration fix
+
+### Monitoring
+- `stalwart-monitoring.yaml` - Prometheus ServiceMonitor and metrics
+- `stalwart-dashboard-configmap.yaml` - Grafana dashboard configuration
+
+### Maintenance
+- `stalwart-console-pod.yaml` - Debug/console pod for troubleshooting
+- `stalwart-service-patch.yaml` - Service configuration patch
+- `stalwart-statefulset-patch.yaml` - StatefulSet patches (v1, v2, v3)
+
+### Documentation
+- `stalwart-backup-restore.md` - Backup and restore procedures
+- `stalwart-bootstrap-completion-guide.md` - Initial setup guide
+
+## Deployment Notes
+
+This is a reference implementation showing the evolution of a production deployment. Multiple variants are preserved to show:
+- Different authentication strategies (local vs. OIDC)
+- Configuration iterations and fixes
+- Monitoring integration
+- Operational procedures
+
+When deploying a new service, you don't need this many files - choose the appropriate manifest for your use case and adapt it.
+
+## Current Production Deployment
+
+Review the files to determine which manifest represents the current production state, or check the cluster directly with:
+
+```bash
+kubectl get deployment,statefulset -n stalwart
+```

apps/stalwart/stalwart-backup-restore.md

@@ -0,0 +1,230 @@
+# Stalwart Mail Server — Backup & Restore Procedures
+
+## Overview
+
+Stalwart stores all data in a RocksDB database mounted at `/opt/stalwart-mail` on a 20Gi encrypted hcloud volume (`hcloud-volumes-encrypted` storage class).
+
+## Backup Strategy
+
+### Option A: Hetzner Volume Snapshots (Recommended)
+
+Hetzner Cloud provides volume snapshots that capture the full encrypted volume state.
+
+#### Manual Snapshot via hcloud CLI
+
+```bash
+# Get the volume ID
+bin/hcloud volume list | grep stalwart
+
+# Create a snapshot (works even while volume is mounted — RocksDB is crash-safe)
+bin/hcloud volume snapshot create <volume-id> --description "stalwart-backup-$(date +%Y%m%d)"
+```
+
+#### Scheduled Snapshots (Daily)
+
+Create a CronJob in Kubernetes to automate snapshots:
+
+```yaml
+# Requires hcloud CLI and API token in a secret
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: stalwart-snapshot
+  namespace: mail
+spec:
+  schedule: "0 3 * * *"  # 3 AM daily
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          restartPolicy: OnFailure
+          containers:
+          - name: hcloud-snapshot
+            image: alpine:latest
+            command:
+            - /bin/sh
+            - -c
+            - |
+              apk add --no-cache curl jq
+              VOLUME_ID=$(curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \
+                https://api.hetzner.cloud/v1/volumes | \
+                jq -r '.volumes[] | select(.name | contains("stalwart")) | .id')
+              curl -X POST -H "Authorization: Bearer $HCLOUD_TOKEN" \
+                -H "Content-Type: application/json" \
+                -d "{\"description\": \"stalwart-auto-$(date +%Y%m%d)\"}" \
+                https://api.hetzner.cloud/v1/volumes/$VOLUME_ID/actions/create_snapshot
+            env:
+            - name: HCLOUD_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: hcloud-credentials
+                  key: token
+```
+
+### Option B: Filesystem-Level Backup
+
+Back up the data directory while Stalwart is paused or using a consistent snapshot.
+
+```bash
+# Scale down Stalwart (brief downtime)
+kubectl scale deployment stalwart -n mail --replicas=0
+
+# Exec into a temporary pod with the same PVC
+kubectl run backup-helper --image=alpine --restart=Never \
+  -n mail \
+  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"stalwart-data"}}],"containers":[{"name":"backup-helper","image":"alpine","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/opt/stalwart-mail"}]}]}}'
+
+# Copy data out
+kubectl cp mail/backup-helper:/opt/stalwart-mail ./stalwart-backup-$(date +%Y%m%d)
+
+# Clean up helper pod
+kubectl delete pod backup-helper -n mail
+
+# Restore Stalwart
+kubectl scale deployment stalwart -n mail --replicas=1
+```
+
+### Option C: Stalwart Admin API Backup (Config Only)
+
+Backup the configuration without downtime:
+
+```bash
+# Backup config via Stalwart admin API
+curl -u admin:PASSWORD https://mail.paperclip.cloud/api/store/backup \
+  -o stalwart-config-backup-$(date +%Y%m%d).zip
+```
+
+## Restore Procedures
+
+### Restore from Volume Snapshot
+
+1. **Create a new volume from the snapshot**:
+   ```bash
+   bin/hcloud volume create --name stalwart-restore --size 20 \
+     --snapshot <snapshot-id> --location fsn1
+   ```
+
+2. **Scale down Stalwart**:
+   ```bash
+   kubectl scale deployment stalwart -n mail --replicas=0
+   ```
+
+3. **Delete old PVC** (after backing up the PV name):
+   ```bash
+   PV_NAME=$(kubectl get pvc stalwart-data -n mail -o jsonpath='{.spec.volumeName}')
+   kubectl delete pvc stalwart-data -n mail
+   ```
+
+4. **Create PV pointing to restored volume**:
+   ```bash
+   RESTORED_VOLUME_ID=<new-volume-id>
+   kubectl apply -f - <<EOF
+   apiVersion: v1
+   kind: PersistentVolume
+   metadata:
+     name: stalwart-restored
+   spec:
+     capacity:
+       storage: 20Gi
+     accessModes:
+       - ReadWriteOnce
+     persistentVolumeReclaimPolicy: Retain
+     storageClassName: hcloud-volumes-encrypted
+     csi:
+       driver: csi.hetzner.cloud
+       volumeHandle: "$RESTORED_VOLUME_ID"
+   EOF
+   ```
+
+5. **Create PVC bound to restored PV**:
+   ```bash
+   kubectl apply -f - <<EOF
+   apiVersion: v1
+   kind: PersistentVolumeClaim
+   metadata:
+     name: stalwart-data
+     namespace: mail
+   spec:
+     accessModes:
+       - ReadWriteOnce
+     storageClassName: hcloud-volumes-encrypted
+     resources:
+       requests:
+         storage: 20Gi
+     volumeName: stalwart-restored
+   EOF
+   ```
+
+6. **Scale Stalwart back up**:
+   ```bash
+   kubectl scale deployment stalwart -n mail --replicas=1
+   kubectl rollout status deployment stalwart -n mail
+   ```
+
+7. **Verify restoration**:
+   ```bash
+   kubectl logs -n mail deployment/stalwart --tail=20
+   curl -k https://mail.paperclip.cloud/api/principal -u admin:PASSWORD
+   ```
+
+### Restore from Filesystem Backup
+
+```bash
+# Scale down Stalwart
+kubectl scale deployment stalwart -n mail --replicas=0
+
+# Create restore helper pod
+kubectl run restore-helper --image=alpine --restart=Never \
+  -n mail \
+  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"stalwart-data"}}],"containers":[{"name":"restore-helper","image":"alpine","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/opt/stalwart-mail"}]}]}}'
+
+# Wait for pod
+kubectl wait pod restore-helper -n mail --for=condition=Ready
+
+# Clear existing data and restore
+kubectl exec -n mail restore-helper -- rm -rf /opt/stalwart-mail/*
+kubectl cp ./stalwart-backup-YYYYMMDD/. mail/restore-helper:/opt/stalwart-mail/
+
+# Clean up helper
+kubectl delete pod restore-helper -n mail
+
+# Scale back up
+kubectl scale deployment stalwart -n mail --replicas=1
+```
+
+## Snapshot Retention Policy
+
+Recommended retention:
+- **Daily snapshots**: Keep 7 days
+- **Weekly snapshots**: Keep 4 weeks  
+- **Monthly snapshots**: Keep 6 months
+
+Hetzner snapshots are billed at €0.01/GB/month, so a 20GB volume costs €0.20/month per snapshot.
+
+## Testing Backup/Restore
+
+Test the restore procedure quarterly:
+
+1. Create a snapshot
+2. Create a new volume from the snapshot in a test namespace
+3. Deploy a test Stalwart instance pointing to the restored volume
+4. Verify mail data is accessible via admin API
+5. Delete test resources
+
+```bash
+# Verification test command
+kubectl run test-restore --image=curlimages/curl --restart=Never \
+  -n mail -- curl -k -u admin:PASSWORD \
+  https://mail.paperclip.cloud/api/principal
+kubectl logs test-restore -n mail
+kubectl delete pod test-restore -n mail
+```
+
+## Recovery Time Objectives
+
+| Scenario | RTO | RPO |
+|----------|-----|-----|
+| Pod crash | ~30 seconds | 0 (persistent volume) |
+| Node failure | ~2 minutes | 0 (PVC reattaches) |
+| Volume corruption | 30-60 minutes | <24 hours (last snapshot) |
+| Data center failure | 1-2 hours | <24 hours (manual restore to new region) |

apps/stalwart/stalwart-bootstrap-completion-guide.md

@@ -0,0 +1,162 @@
+# Stalwart 0.16 Bootstrap Wizard Completion Guide
+
+## Current Status
+
+- ✅ Pod running: `stalwart-0` in namespace `stalwart`
+- ✅ JMAP API accessible (tested)
+- ✅ OIDC credentials prepared in Kubernetes secret
+- ⏳ Bootstrap wizard awaiting completion
+
+## Bootstrap Access Credentials
+
+**URL**: https://mail.basicstack.de/admin  
+**Username**: `admin`  
+**Password**: `YFMySjQYfMB3tYZa`
+
+> **Note**: These credentials are valid only until the bootstrap wizard is completed, then they will be automatically disabled.
+
+## OIDC Configuration Details
+
+Retrieved from Kubernetes secret `stalwart-oidc` in namespace `stalwart`:
+
+```
+Client ID: 0f37a0e3-8d3b-4413-a394-36226f42a980
+Client Secret: LPo8VejJXznisTQ87TGAs4Ad0Typ1MJw
+Issuer URL: https://auth.basicstack.de
+Redirect URI: https://mail.basicstack.de/admin/oauth/callback
+Scopes: openid profile email
+```
+
+## Step-by-Step Bootstrap Wizard Completion
+
+### Step 1: Access Bootstrap Interface
+
+1. Open browser to: https://mail.basicstack.de/admin
+2. Login with bootstrap credentials:
+   - Username: `admin`
+   - Password: `YFMySjQYfMB3tYZa`
+
+### Step 2: Configure Data Store
+
+The wizard should show the data store configuration. This is likely auto-configured:
+
+- **Type**: RocksDB
+- **Path**: `/opt/stalwart-mail/data`
+
+Verify the settings and proceed.
+
+### Step 3: Set Up OIDC Authentication
+
+Configure OAuth/OIDC provider with these exact settings:
+
+| Field | Value |
+|-------|-------|
+| Provider Name | Pocket ID |
+| Issuer URL | `https://auth.basicstack.de` |
+| Client ID | `0f37a0e3-8d3b-4413-a394-36226f42a980` |
+| Client Secret | `LPo8VejJXznisTQ87TGAs4Ad0Typ1MJw` |
+| Redirect URI | `https://mail.basicstack.de/admin/oauth/callback` |
+| Scopes | `openid profile email` |
+
+**Authorization Endpoint** (auto-discovered): `https://auth.basicstack.de/api/oidc/authorize`  
+**Token Endpoint** (auto-discovered): `https://auth.basicstack.de/api/oidc/token`  
+**UserInfo Endpoint** (auto-discovered): `https://auth.basicstack.de/api/oidc/userinfo`
+
+### Step 4: Create/Link Administrator Account
+
+Choose **Option A** (recommended): Link to Pocket ID user
+
+- When prompted, link the admin account to a Pocket ID user
+- The Pocket ID OIDC client is already configured with group restrictions
+- Only users in the `Stalwart-admin` group can access
+
+If the wizard requires creating an internal admin first:
+- Create a temporary internal admin
+- Link it to OIDC
+- The bootstrap password login will be automatically disabled after setup
+
+### Step 5: Disable Password Authentication (if prompted)
+
+- **Web UI Authentication**: OIDC only
+- **Mail Client Authentication**: Uses internal directory (separate from web UI auth)
+
+This ensures the web admin interface only accepts OIDC login while mail clients continue to work normally.
+
+### Step 6: Complete Setup
+
+1. Review all configuration
+2. Click "Complete Setup" or equivalent final button
+3. The system will:
+   - Save configuration to RocksDB
+   - Exit bootstrap mode
+   - Disable the bootstrap credentials
+   - Enable OIDC authentication
+
+### Step 7: Verify OIDC Login
+
+1. Log out from the bootstrap session
+2. Access https://mail.basicstack.de/admin again
+3. Click "Login with Pocket ID" or the OAuth login button
+4. Should redirect to https://auth.basicstack.de for authentication
+5. After successful Pocket ID login, should return to Stalwart admin interface
+
+## Post-Bootstrap Verification
+
+Run these commands to verify the configuration:
+
+```bash
+# Check pod is still running
+export KUBECONFIG=/paperclip/instances/default/workspaces/b4536334-39f2-4e05-b2f1-bb0e4670fba8/k3s.kubeconfig
+/paperclip/instances/default/workspaces/b4536334-39f2-4e05-b2f1-bb0e4670fba8/bin/kubectl get pods -n stalwart
+
+# Test JMAP API (should still work with OIDC credentials now)
+curl -s -X POST https://mail.basicstack.de/jmap \
+  -H "Content-Type: application/json" \
+  -d '{"using":["urn:ietf:params:jmap:core"],"methodCalls":[["Core/echo",{"test":"post-bootstrap"},"0"]]}'
+
+# Verify management API is now accessible
+curl -s https://mail.basicstack.de/api/session
+```
+
+## Troubleshooting
+
+### Issue: Cannot access bootstrap UI
+- Verify pod is running: `kubectl get pods -n stalwart`
+- Check pod logs: `kubectl logs stalwart-0 -n stalwart`
+- Verify ingress: `kubectl get ingress -n stalwart`
+
+### Issue: OIDC login not working after setup
+- Check Stalwart logs for OAuth errors
+- Verify redirect URI matches exactly
+- Confirm user is in `Stalwart-admin` group in Pocket ID
+- Test Pocket ID OIDC client directly
+
+### Issue: Bootstrap credentials don't work
+- Pod may have been restarted and new credentials generated
+- Check parent issue DEV-155 for any updates
+- May need to restart the pod or check the deployment
+
+## Security Notes
+
+- Bootstrap credentials automatically expire after setup completion
+- OIDC client secret is stored in Kubernetes secret `stalwart-oidc`
+- Access restricted to `Stalwart-admin` group members only
+- All communication over HTTPS/TLS
+
+## References
+
+- Parent Issue: DEV-155
+- OIDC Client configured in Pocket ID (client ID: 0f37a0e3-8d3b-4413-a394-36226f42a980)
+- User Group: `Stalwart-admin` (group ID: 99ffc3ae-0112-4fa0-bec9-90da19bdaddd)
+- Pocket ID: https://auth.basicstack.de
+- Stalwart: https://mail.basicstack.de
+
+## Acceptance Criteria Checklist
+
+- [ ] Bootstrap setup wizard completed
+- [ ] OIDC authentication configured with Pocket ID
+- [ ] Can login to web UI via Pocket ID
+- [ ] Password login disabled for web UI
+- [ ] Bootstrap mode exited
+- [ ] Configuration saved to RocksDB
+- [ ] Management API responding (not "no available server")

apps/stalwart/stalwart-config-complete.json

@@ -0,0 +1,78 @@
+{
+  "store": {
+    "data": {
+      "type": "rocksdb",
+      "path": "/var/lib/stalwart/data"
+    },
+    "blob": {
+      "type": "rocksdb",
+      "path": "/var/lib/stalwart/blobs"
+    }
+  },
+  "directory": {
+    "internal": {
+      "type": "internal",
+      "store": "data"
+    }
+  },
+  "server": {
+    "hostname": "mail.basicstack.de",
+    "listener": {
+      "management": {
+        "bind": ["[::]:8080"],
+        "protocol": "http"
+      },
+      "smtp": {
+        "bind": ["[::]:25"],
+        "protocol": "smtp"
+      },
+      "submission": {
+        "bind": ["[::]:587"],
+        "protocol": "smtp"
+      },
+      "submissions": {
+        "bind": ["[::]:465"],
+        "protocol": "smtp",
+        "tls": {
+          "implicit": true
+        }
+      },
+      "imap": {
+        "bind": ["[::]:143"],
+        "protocol": "imap"
+      },
+      "imaps": {
+        "bind": ["[::]:993"],
+        "protocol": "imap",
+        "tls": {
+          "implicit": true
+        }
+      },
+      "sieve": {
+        "bind": ["[::]:4190"],
+        "protocol": "managesieve"
+      }
+    }
+  },
+  "session": {
+    "auth": {
+      "mechanisms": ["plain", "login"],
+      "directory": "internal"
+    }
+  },
+  "queue": {
+    "path": "/var/lib/stalwart/queue"
+  },
+  "report": {
+    "path": "/var/lib/stalwart/reports"
+  },
+  "resolver": {
+    "type": "system"
+  },
+  "certificate": {
+    "default": {
+      "cert": "/etc/stalwart/tls/tls.crt",
+      "private-key": "/etc/stalwart/tls/tls.key"
+    }
+  }
+}

apps/stalwart/stalwart-config-fix-v2.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: stalwart-stalwart-config
+  namespace: mail
+  labels:
+    app.kubernetes.io/instance: stalwart
+    app.kubernetes.io/name: stalwart
+data:
+  config.json: |
+    {
+      "@type": "RocksDb",
+      "path": "/var/lib/stalwart",
+      "compression": "lz4"
+    }

apps/stalwart/stalwart-config-fix.yaml

@@ -0,0 +1,66 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: stalwart-stalwart-config
+  namespace: mail
+  labels:
+    app.kubernetes.io/instance: stalwart
+    app.kubernetes.io/name: stalwart
+data:
+  config.json: |
+    {
+      "store": {
+        "db": {
+          "type": "rocksdb",
+          "path": "/var/lib/stalwart/data"
+        },
+        "blob": {
+          "type": "rocksdb",
+          "path": "/var/lib/stalwart/blobs"
+        }
+      },
+      "directory": {
+        "internal": {
+          "type": "internal",
+          "store": "db"
+        }
+      },
+      "authentication": {
+        "fallback-admin": {
+          "user": "admin",
+          "secret": "***REMOVED***"
+        }
+      },
+      "server": {
+        "hostname": "mail.basicstack.de",
+        "http": {
+          "bind": ["[::]:8080"],
+          "protocol": "http"
+        }
+      },
+      "session": {
+        "ehlo": {
+          "require": true
+        },
+        "auth": {
+          "directory": "internal"
+        }
+      },
+      "queue": {
+        "path": "/var/lib/stalwart/queue",
+        "hash": 64
+      }
+    }
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stalwart-stalwart-env
+  namespace: mail
+  labels:
+    app.kubernetes.io/instance: stalwart
+    app.kubernetes.io/name: stalwart
+type: Opaque
+stringData:
+  STALWART_LOG_LEVEL: "info"
+  STALWART_LOG_FORMAT: "json"

apps/stalwart/stalwart-config-with-data.json

@@ -0,0 +1,5 @@
+{
+  "@type": "RocksDb",
+  "path": "/var/lib/stalwart",
+  "compression": "lz4"
+}

apps/stalwart/stalwart-console-pod.yaml

@@ -0,0 +1,21 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: stalwart-console
+  namespace: mail
+spec:
+  containers:
+  - name: stalwart
+    image: stalwartlabs/stalwart:v0.16.9
+    command: ["/bin/sh", "-c", "sleep 3600"]
+    volumeMounts:
+    - name: data
+      mountPath: /var/lib/stalwart
+    securityContext:
+      runAsUser: 2000
+      runAsGroup: 2000
+  volumes:
+  - name: data
+    persistentVolumeClaim:
+      claimName: data-stalwart-stalwart-0
+  restartPolicy: Never

apps/stalwart/stalwart-dashboard-configmap.yaml

@@ -0,0 +1,593 @@
+apiVersion: v1
+data:
+  stalwart-mail.json: |
+    {
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "id": null,
+      "links": [],
+      "panels": [
+        {
+          "datasource": "Prometheus",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "mappings": [
+                {
+                  "options": {
+                    "0": {
+                      "color": "red",
+                      "index": 0,
+                      "text": "Down"
+                    },
+                    "1": {
+                      "color": "green",
+                      "index": 1,
+                      "text": "Running"
+                    }
+                  },
+                  "type": "value"
+                }
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "red",
+                    "value": null
+                  },
+                  {
+                    "color": "green",
+                    "value": 1
+                  }
+                ]
+              },
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 4,
+            "w": 6,
+            "x": 0,
+            "y": 0
+          },
+          "id": 1,
+          "options": {
+            "colorMode": "background",
+            "graphMode": "none",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "8.0.0",
+          "targets": [
+            {
+              "expr": "kube_pod_status_phase{namespace=\"mail\", pod=~\"stalwart-.*\", phase=\"Running\"}",
+              "refId": "A"
+            }
+          ],
+          "title": "Pod Status",
+          "type": "stat"
+        },
+        {
+          "datasource": "Prometheus",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "yellow",
+                    "value": 1
+                  },
+                  {
+                    "color": "red",
+                    "value": 5
+                  }
+                ]
+              },
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 4,
+            "w": 6,
+            "x": 6,
+            "y": 0
+          },
+          "id": 2,
+          "options": {
+            "colorMode": "background",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "8.0.0",
+          "targets": [
+            {
+              "expr": "kube_pod_container_status_restarts_total{namespace=\"mail\", pod=~\"stalwart-.*\"}",
+              "refId": "A"
+            }
+          ],
+          "title": "Container Restarts",
+          "type": "stat"
+        },
+        {
+          "datasource": "Prometheus",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "mappings": [],
+              "max": 100,
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "yellow",
+                    "value": 75
+                  },
+                  {
+                    "color": "red",
+                    "value": 90
+                  }
+                ]
+              },
+              "unit": "percent"
+            }
+          },
+          "gridPos": {
+            "h": 4,
+            "w": 6,
+            "x": 12,
+            "y": 0
+          },
+          "id": 3,
+          "options": {
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "showThresholdLabels": false,
+            "showThresholdMarkers": true
+          },
+          "pluginVersion": "8.0.0",
+          "targets": [
+            {
+              "expr": "(kubelet_volume_stats_used_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"} / kubelet_volume_stats_capacity_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}) * 100",
+              "refId": "A"
+            }
+          ],
+          "title": "Disk Usage (20Gi PVC)",
+          "type": "gauge"
+        },
+        {
+          "datasource": "Prometheus",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "mappings": [],
+              "max": 100,
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "yellow",
+                    "value": 75
+                  },
+                  {
+                    "color": "red",
+                    "value": 85
+                  }
+                ]
+              },
+              "unit": "percent"
+            }
+          },
+          "gridPos": {
+            "h": 4,
+            "w": 6,
+            "x": 18,
+            "y": 0
+          },
+          "id": 4,
+          "options": {
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "showThresholdLabels": false,
+            "showThresholdMarkers": true
+          },
+          "pluginVersion": "8.0.0",
+          "targets": [
+            {
+              "expr": "(container_memory_working_set_bytes{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"} / container_spec_memory_limit_bytes{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"}) * 100",
+              "refId": "A"
+            }
+          ],
+          "title": "Memory Usage",
+          "type": "gauge"
+        },
+        {
+          "datasource": "Prometheus",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "tooltip": false,
+                  "viz": false,
+                  "legend": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  }
+                ]
+              },
+              "unit": "percentunit"
+            }
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 12,
+            "x": 0,
+            "y": 4
+          },
+          "id": 5,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "multi"
+            }
+          },
+          "pluginVersion": "8.0.0",
+          "targets": [
+            {
+              "expr": "rate(container_cpu_usage_seconds_total{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"}[5m]) / (container_spec_cpu_quota{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"} / container_spec_cpu_period{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"})",
+              "legendFormat": "CPU Usage",
+              "refId": "A"
+            }
+          ],
+          "title": "CPU Usage Over Time",
+          "type": "timeseries"
+        },
+        {
+          "datasource": "Prometheus",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "tooltip": false,
+                  "viz": false,
+                  "legend": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  }
+                ]
+              },
+              "unit": "bytes"
+            }
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 12,
+            "x": 12,
+            "y": 4
+          },
+          "id": 6,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "multi"
+            }
+          },
+          "pluginVersion": "8.0.0",
+          "targets": [
+            {
+              "expr": "container_memory_working_set_bytes{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"}",
+              "legendFormat": "Memory Usage",
+              "refId": "A"
+            },
+            {
+              "expr": "container_spec_memory_limit_bytes{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"}",
+              "legendFormat": "Memory Limit",
+              "refId": "B"
+            }
+          ],
+          "title": "Memory Usage Over Time",
+          "type": "timeseries"
+        },
+        {
+          "datasource": "Prometheus",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "tooltip": false,
+                  "viz": false,
+                  "legend": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  }
+                ]
+              },
+              "unit": "bytes"
+            }
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 12,
+            "x": 0,
+            "y": 12
+          },
+          "id": 7,
+          "options": {
+            "legend": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "displayMode": "table",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "multi"
+            }
+          },
+          "pluginVersion": "8.0.0",
+          "targets": [
+            {
+              "expr": "kubelet_volume_stats_used_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+              "legendFormat": "Used Space",
+              "refId": "A"
+            },
+            {
+              "expr": "kubelet_volume_stats_capacity_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+              "legendFormat": "Total Capacity (20Gi)",
+              "refId": "B"
+            },
+            {
+              "expr": "kubelet_volume_stats_available_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+              "legendFormat": "Available Space",
+              "refId": "C"
+            }
+          ],
+          "title": "Disk Space Over Time (Encrypted Volume)",
+          "type": "timeseries"
+        },
+        {
+          "datasource": "Prometheus",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "tooltip": false,
+                  "viz": false,
+                  "legend": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  }
+                ]
+              },
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 12,
+            "x": 12,
+            "y": 12
+          },
+          "id": 8,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "multi"
+            }
+          },
+          "pluginVersion": "8.0.0",
+          "targets": [
+            {
+              "expr": "kubelet_volume_stats_inodes_used{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+              "legendFormat": "Inodes Used",
+              "refId": "A"
+            },
+            {
+              "expr": "kubelet_volume_stats_inodes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+              "legendFormat": "Total Inodes",
+              "refId": "B"
+            }
+          ],
+          "title": "Disk Inodes Usage",
+          "type": "timeseries"
+        }
+      ],
+      "refresh": "30s",
+      "schemaVersion": 27,
+      "style": "dark",
+      "tags": ["stalwart", "mail", "kubernetes"],
+      "templating": {
+        "list": []
+      },
+      "time": {
+        "from": "now-6h",
+        "to": "now"
+      },
+      "timepicker": {},
+      "timezone": "",
+      "title": "Stalwart Mail Server",
+      "uid": "stalwart-mail",
+      "version": 1
+    }
+kind: ConfigMap
+metadata:
+  name: stalwart-dashboard
+  namespace: observability
+  labels:
+    grafana_dashboard: "1"

apps/stalwart/stalwart-deployment-new.yaml

@@ -0,0 +1,237 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: mail
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: stalwart-data
+  namespace: mail
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: hcloud-volumes-encrypted
+  resources:
+    requests:
+      storage: 20Gi
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stalwart-admin
+  namespace: mail
+type: Opaque
+stringData:
+  admin-password: "***REMOVED***"
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart
+  namespace: mail
+spec:
+  type: ClusterIP
+  clusterIP: None
+  selector:
+    app: stalwart
+  ports:
+    - name: smtp
+      port: 25
+      targetPort: 25
+      protocol: TCP
+    - name: submission
+      port: 587
+      targetPort: 587
+      protocol: TCP
+    - name: submissions
+      port: 465
+      targetPort: 465
+      protocol: TCP
+    - name: imap
+      port: 143
+      targetPort: 143
+      protocol: TCP
+    - name: imaps
+      port: 993
+      targetPort: 993
+      protocol: TCP
+    - name: http
+      port: 8080
+      targetPort: 8080
+      protocol: TCP
+    - name: sieve
+      port: 4190
+      targetPort: 4190
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-smtp
+  namespace: mail
+spec:
+  type: LoadBalancer
+  selector:
+    app: stalwart
+  ports:
+    - name: smtp
+      port: 25
+      targetPort: 25
+      protocol: TCP
+    - name: submission
+      port: 587
+      targetPort: 587
+      protocol: TCP
+    - name: submissions
+      port: 465
+      targetPort: 465
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-imap
+  namespace: mail
+spec:
+  type: LoadBalancer
+  selector:
+    app: stalwart
+  ports:
+    - name: imap
+      port: 143
+      targetPort: 143
+      protocol: TCP
+    - name: imaps
+      port: 993
+      targetPort: 993
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-http
+  namespace: mail
+spec:
+  type: ClusterIP
+  selector:
+    app: stalwart
+  ports:
+    - name: http
+      port: 8080
+      targetPort: 8080
+      protocol: TCP
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: stalwart
+  namespace: mail
+spec:
+  serviceName: stalwart
+  replicas: 1
+  selector:
+    matchLabels:
+      app: stalwart
+  template:
+    metadata:
+      labels:
+        app: stalwart
+    spec:
+      securityContext:
+        fsGroup: 2000
+        runAsUser: 2000
+        runAsGroup: 2000
+      initContainers:
+      - name: fix-permissions
+        image: busybox
+        command: ["sh", "-c", "chown -R 2000:2000 /opt/stalwart && chmod -R 755 /opt/stalwart"]
+        volumeMounts:
+        - name: data
+          mountPath: /opt/stalwart
+      containers:
+      - name: stalwart
+        image: stalwartlabs/stalwart:latest
+        ports:
+        - containerPort: 25
+          name: smtp
+        - containerPort: 587
+          name: submission
+        - containerPort: 465
+          name: submissions
+        - containerPort: 143
+          name: imap
+        - containerPort: 993
+          name: imaps
+        - containerPort: 8080
+          name: http
+        - containerPort: 4190
+          name: sieve
+        volumeMounts:
+        - name: data
+          mountPath: /opt/stalwart
+        - name: data
+          mountPath: /etc/stalwart
+          subPath: etc
+        livenessProbe:
+          httpGet:
+            path: /healthz/live
+            port: 8080
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          timeoutSeconds: 5
+          failureThreshold: 3
+        readinessProbe:
+          httpGet:
+            path: /healthz/ready
+            port: 8080
+          initialDelaySeconds: 10
+          periodSeconds: 5
+          timeoutSeconds: 3
+          failureThreshold: 3
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "250m"
+          limits:
+            memory: "2Gi"
+            cpu: "2000m"
+        securityContext:
+          runAsNonRoot: true
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop: [ALL]
+            add: [NET_BIND_SERVICE]
+          seccompProfile:
+            type: RuntimeDefault
+      volumes:
+      - name: data
+        persistentVolumeClaim:
+          claimName: stalwart-data
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: stalwart-web
+  namespace: mail
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+spec:
+  ingressClassName: traefik
+  tls:
+  - hosts:
+    - mail.basicstack.de
+    secretName: stalwart-tls
+  rules:
+  - host: mail.basicstack.de
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: stalwart-http
+            port:
+              number: 8080

apps/stalwart/stalwart-deployment-oidc-only.yaml

@@ -0,0 +1,233 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: mail
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: stalwart-data
+  namespace: mail
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: hcloud-volumes-encrypted
+  resources:
+    requests:
+      storage: 20Gi
+---
+# OAuth Configuration ConfigMap
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: stalwart-oauth-config
+  namespace: mail
+data:
+  oauth-config.json: |
+    {
+      "oauth": {
+        "providers": [
+          {
+            "id": "pocket-id",
+            "name": "Pocket ID",
+            "issuer": "https://auth.basicstack.de",
+            "authorization-url": "https://auth.basicstack.de/api/oidc/authorize",
+            "token-url": "https://auth.basicstack.de/api/oidc/token",
+            "userinfo-url": "https://auth.basicstack.de/api/oidc/userinfo",
+            "client-id": "19432b61-e27e-444d-9ff0-0fa23fdac6d7",
+            "client-secret": "***REMOVED***",
+            "scopes": ["openid", "profile", "email"],
+            "redirect-url": "https://mail.basicstack.de/login/oauth",
+            "user-mapping": {
+              "username": "preferred_username",
+              "email": "email",
+              "name": "name"
+            }
+          }
+        ],
+        "enabled": true,
+        "allow-password-auth": false
+      }
+    }
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stalwart-oidc-secret
+  namespace: mail
+type: Opaque
+stringData:
+  client-id: "19432b61-e27e-444d-9ff0-0fa23fdac6d7"
+  client-secret: "***REMOVED***"
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-smtp
+  namespace: mail
+spec:
+  type: LoadBalancer
+  selector:
+    app: stalwart
+  ports:
+    - name: smtp
+      port: 25
+      targetPort: 25
+      protocol: TCP
+    - name: submission
+      port: 587
+      targetPort: 587
+      protocol: TCP
+    - name: submissions
+      port: 465
+      targetPort: 465
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-imap
+  namespace: mail
+spec:
+  type: LoadBalancer
+  selector:
+    app: stalwart
+  ports:
+    - name: imap
+      port: 143
+      targetPort: 143
+      protocol: TCP
+    - name: imaps
+      port: 993
+      targetPort: 993
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-http
+  namespace: mail
+spec:
+  type: ClusterIP
+  selector:
+    app: stalwart
+  ports:
+    - name: http
+      port: 8080
+      targetPort: 8080
+      protocol: TCP
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: stalwart
+  namespace: mail
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: stalwart
+  template:
+    metadata:
+      labels:
+        app: stalwart
+    spec:
+      initContainers:
+      - name: fix-permissions
+        image: busybox
+        command: ["sh", "-c", "chown -R 2000:2000 /opt/stalwart-mail && chmod -R 755 /opt/stalwart-mail"]
+        volumeMounts:
+        - name: data
+          mountPath: /opt/stalwart-mail
+      containers:
+      - name: stalwart
+        image: stalwartlabs/stalwart:v0.16
+        command: ["/usr/local/bin/stalwart"]
+        args: ["--config", "/etc/stalwart/config.json"]
+        ports:
+        - containerPort: 25
+          name: smtp
+        - containerPort: 587
+          name: submission
+        - containerPort: 465
+          name: submissions
+        - containerPort: 143
+          name: imap
+        - containerPort: 993
+          name: imaps
+        - containerPort: 8080
+          name: http
+        env:
+        # Emergency recovery admin (can be disabled after OIDC is working)
+        - name: STALWART_RECOVERY_ADMIN
+          value: "admin@basicstack.de:***REMOVED***"
+        # OAuth configuration
+        - name: STALWART_OAUTH_ENABLED
+          value: "true"
+        - name: STALWART_OAUTH_PROVIDER
+          value: "pocket-id"
+        - name: STALWART_OAUTH_CLIENT_ID
+          valueFrom:
+            secretKeyRef:
+              name: stalwart-oidc-secret
+              key: client-id
+        - name: STALWART_OAUTH_CLIENT_SECRET
+          valueFrom:
+            secretKeyRef:
+              name: stalwart-oidc-secret
+              key: client-secret
+        # Disable password authentication (except recovery admin)
+        - name: STALWART_PASSWORD_AUTH_ENABLED
+          value: "false"
+        volumeMounts:
+        - name: data
+          mountPath: /var/lib/stalwart
+        - name: data
+          mountPath: /etc/stalwart
+          subPath: etc
+        - name: oauth-config
+          mountPath: /etc/stalwart/oauth
+          readOnly: true
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "250m"
+          limits:
+            memory: "1Gi"
+            cpu: "1000m"
+      volumes:
+      - name: data
+        persistentVolumeClaim:
+          claimName: stalwart-data
+      - name: oauth-config
+        configMap:
+          name: stalwart-oauth-config
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: stalwart-web
+  namespace: mail
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+spec:
+  ingressClassName: traefik
+  tls:
+  - hosts:
+    - mail.basicstack.de
+    secretName: stalwart-tls
+  rules:
+  - host: mail.basicstack.de
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: stalwart-http
+            port:
+              number: 8080

apps/stalwart/stalwart-deployment-with-oidc.yaml

@@ -0,0 +1,203 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: mail
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: stalwart-data
+  namespace: mail
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: hcloud-volumes-encrypted
+  resources:
+    requests:
+      storage: 20Gi
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stalwart-admin
+  namespace: mail
+type: Opaque
+stringData:
+  admin-password: "***REMOVED***"
+---
+# OIDC Secret for Stalwart
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stalwart-oidc
+  namespace: mail
+type: Opaque
+stringData:
+  oidc-client-id: "19432b61-e27e-444d-9ff0-0fa23fdac6d7"
+  oidc-client-secret: "***REMOVED***"
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-smtp
+  namespace: mail
+spec:
+  type: LoadBalancer
+  selector:
+    app: stalwart
+  ports:
+    - name: smtp
+      port: 25
+      targetPort: 25
+      protocol: TCP
+    - name: submission
+      port: 587
+      targetPort: 587
+      protocol: TCP
+    - name: submissions
+      port: 465
+      targetPort: 465
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-imap
+  namespace: mail
+spec:
+  type: LoadBalancer
+  selector:
+    app: stalwart
+  ports:
+    - name: imap
+      port: 143
+      targetPort: 143
+      protocol: TCP
+    - name: imaps
+      port: 993
+      targetPort: 993
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-http
+  namespace: mail
+spec:
+  type: ClusterIP
+  selector:
+    app: stalwart
+  ports:
+    - name: http
+      port: 8080
+      targetPort: 8080
+      protocol: TCP
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: stalwart
+  namespace: mail
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: stalwart
+  template:
+    metadata:
+      labels:
+        app: stalwart
+    spec:
+      initContainers:
+      - name: fix-permissions
+        image: busybox
+        command: ["sh", "-c", "chown -R 2000:2000 /opt/stalwart-mail && chmod -R 755 /opt/stalwart-mail"]
+        volumeMounts:
+        - name: data
+          mountPath: /opt/stalwart-mail
+      containers:
+      - name: stalwart
+        image: stalwartlabs/stalwart:v0.16
+        command: ["/usr/local/bin/stalwart"]
+        args: ["--config", "/etc/stalwart/config.json"]
+        ports:
+        - containerPort: 25
+          name: smtp
+        - containerPort: 587
+          name: submission
+        - containerPort: 465
+          name: submissions
+        - containerPort: 143
+          name: imap
+        - containerPort: 993
+          name: imaps
+        - containerPort: 8080
+          name: http
+        env:
+        - name: STALWART_RECOVERY_ADMIN
+          value: "admin@basicstack.de:***REMOVED***"
+        # OIDC Configuration
+        - name: STALWART_OAUTH_ENABLE
+          value: "true"
+        - name: STALWART_OAUTH_ISSUER
+          value: "https://auth.basicstack.de"
+        - name: STALWART_OAUTH_CLIENT_ID
+          valueFrom:
+            secretKeyRef:
+              name: stalwart-oidc
+              key: oidc-client-id
+        - name: STALWART_OAUTH_CLIENT_SECRET
+          valueFrom:
+            secretKeyRef:
+              name: stalwart-oidc
+              key: oidc-client-secret
+        - name: STALWART_OAUTH_REDIRECT_URI
+          value: "https://mail.basicstack.de/login/oauth"
+        - name: STALWART_OAUTH_SCOPES
+          value: "openid profile email"
+        volumeMounts:
+        - name: data
+          mountPath: /var/lib/stalwart
+        - name: data
+          mountPath: /etc/stalwart
+          subPath: etc
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "250m"
+          limits:
+            memory: "1Gi"
+            cpu: "1000m"
+      volumes:
+      - name: data
+        persistentVolumeClaim:
+          claimName: stalwart-data
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: stalwart-web
+  namespace: mail
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+spec:
+  ingressClassName: traefik
+  tls:
+  - hosts:
+    - mail.basicstack.de
+    secretName: stalwart-tls
+  rules:
+  - host: mail.basicstack.de
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: stalwart-http
+            port:
+              number: 8080

apps/stalwart/stalwart-deployment.yaml

@@ -0,0 +1,173 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: mail
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: stalwart-data
+  namespace: mail
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: hcloud-volumes-encrypted
+  resources:
+    requests:
+      storage: 20Gi
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stalwart-admin
+  namespace: mail
+type: Opaque
+stringData:
+  admin-password: "***REMOVED***"
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-smtp
+  namespace: mail
+spec:
+  type: LoadBalancer
+  selector:
+    app: stalwart
+  ports:
+    - name: smtp
+      port: 25
+      targetPort: 25
+      protocol: TCP
+    - name: submission
+      port: 587
+      targetPort: 587
+      protocol: TCP
+    - name: submissions
+      port: 465
+      targetPort: 465
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-imap
+  namespace: mail
+spec:
+  type: LoadBalancer
+  selector:
+    app: stalwart
+  ports:
+    - name: imap
+      port: 143
+      targetPort: 143
+      protocol: TCP
+    - name: imaps
+      port: 993
+      targetPort: 993
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: stalwart-http
+  namespace: mail
+spec:
+  type: ClusterIP
+  selector:
+    app: stalwart
+  ports:
+    - name: http
+      port: 8080
+      targetPort: 8080
+      protocol: TCP
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: stalwart
+  namespace: mail
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: stalwart
+  template:
+    metadata:
+      labels:
+        app: stalwart
+    spec:
+      initContainers:
+      - name: fix-permissions
+        image: busybox
+        command: ["sh", "-c", "chown -R 2000:2000 /opt/stalwart-mail && chmod -R 755 /opt/stalwart-mail"]
+        volumeMounts:
+        - name: data
+          mountPath: /opt/stalwart-mail
+      containers:
+      - name: stalwart
+        image: stalwartlabs/stalwart:v0.16
+        command: ["/usr/local/bin/stalwart"]
+        args: ["--config", "/etc/stalwart/config.json"]
+        ports:
+        - containerPort: 25
+          name: smtp
+        - containerPort: 587
+          name: submission
+        - containerPort: 465
+          name: submissions
+        - containerPort: 143
+          name: imap
+        - containerPort: 993
+          name: imaps
+        - containerPort: 8080
+          name: http
+        env:
+        - name: STALWART_RECOVERY_ADMIN
+          value: "admin@basicstack.de:***REMOVED***"
+        volumeMounts:
+        - name: data
+          mountPath: /var/lib/stalwart
+        - name: data
+          mountPath: /etc/stalwart
+          subPath: etc
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "250m"
+          limits:
+            memory: "1Gi"
+            cpu: "1000m"
+      volumes:
+      - name: data
+        persistentVolumeClaim:
+          claimName: stalwart-data
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: stalwart-web
+  namespace: mail
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+spec:
+  ingressClassName: traefik
+  tls:
+  - hosts:
+    - mail.basicstack.de
+    secretName: stalwart-tls
+  rules:
+  - host: mail.basicstack.de
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: stalwart-http
+            port:
+              number: 8080

apps/stalwart/stalwart-grafana-dashboard.json

@@ -0,0 +1,584 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": "-- Grafana --",
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "gnetId": null,
+  "graphTooltip": 0,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": "Prometheus",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [
+            {
+              "options": {
+                "0": {
+                  "color": "red",
+                  "index": 0,
+                  "text": "Down"
+                },
+                "1": {
+                  "color": "green",
+                  "index": 1,
+                  "text": "Running"
+                }
+              },
+              "type": "value"
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          },
+          "unit": "none"
+        }
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "8.0.0",
+      "targets": [
+        {
+          "expr": "kube_pod_status_phase{namespace=\"mail\", pod=~\"stalwart-.*\", phase=\"Running\"}",
+          "refId": "A"
+        }
+      ],
+      "title": "Pod Status",
+      "type": "stat"
+    },
+    {
+      "datasource": "Prometheus",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "none"
+        }
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 6,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "8.0.0",
+      "targets": [
+        {
+          "expr": "kube_pod_container_status_restarts_total{namespace=\"mail\", pod=~\"stalwart-.*\"}",
+          "refId": "A"
+        }
+      ],
+      "title": "Container Restarts",
+      "type": "stat"
+    },
+    {
+      "datasource": "Prometheus",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 75
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          },
+          "unit": "percent"
+        }
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 12,
+        "y": 0
+      },
+      "id": 3,
+      "options": {
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showThresholdLabels": false,
+        "showThresholdMarkers": true
+      },
+      "pluginVersion": "8.0.0",
+      "targets": [
+        {
+          "expr": "(kubelet_volume_stats_used_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"} / kubelet_volume_stats_capacity_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}) * 100",
+          "refId": "A"
+        }
+      ],
+      "title": "Disk Usage (20Gi PVC)",
+      "type": "gauge"
+    },
+    {
+      "datasource": "Prometheus",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 75
+              },
+              {
+                "color": "red",
+                "value": 85
+              }
+            ]
+          },
+          "unit": "percent"
+        }
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 6,
+        "x": 18,
+        "y": 0
+      },
+      "id": 4,
+      "options": {
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showThresholdLabels": false,
+        "showThresholdMarkers": true
+      },
+      "pluginVersion": "8.0.0",
+      "targets": [
+        {
+          "expr": "(container_memory_working_set_bytes{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"} / container_spec_memory_limit_bytes{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"}) * 100",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory Usage",
+      "type": "gauge"
+    },
+    {
+      "datasource": "Prometheus",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "tooltip": false,
+              "viz": false,
+              "legend": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "percentunit"
+        }
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      },
+      "pluginVersion": "8.0.0",
+      "targets": [
+        {
+          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"}[5m]) / (container_spec_cpu_quota{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"} / container_spec_cpu_period{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"})",
+          "legendFormat": "CPU Usage",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage Over Time",
+      "type": "timeseries"
+    },
+    {
+      "datasource": "Prometheus",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "tooltip": false,
+              "viz": false,
+              "legend": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "bytes"
+        }
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "id": 6,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      },
+      "pluginVersion": "8.0.0",
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"}",
+          "legendFormat": "Memory Usage",
+          "refId": "A"
+        },
+        {
+          "expr": "container_spec_memory_limit_bytes{namespace=\"mail\", pod=~\"stalwart-.*\", container=\"stalwart\"}",
+          "legendFormat": "Memory Limit",
+          "refId": "B"
+        }
+      ],
+      "title": "Memory Usage Over Time",
+      "type": "timeseries"
+    },
+    {
+      "datasource": "Prometheus",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "tooltip": false,
+              "viz": false,
+              "legend": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "bytes"
+        }
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 12
+      },
+      "id": 7,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      },
+      "pluginVersion": "8.0.0",
+      "targets": [
+        {
+          "expr": "kubelet_volume_stats_used_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+          "legendFormat": "Used Space",
+          "refId": "A"
+        },
+        {
+          "expr": "kubelet_volume_stats_capacity_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+          "legendFormat": "Total Capacity (20Gi)",
+          "refId": "B"
+        },
+        {
+          "expr": "kubelet_volume_stats_available_bytes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+          "legendFormat": "Available Space",
+          "refId": "C"
+        }
+      ],
+      "title": "Disk Space Over Time (Encrypted Volume)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": "Prometheus",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "tooltip": false,
+              "viz": false,
+              "legend": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "none"
+        }
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 12
+      },
+      "id": 8,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      },
+      "pluginVersion": "8.0.0",
+      "targets": [
+        {
+          "expr": "kubelet_volume_stats_inodes_used{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+          "legendFormat": "Inodes Used",
+          "refId": "A"
+        },
+        {
+          "expr": "kubelet_volume_stats_inodes{namespace=\"mail\", persistentvolumeclaim=\"stalwart-data\"}",
+          "legendFormat": "Total Inodes",
+          "refId": "B"
+        }
+      ],
+      "title": "Disk Inodes Usage",
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 27,
+  "style": "dark",
+  "tags": ["stalwart", "mail", "kubernetes"],
+  "templating": {
+    "list": []
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "",
+  "title": "Stalwart Mail Server",
+  "uid": "stalwart-mail",
+  "version": 1
+}

apps/stalwart/stalwart-helm-fix.yaml

@@ -0,0 +1,116 @@
+# Stalwart Helm Fix - Proper Environment Configuration
+image:
+  repository: stalwartlabs/stalwart
+  tag: "v0.16.9"
+  pullPolicy: Always
+
+# IMPORTANT: Remove args to let Stalwart use environment variables
+# The Helm chart should not pass --config if we want env-based config
+extraArgs: []
+
+# Enable recovery admin
+recoveryAdmin:
+  enabled: true
+  username: admin
+  password: ***REMOVED***
+
+# Persistence
+persistence:
+  enabled: true
+  storageClass: hcloud-volumes-encrypted
+  accessMode: ReadWriteOnce
+  size: 20Gi
+
+# Service configuration
+service:
+  type: LoadBalancer
+  ports:
+    smtp: 25
+    submission: 587
+    smtps: 465
+    imap: 143
+    imaps: 993
+    pop3: 110
+    pop3s: 995
+    sieve: 4190
+    mgmt: 8080
+
+# Ingress
+ingress:
+  enabled: true
+  className: traefik
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+  hosts:
+    - host: mail.basicstack.de
+      paths:
+        - path: /
+          pathType: Prefix
+          portName: mgmt
+  tls:
+    - secretName: stalwart-tls
+      hosts:
+        - mail.basicstack.de
+
+# Minimal config - just database location
+config:
+  "@type": "RocksDb"
+  path: "/var/lib/stalwart"
+
+# Security contexts
+podSecurityContext:
+  fsGroup: 2000
+  runAsUser: 2000
+  runAsGroup: 2000
+
+containerSecurityContext:
+  runAsNonRoot: true
+  allowPrivilegeEscalation: false
+  capabilities:
+    drop: [ALL]
+    add: [NET_BIND_SERVICE]
+  seccompProfile:
+    type: RuntimeDefault
+
+# Resources
+resources:
+  requests:
+    memory: "512Mi"
+    cpu: "250m"
+  limits:
+    memory: "2Gi"
+    cpu: "2"
+
+# Environment variables for proper configuration
+env:
+  - name: RUST_LOG
+    value: "debug"
+  - name: STALWART_LOG_LEVEL
+    value: "debug"
+
+# OIDC Configuration
+envFrom:
+  - secretRef:
+      name: stalwart-stalwart-env
+
+# Additional OIDC env vars
+extraEnv:
+  - name: STALWART_OAUTH_ENABLE
+    value: "true"
+  - name: STALWART_OAUTH_ISSUER
+    value: "https://auth.basicstack.de"
+  - name: STALWART_OAUTH_REDIRECT_URI
+    value: "https://mail.basicstack.de/admin/oauth/callback"
+  - name: STALWART_OAUTH_SCOPES
+    value: "openid profile email"
+  - name: STALWART_OAUTH_CLIENT_ID
+    valueFrom:
+      secretKeyRef:
+        name: stalwart-oidc
+        key: oidc-client-id
+  - name: STALWART_OAUTH_CLIENT_SECRET
+    valueFrom:
+      secretKeyRef:
+        name: stalwart-oidc
+        key: oidc-client-secret

apps/stalwart/stalwart-helm-values-oidc.yaml

@@ -0,0 +1,100 @@
+# Stalwart Helm Values with OIDC Configuration
+image:
+  repository: stalwartlabs/stalwart
+  tag: latest
+  pullPolicy: Always
+
+replicaCount: 1
+
+# Recovery admin for initial setup
+recoveryAdmin:
+  enabled: true
+  username: admin
+  password: ***REMOVED***
+
+# Persistent storage with encrypted volumes
+persistence:
+  enabled: true
+  storageClass: hcloud-volumes-encrypted
+  accessMode: ReadWriteOnce
+  size: 20Gi
+
+# Service configuration
+service:
+  type: LoadBalancer
+  ports:
+    smtp: 25
+    submission: 587
+    smtps: 465
+    imap: 143
+    imaps: 993
+    pop3: 110
+    pop3s: 995
+    sieve: 4190
+    mgmt: 8080
+
+# Ingress for web UI
+ingress:
+  enabled: true
+  className: traefik
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+  hosts:
+    - host: mail.basicstack.de
+      paths:
+        - path: /
+          pathType: Prefix
+          portName: mgmt
+  tls:
+    - secretName: stalwart-tls
+      hosts:
+        - mail.basicstack.de
+
+# RocksDB configuration
+config:
+  "@type": "RocksDb"
+  path: "/var/lib/stalwart"
+
+# Pod security context
+podSecurityContext:
+  fsGroup: 2000
+  runAsUser: 2000
+  runAsGroup: 2000
+
+# Container security context
+containerSecurityContext:
+  runAsNonRoot: true
+  allowPrivilegeEscalation: false
+  capabilities:
+    drop: [ALL]
+    add: [NET_BIND_SERVICE]
+  seccompProfile:
+    type: RuntimeDefault
+
+# Resources
+resources:
+  requests:
+    memory: "512Mi"
+    cpu: "250m"
+  limits:
+    memory: "2Gi"
+    cpu: "2000m"
+
+# OIDC Configuration via environment variables
+env:
+  - name: STALWART_OAUTH_ENABLE
+    value: "true"
+  - name: STALWART_OAUTH_ISSUER
+    value: "https://auth.basicstack.de"
+  - name: STALWART_OAUTH_CLIENT_ID
+    value: "stalwart-webui"
+  - name: STALWART_OAUTH_CLIENT_SECRET
+    valueFrom:
+      secretKeyRef:
+        name: stalwart-oidc
+        key: oidc-client-secret
+  - name: STALWART_OAUTH_REDIRECT_URI
+    value: "https://mail.basicstack.de/admin/oauth/callback"
+  - name: STALWART_OAUTH_SCOPES
+    value: "openid profile email"

apps/stalwart/stalwart-monitoring.yaml

@@ -0,0 +1,100 @@
+---
+# PrometheusRule for Stalwart alerting
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: stalwart-alerts
+  namespace: mail
+  labels:
+    app: stalwart
+    prometheus: kube-prometheus-stack
+    release: kube-prometheus-stack
+spec:
+  groups:
+  - name: stalwart.rules
+    interval: 30s
+    rules:
+    # Pod availability alert
+    - alert: StalwartPodDown
+      expr: kube_pod_status_phase{namespace="mail", pod=~"stalwart-.*", phase!="Running"} == 1
+      for: 5m
+      labels:
+        severity: critical
+        component: mail
+      annotations:
+        summary: "Stalwart mail server pod is down"
+        description: "Stalwart pod {{ $labels.pod }} in namespace {{ $labels.namespace }} has been down for more than 5 minutes."
+
+    # Container restart alert
+    - alert: StalwartContainerRestarting
+      expr: rate(kube_pod_container_status_restarts_total{namespace="mail", pod=~"stalwart-.*"}[15m]) > 0
+      for: 5m
+      labels:
+        severity: warning
+        component: mail
+      annotations:
+        summary: "Stalwart container is restarting"
+        description: "Stalwart container in pod {{ $labels.pod }} has restarted {{ $value }} times in the last 15 minutes."
+
+    # Memory usage alert
+    - alert: StalwartHighMemoryUsage
+      expr: |
+        (container_memory_working_set_bytes{namespace="mail", pod=~"stalwart-.*", container="stalwart"}
+        / container_spec_memory_limit_bytes{namespace="mail", pod=~"stalwart-.*", container="stalwart"}) > 0.85
+      for: 10m
+      labels:
+        severity: warning
+        component: mail
+      annotations:
+        summary: "Stalwart memory usage is high"
+        description: "Stalwart container {{ $labels.pod }} is using {{ $value | humanizePercentage }} of its memory limit."
+
+    # CPU usage alert
+    - alert: StalwartHighCPUUsage
+      expr: |
+        (rate(container_cpu_usage_seconds_total{namespace="mail", pod=~"stalwart-.*", container="stalwart"}[5m])
+        / container_spec_cpu_quota{namespace="mail", pod=~"stalwart-.*", container="stalwart"}
+        * container_spec_cpu_period{namespace="mail", pod=~"stalwart-.*", container="stalwart"}) > 0.85
+      for: 10m
+      labels:
+        severity: warning
+        component: mail
+      annotations:
+        summary: "Stalwart CPU usage is high"
+        description: "Stalwart container {{ $labels.pod }} is using {{ $value | humanizePercentage }} of its CPU limit."
+
+    # Disk usage alert for PVC
+    - alert: StalwartDiskSpaceLow
+      expr: |
+        (kubelet_volume_stats_used_bytes{namespace="mail", persistentvolumeclaim="stalwart-data"}
+        / kubelet_volume_stats_capacity_bytes{namespace="mail", persistentvolumeclaim="stalwart-data"}) > 0.75
+      for: 5m
+      labels:
+        severity: warning
+        component: mail
+      annotations:
+        summary: "Stalwart disk space is running low"
+        description: "Stalwart PVC stalwart-data is {{ $value | humanizePercentage }} full. Consider expanding the volume."
+
+    - alert: StalwartDiskSpaceCritical
+      expr: |
+        (kubelet_volume_stats_used_bytes{namespace="mail", persistentvolumeclaim="stalwart-data"}
+        / kubelet_volume_stats_capacity_bytes{namespace="mail", persistentvolumeclaim="stalwart-data"}) > 0.90
+      for: 5m
+      labels:
+        severity: critical
+        component: mail
+      annotations:
+        summary: "Stalwart disk space is critically low"
+        description: "Stalwart PVC stalwart-data is {{ $value | humanizePercentage }} full. Immediate action required!"
+
+    # PVC availability alert
+    - alert: StalwartPVCNotBound
+      expr: kube_persistentvolumeclaim_status_phase{namespace="mail", persistentvolumeclaim="stalwart-data", phase!="Bound"} == 1
+      for: 5m
+      labels:
+        severity: critical
+        component: mail
+      annotations:
+        summary: "Stalwart PVC is not bound"
+        description: "Stalwart PVC stalwart-data is in {{ $labels.phase }} state. Mail data may be unavailable."

apps/stalwart/stalwart-oidc-config.yaml

@@ -0,0 +1,47 @@
+---
+# Stalwart OIDC Configuration
+# This ConfigMap configures Pocket ID as the OAuth provider for Stalwart Mail Server
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: stalwart-oidc-config
+  namespace: mail
+data:
+  oauth.toml: |
+    # OAuth/OIDC Configuration for Pocket ID
+    [oauth]
+
+    # OAuth provider configuration
+    [oauth.pocket-id]
+    issuer-url = "https://auth.basicstack.de"
+    client-id = "19432b61-e27e-444d-9ff0-0fa23fdac6d7"
+    client-secret = "***REMOVED***"
+
+    # OAuth endpoints (auto-discovered from issuer-url)
+    authorization-endpoint = "https://auth.basicstack.de/api/oidc/authorize"
+    token-endpoint = "https://auth.basicstack.de/api/oidc/token"
+    userinfo-endpoint = "https://auth.basicstack.de/api/oidc/userinfo"
+
+    # Scopes to request
+    scopes = ["openid", "profile", "email"]
+
+    # Callback URL (must match what's configured in Pocket ID)
+    redirect-uri = "https://mail.basicstack.de/login/oauth"
+
+    # User attribute mapping
+    [oauth.pocket-id.user-mapping]
+    username = "preferred_username"
+    email = "email"
+    name = "name"
+---
+# Secret for OAuth client credentials
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stalwart-oidc-secret
+  namespace: mail
+type: Opaque
+stringData:
+  client-id: "19432b61-e27e-444d-9ff0-0fa23fdac6d7"
+  client-secret: "***REMOVED***"

apps/stalwart/stalwart-service-patch.yaml

@@ -0,0 +1,5 @@
+spec:
+  externalIPs:
+    - 178.105.17.239  # k3s-cp-1 (where DNS points)
+    - 178.105.216.48  # k3s-worker-1
+    - 49.13.134.255   # k3s-worker-2

apps/stalwart/stalwart-statefulset-patch-v2.yaml

@@ -0,0 +1,6 @@
+spec:
+  template:
+    spec:
+      containers:
+      - name: stalwart
+        args: ["--help"]

apps/stalwart/stalwart-statefulset-patch-v3.yaml

@@ -0,0 +1,9 @@
+spec:
+  template:
+    spec:
+      containers:
+      - name: stalwart
+        args: []
+        env:
+        - name: STALWART_LOG_LEVEL
+          value: "debug"

apps/stalwart/stalwart-statefulset-patch.yaml

@@ -0,0 +1,19 @@
+spec:
+  template:
+    spec:
+      containers:
+      - name: stalwart
+        args: ["--init"]
+        env:
+        - name: STALWART_STORAGE_DATA
+          value: "rocksdb"
+        - name: STALWART_STORAGE_BLOB
+          value: "rocksdb"
+        - name: STALWART_STORAGE_ROCKSDB_PATH
+          value: "/var/lib/stalwart"
+        - name: STALWART_SERVER_HOSTNAME
+          value: "mail.basicstack.de"
+        - name: STALWART_SERVER_HTTP_BIND
+          value: "[::]:8080"
+        - name: STALWART_LOG_LEVEL
+          value: "info"

apps/stalwart/stalwart-values-correct.yaml

@@ -0,0 +1,86 @@
+# Stalwart Helm Values - Fresh Deployment
+# Domain: mail.basicstack.de
+# Storage: hcloud-volumes-encrypted
+
+image:
+  repository: stalwartlabs/stalwart
+  tag: "latest"
+  pullPolicy: Always
+
+replicaCount: 1
+
+# Enable recovery admin for initial setup
+recoveryAdmin:
+  enabled: true
+  username: "admin"
+  password: "***REMOVED***"
+
+# RocksDB data store
+config:
+  "@type": RocksDb
+  path: /var/lib/stalwart
+
+# LoadBalancer service to expose mail ports
+service:
+  type: LoadBalancer
+  ports:
+    smtp: 25
+    smtps: 465
+    submission: 587
+    imap: 143
+    imaps: 993
+    pop3: 110
+    pop3s: 995
+    sieve: 4190
+    http: 80
+    https: 443
+    mgmt: 8080
+
+# Ingress for web management interface
+ingress:
+  enabled: true
+  className: traefik
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+  hosts:
+    - host: mail.basicstack.de
+      paths:
+        - path: /
+          pathType: Prefix
+          portName: mgmt
+  tls:
+    - secretName: stalwart-tls
+      hosts:
+        - mail.basicstack.de
+
+# Persistent storage with encrypted volumes
+persistence:
+  enabled: true
+  accessMode: ReadWriteOnce
+  storageClass: "hcloud-volumes-encrypted"
+  size: 20Gi
+
+# Pod security
+podSecurityContext:
+  fsGroup: 2000
+  runAsUser: 2000
+  runAsGroup: 2000
+
+containerSecurityContext:
+  runAsNonRoot: true
+  allowPrivilegeEscalation: false
+  capabilities:
+    drop: [ALL]
+    add: [NET_BIND_SERVICE]
+  seccompProfile:
+    type: RuntimeDefault
+
+# Resource limits
+resources:
+  requests:
+    memory: "512Mi"
+    cpu: "250m"
+  limits:
+    memory: "2Gi"
+    cpu: "2000m"

apps/stalwart/stalwart-values-fixed-ports.yaml

@@ -0,0 +1,90 @@
+# Stalwart Helm Values - Fixed Port Configuration
+# Domain: mail.basicstack.de
+# Storage: hcloud-volumes-encrypted
+
+image:
+  repository: stalwartlabs/stalwart
+  tag: "latest"
+  pullPolicy: Always
+
+replicaCount: 1
+
+# Enable recovery admin for initial setup
+recoveryAdmin:
+  enabled: true
+  username: "admin"
+  password: "***REMOVED***"
+
+# RocksDB data store
+config:
+  "@type": RocksDb
+  path: /var/lib/stalwart
+
+# LoadBalancer service - ONLY mail ports, NO HTTP ports (80, 443, 8080)
+# HTTP access is handled by Ingress/Traefik
+service:
+  type: LoadBalancer
+  ports:
+    # SMTP ports
+    smtp: 25
+    smtps: 465
+    submission: 587
+    # IMAP ports
+    imap: 143
+    imaps: 993
+    # POP3 ports (optional, can be disabled)
+    pop3: 110
+    pop3s: 995
+    # Sieve port
+    sieve: 4190
+    # Management port - exposed internally only, NOT on LoadBalancer
+    mgmt: 8080
+
+# Ingress for web management interface via Traefik
+ingress:
+  enabled: true
+  className: traefik
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+  hosts:
+    - host: mail.basicstack.de
+      paths:
+        - path: /
+          pathType: Prefix
+          portName: mgmt
+  tls:
+    - secretName: stalwart-tls
+      hosts:
+        - mail.basicstack.de
+
+# Persistent storage with encrypted volumes
+persistence:
+  enabled: true
+  accessMode: ReadWriteOnce
+  storageClass: "hcloud-volumes-encrypted"
+  size: 20Gi
+
+# Pod security
+podSecurityContext:
+  fsGroup: 2000
+  runAsUser: 2000
+  runAsGroup: 2000
+
+containerSecurityContext:
+  runAsNonRoot: true
+  allowPrivilegeEscalation: false
+  capabilities:
+    drop: [ALL]
+    add: [NET_BIND_SERVICE]
+  seccompProfile:
+    type: RuntimeDefault
+
+# Resource limits
+resources:
+  requests:
+    memory: "512Mi"
+    cpu: "250m"
+  limits:
+    memory: "2Gi"
+    cpu: "2000m"

apps/stalwart/stalwart-values.yaml

@@ -0,0 +1,110 @@
+# Stalwart Helm Chart Values
+# Domain: mail.basicstack.de
+# Storage: hcloud-volumes-encrypted
+
+image:
+  repository: stalwartlabs/stalwart
+  tag: "latest"
+  pullPolicy: Always
+
+replicaCount: 1
+
+# Recovery admin for initial setup (disabled)
+recoveryAdmin:
+  enabled: false
+
+# Persistent storage with encrypted volumes
+persistence:
+  enabled: true
+  storageClassName: hcloud-volumes-encrypted
+  accessMode: ReadWriteOnce
+  size: 20Gi
+
+# Service configuration
+service:
+  type: LoadBalancer
+  smtp:
+    enabled: true
+    ports:
+      - port: 25
+        name: smtp
+      - port: 587
+        name: submission
+      - port: 465
+        name: submissions
+  imap:
+    enabled: true
+    ports:
+      - port: 143
+        name: imap
+      - port: 993
+        name: imaps
+  http:
+    enabled: true
+    port: 8080
+
+# Ingress for web UI
+ingress:
+  enabled: true
+  className: traefik
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+  hosts:
+    - host: mail.basicstack.de
+      paths:
+        - path: /
+          pathType: Prefix
+  tls:
+    - secretName: stalwart-tls
+      hosts:
+        - mail.basicstack.de
+
+# RocksDB configuration (default)
+config:
+  "@type": "RocksDb"
+  path: "/opt/stalwart"
+
+# Pod security context
+podSecurityContext:
+  fsGroup: 2000
+  runAsUser: 2000
+  runAsGroup: 2000
+
+# Container security context
+containerSecurityContext:
+  runAsNonRoot: true
+  allowPrivilegeEscalation: false
+  capabilities:
+    drop: [ALL]
+    add: [NET_BIND_SERVICE]
+  seccompProfile:
+    type: RuntimeDefault
+
+# Health probes
+livenessProbe:
+  httpGet:
+    path: /healthz/live
+    port: 8080
+  initialDelaySeconds: 30
+  periodSeconds: 10
+  timeoutSeconds: 5
+  failureThreshold: 3
+
+readinessProbe:
+  httpGet:
+    path: /healthz/ready
+    port: 8080
+  initialDelaySeconds: 10
+  periodSeconds: 5
+  timeoutSeconds: 3
+  failureThreshold: 3
+
+# Resources
+resources:
+  requests:
+    memory: "512Mi"
+    cpu: "250m"
+  limits:
+    memory: "2Gi"
+    cpu: "2000m"

docs/README.md

@@ -0,0 +1,16 @@
+# Documentation
+
+This directory contains general documentation for the basicstack.de cluster and deployment processes.
+
+## Suggested Content
+
+- **Architecture**: Overall cluster architecture and design decisions
+- **Deployment Guides**: Step-by-step deployment procedures
+- **Troubleshooting**: Common issues and solutions
+- **Best Practices**: Standards and conventions for this cluster
+- **Operations Runbooks**: Incident response and operational procedures
+- **Disaster Recovery**: Backup and restore procedures at the cluster level
+
+## Note
+
+Application-specific documentation should live in the respective `apps/<app-name>/` directories, not here. This directory is for cross-cutting concerns and cluster-wide documentation.

infrastructure/README.md

@@ -0,0 +1,41 @@
+# Infrastructure
+
+This directory contains cluster-wide infrastructure configurations that support all applications.
+
+## Structure
+
+### `networking/`
+Network-level configurations including:
+- Ingress controller configurations
+- Network policies
+- DNS settings
+- Load balancer configurations
+- Certificate management (cert-manager, TLS)
+- Service mesh configurations (if applicable)
+
+### `monitoring/`
+Observability infrastructure:
+- Prometheus operator and configurations
+- Grafana dashboards and datasources
+- Logging stack (Loki, Promtail, etc.)
+- Alert rules and notification channels
+- Service monitors and pod monitors
+
+## Purpose
+
+Infrastructure configurations in this directory are shared across all applications. Changes here can affect the entire cluster, so:
+
+1. Test thoroughly before applying
+2. Document all changes
+3. Consider the impact on existing deployments
+4. Coordinate with other team members
+
+## Adding Infrastructure Components
+
+When adding new infrastructure components:
+
+1. Create appropriate subdirectories if needed
+2. Include clear documentation
+3. Define dependencies and prerequisites
+4. Provide rollback procedures
+5. Update this README with the new component