Migrates the last plaintext-at-rest monitoring backup off rclone-sync onto the same restic pipeline as loki/grafana/k8s-resources. Closes the Hetzner Object Storage gap flagged in DEV-482 — no SSE-S3/SSE-KMS means every object we push must be encrypted client-side. - apps/monitoring/prometheus-backup-cronjob.yaml: pin the image to the Harbor mirror `harbor.basicstack.de/library/restic:0.17.3` (DEV-493) so the pull path matches the sibling CronJobs. - docs/monitoring/restic-restore.md: append the DEV-492 drill entry — cold restic init, 8.017 GiB → 2.441 GiB stored, `restic check` clean, restore of `latest` into a scratch namespace, then `promtool tsdb list` + `promtool tsdb analyze` against all 19 blocks (0 failed). Confirms the compaction-race mitigation (`--exclude wal/*`/`chunks_head/*` + accept exit 3) does not leave a corrupt snapshot. Also documents the DEV-492 Prometheus repo in the layout table + per-repo restore commands. - apps/monitoring/README.md: point the restore section at all four repos and cite DEV-488 (loki/k8s) + DEV-492 (prometheus) drill entries. Definition of done (DEV-492): - backup ships client-side-encrypted to Hetzner: ✅ (restic init succeeded on the manual run; snapshot 71420465 written) - restore drill promtool tsdb analyze clean: ✅ (19/19 blocks OK) - OBSERVABILITY_BASELINE.md + apps/monitoring/README.md updated: ✅ - restic-password ownership documented: ✅ (README shared-SealedSecret table already covered all four repos after this change) Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|---|---|---|
| apps | ||
| docs | ||
| infrastructure | ||
| .gitignore | ||
| add-user-andreas.ldif | ||
| DEV-349-progress.md | ||
| README.md | ||
stack.basicstack.de
CD/CI deployment manifests and configurations for the basicstack.de Kubernetes cluster.
Repository Structure
stack.basicstack.de/
├── apps/ # Application deployments
│ ├── stalwart/ # Stalwart mail server (example)
│ └── forgejo/ # Forgejo Git service (placeholder)
├── infrastructure/ # Infrastructure-level configurations
│ ├── networking/ # Network policies, ingress, DNS
│ └── monitoring/ # Monitoring, logging, observability
└── docs/ # Documentation and guides
Purpose
This repository serves as the central source of truth for all deployment configurations targeting the basicstack.de Kubernetes cluster. It follows GitOps principles where infrastructure and application state is declaratively defined and version-controlled.
Directory Details
apps/
Contains deployment configurations for individual applications and services running on the cluster. Each application should have its own subdirectory with:
- Kubernetes manifests (Deployments, StatefulSets, Services, etc.)
- Helm values files
- Configuration files
- Application-specific documentation
Example: The stalwart/ directory contains the complete deployment configuration for the Stalwart mail server, including multiple deployment variants, monitoring setup, and operational guides.
infrastructure/
Contains cluster-wide infrastructure configurations:
- networking/: Ingress controllers, network policies, DNS configurations, load balancers
- monitoring/: Prometheus, Grafana, logging infrastructure, observability tools
docs/
General documentation including:
- Deployment procedures
- Cluster architecture
- Troubleshooting guides
- Best practices
GitOps with Argo CD
This repository is managed via Argo CD, the GitOps deployment platform for the cluster.
- Argo CD UI: https://argo.basicstack.de
- Authentication: Pocket ID SSO (https://auth.basicstack.de)
- Documentation: docs/argocd.md
All changes pushed to the main branch are automatically synchronized to the cluster. Applications are defined in apps/app-*.yaml files and reference subdirectories for their manifests.
For details on managing applications, repository credentials, troubleshooting, and emergency procedures, see the Argo CD documentation.
Getting Started
- Clone this repository
- Review the example Stalwart deployment in
apps/stalwart/ - Follow the pattern for new application deployments
- Ensure all manifests are tested before committing
- Argo CD will automatically sync changes to the cluster (or use manual sync for critical changes)
Contributing
All changes should be:
- Committed with clear, descriptive messages
- Tested in a development environment when possible
- Documented appropriately
- Reviewed before deployment to production
Cluster Information
- Cluster: basicstack.de
- Platform: K3s on Hetzner Cloud
- Namespace Strategy: One namespace per application (recommended)
- Ingress: Traefik (default K3s ingress controller)