ef62dde67c
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
|
||
|---|---|---|
| .. | ||
| argocd | ||
| basicstack-org | ||
| bookstack | ||
| directus | ||
| dozzle | ||
| forgejo | ||
| forgejo-runner | ||
| harbor | ||
| headlamp | ||
| monitoring | ||
| observability/patches | ||
| opencloud | ||
| pangolin | ||
| paperclip | ||
| passbolt | ||
| pocket-id | ||
| stalwart | ||
| app-argocd.yaml | ||
| app-basicstack-org.yaml | ||
| app-bookstack.yaml | ||
| app-directus.yaml | ||
| app-dozzle.yaml | ||
| app-forgejo-runner.yaml | ||
| app-forgejo.yaml | ||
| app-harbor.yaml | ||
| app-headlamp.yaml | ||
| app-opencloud.yaml | ||
| app-pangolin.yaml | ||
| app-paperclip.yaml | ||
| app-passbolt.yaml | ||
| app-pocket-id.yaml | ||
| app-stalwart.yaml | ||
| ARGOCD-MIGRATION.md | ||
| README.md | ||
Applications
This directory contains deployment configurations for all applications running on the basicstack.de cluster.
ArgoCD Application Management
Each application has two types of files:
app-<name>.yaml: ArgoCD Application manifest that tells ArgoCD to sync the app subdirectory<name>/: Application-specific Kubernetes manifests and configuration
The app-*.yaml files are synced by ArgoCD and create/manage the corresponding Application resources.
Each application's manifests in its subdirectory are then synced by its Application resource.
Structure
Each application should have its own subdirectory containing:
- Kubernetes manifests: Deployment, StatefulSet, Service, ConfigMap, Secret definitions
- Helm values: If using Helm charts, include values.yaml files
- Configuration files: Application-specific configs (TOML, JSON, YAML)
- Documentation: README or guide specific to the application deployment
- Patches: Any kubectl patches or modifications needed
Example: Stalwart
The stalwart/ directory serves as a reference implementation, containing:
- Multiple deployment variants (basic, with OIDC, etc.)
- Helm values files
- Monitoring dashboard configurations
- Backup/restore procedures
- Operational documentation
Adding a New Application
- Create a new directory:
apps/<application-name>/ - Add your Kubernetes manifests
- Include a README.md explaining:
- What the application does
- How to deploy it
- Configuration options
- Troubleshooting steps
- Test the deployment in a dev environment
- Commit with a descriptive message
Naming Conventions
- Directory names: lowercase, hyphen-separated (e.g.,
my-app) - Manifest files: descriptive names indicating resource type (e.g.,
deployment.yaml,service.yaml) - Use consistent naming across applications