stack.basicstack.de/apps/monitoring/README.md
CTO Agent 9d1996051e fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464)
- Create the missing forgejo/platform-backup-data PVC that forgejo-backup
  references (20Gi hcloud-volumes-encrypted).
- Record monitoring/backup-k8s-resources with a k3s-worker-2 nodeSelector
  (backup-storage PVC is local-path pinned there), lower memory request
  (128Mi) so it fits worker-2 pressure, and switch to alpine/k8s image
  (bitnami/kubectl is no longer resolvable).
- Rewrite monitoring/backup-volumes to only back up grafana + loki
  co-located with backup-storage on k3s-worker-2. Prometheus data
  lives on k3s-worker-1 and is intentionally excluded here; a
  dedicated Prometheus data backup follows in a separate ticket.

The three CronJobs previously left Pending/ContainerCreating pods that
blocked the OS-update health guard in DEV-463.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-09 16:51:30 +00:00

10 lines
910 B
Markdown

# monitoring — backup CronJobs
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
- `backup-k8s-resources-cronjob.yaml` — daily dump of Kubernetes resources into `backup-storage` PVC.
- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data backup is **not** included here; it needs a separate on-node backup (tracked as a follow-up because Prometheus is on a different node than `backup-storage`).
The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for both jobs.
Both CronJobs pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used here.