basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/apps/monitoring
History
CTO Agent 890809c91c fix(monitoring): Loki uses Recreate strategy for RWO PVC rollouts (DEV-483) 
RollingUpdate deadlocks the Loki rollout because loki-storage-encrypted
is a Hetzner CSI RWO volume — the new pod on worker-2 cannot attach the
PVC while the old pod on worker-1 still holds it. Recreate drains the
old pod first so the CSI detaches the volume cleanly.

Same pattern as the Harbor fix (commit 56e598f).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 12:54:26 +00:00
..
backup-k8s-resources-cronjob.yaml fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464) 2026-08-09 16:51:30 +00:00
backup-volumes-cronjob.yaml fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464) 2026-08-09 16:51:30 +00:00
loki-deployment.yaml fix(monitoring): Loki uses Recreate strategy for RWO PVC rollouts (DEV-483) 2026-08-16 12:54:26 +00:00
prometheus-backup-cronjob.yaml fix(monitoring): prune Prometheus S3 backups by prefix-date, not file mtime (DEV-465) 2026-08-09 17:40:50 +00:00
prometheus-backup-sealed.yaml Add dedicated Prometheus data backup CronJob (DEV-465) 2026-08-09 17:18:21 +00:00
README.md Add dedicated Prometheus data backup CronJob (DEV-465) 2026-08-09 17:18:21 +00:00

README.md

monitoring — backup CronJobs

Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (kubectl apply -f apps/monitoring/). See DEV-464 for the repair context.

  • backup-k8s-resources-cronjob.yaml — daily dump of Kubernetes resources into backup-storage PVC.
  • backup-volumes-cronjob.yaml — daily rsync/tar of Grafana + Loki PVCs into backup-storage. Prometheus data is NOT included here — it lives on a different node (see below).
  • prometheus-backup-cronjob.yaml + prometheus-backup-sealed.yaml — dedicated Prometheus data backup that streams prometheus-data-encrypted to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via podAffinity so the RWO PVC attaches on the same node (DEV-465).

The backup-storage PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for backup-k8s-resources and backup-volumes.

backup-k8s-resources and backup-volumes pin themselves to k3s-worker-2 via nodeSelector because that is the node that holds all destination + source PVCs used there. prometheus-backup follows the Prometheus pod via podAffinity, writing to Hetzner S3 (hetzner-s3:basicstack-backup/prometheus/) so it stays independent of backup-storage.