basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/apps/monitoring
History
CTO Agent e4a7e5629a fix(monitoring): pin Loki to k3s-worker-2 to unblock backup-volumes (DEV-483) 
Bridge fix so DEV-478 weekly OS-update preflight can proceed while
DEV-482 (Option 4: restic -> Hetzner Object Storage) is designed and
rolled out. Captures the previously out-of-band Loki Deployment as a
manifest and adds nodeSelector kubernetes.io/hostname: k3s-worker-2 so
the loki-storage-encrypted RWO CSI volume co-locates with the
backup-volumes CronJob on worker-2, resolving the RWO/nodeSelector race.

This nodeSelector is temporary. It comes back out as part of DEV-482
step 6 when the legacy backup-volumes CronJob is retired.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 12:50:22 +00:00
..
backup-k8s-resources-cronjob.yaml fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464) 2026-08-09 16:51:30 +00:00
backup-volumes-cronjob.yaml fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464) 2026-08-09 16:51:30 +00:00
loki-deployment.yaml fix(monitoring): pin Loki to k3s-worker-2 to unblock backup-volumes (DEV-483) 2026-08-16 12:50:22 +00:00
prometheus-backup-cronjob.yaml fix(monitoring): prune Prometheus S3 backups by prefix-date, not file mtime (DEV-465) 2026-08-09 17:40:50 +00:00
prometheus-backup-sealed.yaml Add dedicated Prometheus data backup CronJob (DEV-465) 2026-08-09 17:18:21 +00:00
README.md Add dedicated Prometheus data backup CronJob (DEV-465) 2026-08-09 17:18:21 +00:00

README.md

monitoring — backup CronJobs

Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (kubectl apply -f apps/monitoring/). See DEV-464 for the repair context.

  • backup-k8s-resources-cronjob.yaml — daily dump of Kubernetes resources into backup-storage PVC.
  • backup-volumes-cronjob.yaml — daily rsync/tar of Grafana + Loki PVCs into backup-storage. Prometheus data is NOT included here — it lives on a different node (see below).
  • prometheus-backup-cronjob.yaml + prometheus-backup-sealed.yaml — dedicated Prometheus data backup that streams prometheus-data-encrypted to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via podAffinity so the RWO PVC attaches on the same node (DEV-465).

The backup-storage PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for backup-k8s-resources and backup-volumes.

backup-k8s-resources and backup-volumes pin themselves to k3s-worker-2 via nodeSelector because that is the node that holds all destination + source PVCs used there. prometheus-backup follows the Prometheus pod via podAffinity, writing to Hetzner S3 (hetzner-s3:basicstack-backup/prometheus/) so it stays independent of backup-storage.