e434fa751b
Rewrites the backup-k8s-resources CronJob to drop the local-path tarball
path and stream a YAML dump through `restic backup --stdin` into
`hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern:
init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3`
reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops
the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling
loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and
textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` /
`_check_status`).
Deployed in parallel with the legacy `backup-volumes` CronJob — the
`backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6
(restore drill).
Server-side dry-run validated on the k3s control plane.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
|
||
|---|---|---|
| .. | ||
| backup-grafana-restic-cronjob.yaml | ||
| backup-k8s-resources-cronjob.yaml | ||
| backup-loki-restic-cronjob.yaml | ||
| backup-volumes-cronjob.yaml | ||
| loki-deployment.yaml | ||
| prometheus-backup-cronjob.yaml | ||
| prometheus-backup-sealed.yaml | ||
| README.md | ||
monitoring — backup CronJobs
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (kubectl apply -f apps/monitoring/). See DEV-464 for the repair context.
backup-k8s-resources-cronjob.yaml— daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed throughrestic backup --stdintohetzner-s3:${BUCKET}/restic/k8s-resources. UsesserviceAccountName: backup-saand no PVC mount (init containeralpine/k8s:1.29.4writes an emptyDir, main containerrestic/restic:0.17.3reads it on stdin). Rewritten from the local-path tarball per the DEV-482 Option 4 rollout (DEV-487).backup-volumes-cronjob.yaml— daily rsync/tar of Grafana + Loki PVCs intobackup-storage. Prometheus data is NOT included here — it lives on a different node (see below). Being retired by the restic pipeline in DEV-482 — keep running until step 6 (restore drill passed).backup-loki-restic-cronjob.yaml— daily restic backup ofloki-storage-encryptedtohetzner-s3:${BUCKET}/restic/loki. Co-schedules with the Loki pod viapodAffinity(RWO permits additional read-only mounts on the same node). Deployed in parallel withbackup-volumesper the DEV-482 Option 4 rollout (DEV-485).backup-grafana-restic-cronjob.yaml— daily restic backup ofgrafana-storagetohetzner-s3:${BUCKET}/restic/grafana. Pinned tok3s-worker-2vianodeSelector(the local-path PV anchors the grafana pod there already, nopodAffinityneeded). Schedule15 3 * * *— offset from the loki run at03:00. Deployed in parallel withbackup-volumesper the DEV-482 Option 4 rollout (DEV-486).prometheus-backup-cronjob.yaml+prometheus-backup-sealed.yaml— dedicated Prometheus data backup that streamsprometheus-data-encryptedto Hetzner S3 via rclone. Co-schedules with the Prometheus pod viapodAffinityso the RWO PVC attaches on the same node (DEV-465).
The backup-storage PVC (100Gi, local-path, bound to k3s-worker-2) is the destination for backup-volumes only. backup-k8s-resources no longer writes there — the tarball path was retired in DEV-487. The PVC is scheduled for retirement together with backup-volumes once DEV-482 step 6 (restore drill) has passed.
backup-volumes pins itself to k3s-worker-2 via nodeSelector because that is the node that holds its destination PVC. backup-k8s-resources is no longer node-pinned. prometheus-backup follows the Prometheus pod via podAffinity, writing to Hetzner S3 (hetzner-s3:basicstack-backup/prometheus/) so it stays independent of backup-storage.