diff --git a/apps/monitoring/README.md b/apps/monitoring/README.md index 08f3036..01675ea 100644 --- a/apps/monitoring/README.md +++ b/apps/monitoring/README.md @@ -3,11 +3,10 @@ Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context. - `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)). -- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below). Being retired by the restic pipeline in [DEV-482](/DEV/issues/DEV-482) — keep running until step 6 (restore drill passed). -- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)). -- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)). +- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)). +- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)). - `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)). -The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the destination for `backup-volumes` only. `backup-k8s-resources` no longer writes there — the tarball path was retired in [DEV-487](/DEV/issues/DEV-487). The PVC is scheduled for retirement together with `backup-volumes` once [DEV-482](/DEV/issues/DEV-482) step 6 (restore drill) has passed. +Restore procedure for all three restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16) passed — see [DEV-488](/DEV/issues/DEV-488). -`backup-volumes` pins itself to `k3s-worker-2` via `nodeSelector` because that is the node that holds its destination PVC. `backup-k8s-resources` is no longer node-pinned. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`. +The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` and `prometheus-backup` remain unpinned. diff --git a/apps/monitoring/backup-grafana-restic-cronjob.yaml b/apps/monitoring/backup-grafana-restic-cronjob.yaml index 2d2bdac..c888c8a 100644 --- a/apps/monitoring/backup-grafana-restic-cronjob.yaml +++ b/apps/monitoring/backup-grafana-restic-cronjob.yaml @@ -4,9 +4,7 @@ # # Streams the RWO PVC `grafana-storage` (mounted read-only) into # `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana`, a client-side -# encrypted restic repository. Deployed in parallel with the legacy -# `backup-volumes` CronJob — do not retire that job until DEV-482 -# step 6 (restore drill passed). +# encrypted restic repository. # # `grafana-storage` is a local-path PV anchored on k3s-worker-2, so # the grafana pod is already pinned there; a plain nodeSelector on diff --git a/apps/monitoring/backup-k8s-resources-cronjob.yaml b/apps/monitoring/backup-k8s-resources-cronjob.yaml index 9e54831..3d7c723 100644 --- a/apps/monitoring/backup-k8s-resources-cronjob.yaml +++ b/apps/monitoring/backup-k8s-resources-cronjob.yaml @@ -14,10 +14,6 @@ # 2. `restic` main container (`restic/restic:0.17.3`, matches the # loki/grafana siblings) reads that file on stdin and streams it # into the restic repo with `--stdin-filename cluster.yaml`. -# -# Deployed in parallel with the legacy `backup-volumes` CronJob — do -# not retire that job or the `backup-storage` PVC until DEV-482 step 6 -# (restore drill) has passed. apiVersion: batch/v1 kind: CronJob metadata: diff --git a/apps/monitoring/backup-loki-restic-cronjob.yaml b/apps/monitoring/backup-loki-restic-cronjob.yaml index 40d53eb..8274191 100644 --- a/apps/monitoring/backup-loki-restic-cronjob.yaml +++ b/apps/monitoring/backup-loki-restic-cronjob.yaml @@ -4,9 +4,7 @@ # # Streams the RWO PVC `loki-storage-encrypted` (mounted read-only) # into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki`, a client-side -# encrypted restic repository. Deployed in parallel with the legacy -# `backup-volumes` CronJob — do not retire that job until DEV-482 -# step 6 (restore drill passed). +# encrypted restic repository. # # podAffinity co-schedules with the Loki pod (app=loki, topology # kubernetes.io/hostname). RWO permits additional read-only mounts @@ -122,10 +120,10 @@ spec: echo "restic check exit: ${CHECK_STATUS}" # Textfile-collector metrics. Written into an emptyDir per - # the current pattern used by backup-volumes and - # backup-k8s-resources. Once a node-exporter textfile - # collector path is wired up, these become scrapeable — - # see the follow-up notes in DEV-482. + # the current pattern used by the other backup CronJobs. + # Once a node-exporter textfile collector path is wired + # up, these become scrapeable — see the follow-up notes + # in DEV-482. { echo "backup_loki_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)" echo "backup_loki_timestamp_seconds $(date +%s)" diff --git a/apps/monitoring/backup-volumes-cronjob.yaml b/apps/monitoring/backup-volumes-cronjob.yaml deleted file mode 100644 index a03c7bc..0000000 --- a/apps/monitoring/backup-volumes-cronjob.yaml +++ /dev/null @@ -1,102 +0,0 @@ -apiVersion: batch/v1 -kind: CronJob -metadata: - name: backup-volumes - namespace: monitoring - labels: - app: backup - type: volumes -spec: - schedule: "0 3 * * *" - concurrencyPolicy: Forbid - successfulJobsHistoryLimit: 3 - failedJobsHistoryLimit: 3 - jobTemplate: - metadata: - annotations: - prometheus.io/scrape: "true" - labels: - app: backup - type: volumes - spec: - backoffLimit: 2 - template: - metadata: - labels: - app: backup - spec: - restartPolicy: OnFailure - # backup-storage, grafana-storage and loki-storage-encrypted are all - # RWO PVCs pinned to k3s-worker-2. Prometheus data lives on - # k3s-worker-1 and is intentionally NOT backed up here — see - # DEV-464 for the split rationale and the follow-up ticket for - # a dedicated Prometheus data backup. - nodeSelector: - kubernetes.io/hostname: k3s-worker-2 - containers: - - name: volume-backup - image: alpine:3.19 - command: - - /bin/sh - - -c - - | - set -e - apk add --no-cache rsync - - BACKUP_DATE=$(date +%Y%m%d-%H%M%S) - BACKUP_DIR="/backup/volumes-$BACKUP_DATE" - mkdir -p "$BACKUP_DIR" - - echo "Starting volume backup to $BACKUP_DIR" - - if [ -d "/source/grafana" ]; then - echo "Backing up Grafana data..." - rsync -a /source/grafana/ "$BACKUP_DIR/grafana/" || echo "Warning: Grafana backup incomplete" - fi - - if [ -d "/source/loki" ]; then - echo "Backing up Loki data..." - rsync -a /source/loki/ "$BACKUP_DIR/loki/" || echo "Warning: Loki backup incomplete" - fi - - cd /backup - tar -czf "volumes-backup-$BACKUP_DATE.tar.gz" "$(basename $BACKUP_DIR)" - rm -rf "$BACKUP_DIR" - - find /backup -name "volumes-backup-*.tar.gz" -mtime +7 -delete - - BACKUP_SIZE=$(du -sh "/backup/volumes-backup-$BACKUP_DATE.tar.gz" | cut -f1) - echo "Volume backup completed successfully: $BACKUP_SIZE" - - echo "backup_volumes_success 1" > /metrics/backup_success.prom - echo "backup_volumes_timestamp $(date +%s)" >> /metrics/backup_success.prom - resources: - requests: - cpu: 100m - memory: 128Mi - limits: - cpu: 1000m - memory: 512Mi - volumeMounts: - - mountPath: /backup - name: backup-storage - - mountPath: /source/grafana - name: grafana-data - readOnly: true - - mountPath: /source/loki - name: loki-data - readOnly: true - - mountPath: /metrics - name: metrics - volumes: - - name: backup-storage - persistentVolumeClaim: - claimName: backup-storage - - name: grafana-data - persistentVolumeClaim: - claimName: grafana-storage - - name: loki-data - persistentVolumeClaim: - claimName: loki-storage-encrypted - - name: metrics - emptyDir: {} diff --git a/apps/monitoring/loki-deployment.yaml b/apps/monitoring/loki-deployment.yaml index 4336dd4..a729069 100644 --- a/apps/monitoring/loki-deployment.yaml +++ b/apps/monitoring/loki-deployment.yaml @@ -21,12 +21,6 @@ spec: labels: app: loki spec: - # DEV-483 bridge fix: pin Loki to k3s-worker-2 so the loki-storage-encrypted - # RWO CSI volume co-locates with the backup-volumes CronJob (also on worker-2). - # Removed as part of DEV-482 step 6 once the legacy backup-volumes CronJob - # is retired (Option 4: restic -> Hetzner Object Storage). - nodeSelector: - kubernetes.io/hostname: k3s-worker-2 containers: - name: loki image: grafana/loki:2.9.2 diff --git a/apps/monitoring/prometheus-backup-cronjob.yaml b/apps/monitoring/prometheus-backup-cronjob.yaml index 62e1a44..a8428f3 100644 --- a/apps/monitoring/prometheus-backup-cronjob.yaml +++ b/apps/monitoring/prometheus-backup-cronjob.yaml @@ -2,11 +2,8 @@ # Prometheus data backup (DEV-465). # # Prometheus data lives on the RWO PVC `prometheus-data-encrypted` in -# namespace `monitoring`. That PVC is mounted by the Prometheus pod which -# currently lives on k3s-worker-1, and the Hetzner CSI volume can only be -# attached to one node at a time. The shared `backup-volumes` CronJob (see -# `apps/monitoring/backup-volumes-cronjob.yaml`) is pinned to k3s-worker-2 -# (where grafana + loki live) and therefore cannot back up Prometheus. +# namespace `monitoring`. That PVC is mounted by the Prometheus pod, and +# the Hetzner CSI volume can only be attached to one node at a time. # # This CronJob co-schedules with the Prometheus pod via podAffinity, so it # lands on whichever node currently holds `prometheus-data-encrypted`. The @@ -23,7 +20,7 @@ metadata: app: backup type: prometheus spec: - schedule: "30 3 * * *" # daily 03:30, offset from backup-volumes (03:00) + schedule: "30 3 * * *" # daily 03:30, offset from the loki/grafana restic runs concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3