feat(monitoring): retire legacy backup-volumes CronJob + backup-storage PVC (DEV-489)

The restic pipeline (loki/grafana/k8s-resources) is proven end-to-end
after the DEV-488 restore drill, so the legacy rsync/tar pipeline into
the 100 Gi local-path `backup-storage` PVC is removed.

- Delete `apps/monitoring/backup-volumes-cronjob.yaml`.
- Remove the DEV-483 bridge `nodeSelector: k3s-worker-2` from
  `apps/monitoring/loki-deployment.yaml`. Loki's data protection now
  runs via `backup-loki-restic`, which follows the pod via podAffinity
  regardless of which node the RWO CSI volume attaches on. The
  `Recreate` rollout strategy stays — it is unrelated (avoids the
  attach-deadlock during a rollout). Resolves the RWO/nodeSelector
  attach race that was blocking DEV-478 weekly OS updates.
- Update `apps/monitoring/README.md` to drop the `backup-volumes`
  section, link the restic restore runbook, and record the pin
  removal.
- Clean stale coexistence comments in the restic/prometheus CronJob
  manifests now that the legacy job is gone.

Cluster-side (already applied out-of-band, since these manifests are
`kubectl apply`-based, not Argo-managed):
- `kubectl -n monitoring delete cronjob backup-volumes` -> NotFound.
- `kubectl -n monitoring delete pvc backup-storage` -> gone; local-path
  PV `pvc-d0db0ba9-8f89-4f66-9e65-d573ebe1085a` reclaimed automatically
  (Delete policy). Two stale pre-DEV-487 `backup-k8s-resources` job
  pods that still referenced the PVC were deleted to release the
  `pvc-protection` finalizer.
- `kubectl -n monitoring apply -f loki-deployment.yaml` -> Recreate
  rollout, new pod Ready in ~60s, no nodeSelector on the new spec.
- No `VolumeAttachment` for the retired PV.
- Restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
  `backup-k8s-resources`, `prometheus-backup`) intact.

Pre-delete snapshots retained on k3s-cp-1 under
`/root/dev489-snapshots-20260816T155411Z/` for post-mortem.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/monitoring/README.md

@@ -3,11 +3,10 @@
 Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
 
 - `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
-- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below). Being retired by the restic pipeline in [DEV-482](/DEV/issues/DEV-482) — keep running until step 6 (restore drill passed).
+- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
-- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
+- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
-- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
 - `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)).
 
-The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the destination for `backup-volumes` only. `backup-k8s-resources` no longer writes there — the tarball path was retired in [DEV-487](/DEV/issues/DEV-487). The PVC is scheduled for retirement together with `backup-volumes` once [DEV-482](/DEV/issues/DEV-482) step 6 (restore drill) has passed.
+Restore procedure for all three restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16) passed — see [DEV-488](/DEV/issues/DEV-488).
 
-`backup-volumes` pins itself to `k3s-worker-2` via `nodeSelector` because that is the node that holds its destination PVC. `backup-k8s-resources` is no longer node-pinned. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`.
+The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` and `prometheus-backup` remain unpinned.

apps/monitoring/backup-grafana-restic-cronjob.yaml

@@ -4,9 +4,7 @@
 #
 # Streams the RWO PVC `grafana-storage` (mounted read-only) into
 # `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana`, a client-side
-# encrypted restic repository. Deployed in parallel with the legacy
+# encrypted restic repository.
-# `backup-volumes` CronJob — do not retire that job until DEV-482
-# step 6 (restore drill passed).
 #
 # `grafana-storage` is a local-path PV anchored on k3s-worker-2, so
 # the grafana pod is already pinned there; a plain nodeSelector on

apps/monitoring/backup-k8s-resources-cronjob.yaml

@@ -14,10 +14,6 @@
 #   2. `restic` main container (`restic/restic:0.17.3`, matches the
 #      loki/grafana siblings) reads that file on stdin and streams it
 #      into the restic repo with `--stdin-filename cluster.yaml`.
-#
-# Deployed in parallel with the legacy `backup-volumes` CronJob — do
-# not retire that job or the `backup-storage` PVC until DEV-482 step 6
-# (restore drill) has passed.
 apiVersion: batch/v1
 kind: CronJob
 metadata:

apps/monitoring/backup-loki-restic-cronjob.yaml

@@ -4,9 +4,7 @@
 #
 # Streams the RWO PVC `loki-storage-encrypted` (mounted read-only)
 # into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki`, a client-side
-# encrypted restic repository. Deployed in parallel with the legacy
+# encrypted restic repository.
-# `backup-volumes` CronJob — do not retire that job until DEV-482
-# step 6 (restore drill passed).
 #
 # podAffinity co-schedules with the Loki pod (app=loki, topology
 # kubernetes.io/hostname). RWO permits additional read-only mounts
@@ -122,10 +120,10 @@ spec:
                   echo "restic check exit: ${CHECK_STATUS}"
 
                   # Textfile-collector metrics. Written into an emptyDir per
-                  # the current pattern used by backup-volumes and
+                  # the current pattern used by the other backup CronJobs.
-                  # backup-k8s-resources. Once a node-exporter textfile
+                  # Once a node-exporter textfile collector path is wired
-                  # collector path is wired up, these become scrapeable —
+                  # up, these become scrapeable — see the follow-up notes
-                  # see the follow-up notes in DEV-482.
+                  # in DEV-482.
                   {
                     echo "backup_loki_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
                     echo "backup_loki_timestamp_seconds $(date +%s)"

apps/monitoring/backup-volumes-cronjob.yaml

@@ -1,102 +0,0 @@
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: backup-volumes
-  namespace: monitoring
-  labels:
-    app: backup
-    type: volumes
-spec:
-  schedule: "0 3 * * *"
-  concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    metadata:
-      annotations:
-        prometheus.io/scrape: "true"
-      labels:
-        app: backup
-        type: volumes
-    spec:
-      backoffLimit: 2
-      template:
-        metadata:
-          labels:
-            app: backup
-        spec:
-          restartPolicy: OnFailure
-          # backup-storage, grafana-storage and loki-storage-encrypted are all
-          # RWO PVCs pinned to k3s-worker-2. Prometheus data lives on
-          # k3s-worker-1 and is intentionally NOT backed up here — see
-          # DEV-464 for the split rationale and the follow-up ticket for
-          # a dedicated Prometheus data backup.
-          nodeSelector:
-            kubernetes.io/hostname: k3s-worker-2
-          containers:
-          - name: volume-backup
-            image: alpine:3.19
-            command:
-            - /bin/sh
-            - -c
-            - |
-              set -e
-              apk add --no-cache rsync
-
-              BACKUP_DATE=$(date +%Y%m%d-%H%M%S)
-              BACKUP_DIR="/backup/volumes-$BACKUP_DATE"
-              mkdir -p "$BACKUP_DIR"
-
-              echo "Starting volume backup to $BACKUP_DIR"
-
-              if [ -d "/source/grafana" ]; then
-                echo "Backing up Grafana data..."
-                rsync -a /source/grafana/ "$BACKUP_DIR/grafana/" || echo "Warning: Grafana backup incomplete"
-              fi
-
-              if [ -d "/source/loki" ]; then
-                echo "Backing up Loki data..."
-                rsync -a /source/loki/ "$BACKUP_DIR/loki/" || echo "Warning: Loki backup incomplete"
-              fi
-
-              cd /backup
-              tar -czf "volumes-backup-$BACKUP_DATE.tar.gz" "$(basename $BACKUP_DIR)"
-              rm -rf "$BACKUP_DIR"
-
-              find /backup -name "volumes-backup-*.tar.gz" -mtime +7 -delete
-
-              BACKUP_SIZE=$(du -sh "/backup/volumes-backup-$BACKUP_DATE.tar.gz" | cut -f1)
-              echo "Volume backup completed successfully: $BACKUP_SIZE"
-
-              echo "backup_volumes_success 1" > /metrics/backup_success.prom
-              echo "backup_volumes_timestamp $(date +%s)" >> /metrics/backup_success.prom
-            resources:
-              requests:
-                cpu: 100m
-                memory: 128Mi
-              limits:
-                cpu: 1000m
-                memory: 512Mi
-            volumeMounts:
-            - mountPath: /backup
-              name: backup-storage
-            - mountPath: /source/grafana
-              name: grafana-data
-              readOnly: true
-            - mountPath: /source/loki
-              name: loki-data
-              readOnly: true
-            - mountPath: /metrics
-              name: metrics
-          volumes:
-          - name: backup-storage
-            persistentVolumeClaim:
-              claimName: backup-storage
-          - name: grafana-data
-            persistentVolumeClaim:
-              claimName: grafana-storage
-          - name: loki-data
-            persistentVolumeClaim:
-              claimName: loki-storage-encrypted
-          - name: metrics
-            emptyDir: {}

apps/monitoring/loki-deployment.yaml

@@ -21,12 +21,6 @@ spec:
       labels:
         app: loki
     spec:
-      # DEV-483 bridge fix: pin Loki to k3s-worker-2 so the loki-storage-encrypted
-      # RWO CSI volume co-locates with the backup-volumes CronJob (also on worker-2).
-      # Removed as part of DEV-482 step 6 once the legacy backup-volumes CronJob
-      # is retired (Option 4: restic -> Hetzner Object Storage).
-      nodeSelector:
-        kubernetes.io/hostname: k3s-worker-2
       containers:
       - name: loki
         image: grafana/loki:2.9.2

apps/monitoring/prometheus-backup-cronjob.yaml

@@ -2,11 +2,8 @@
 # Prometheus data backup (DEV-465).
 #
 # Prometheus data lives on the RWO PVC `prometheus-data-encrypted` in
-# namespace `monitoring`. That PVC is mounted by the Prometheus pod which
+# namespace `monitoring`. That PVC is mounted by the Prometheus pod, and
-# currently lives on k3s-worker-1, and the Hetzner CSI volume can only be
+# the Hetzner CSI volume can only be attached to one node at a time.
-# attached to one node at a time. The shared `backup-volumes` CronJob (see
-# `apps/monitoring/backup-volumes-cronjob.yaml`) is pinned to k3s-worker-2
-# (where grafana + loki live) and therefore cannot back up Prometheus.
 #
 # This CronJob co-schedules with the Prometheus pod via podAffinity, so it
 # lands on whichever node currently holds `prometheus-data-encrypted`. The
@@ -23,7 +20,7 @@ metadata:
     app: backup
     type: prometheus
 spec:
-  schedule: "30 3 * * *"  # daily 03:30, offset from backup-volumes (03:00)
+  schedule: "30 3 * * *"  # daily 03:30, offset from the loki/grafana restic runs
   concurrencyPolicy: Forbid
   successfulJobsHistoryLimit: 3
   failedJobsHistoryLimit: 3