The restic pipeline (loki/grafana/k8s-resources) is proven end-to-end
after the DEV-488 restore drill, so the legacy rsync/tar pipeline into
the 100 Gi local-path `backup-storage` PVC is removed.
- Delete `apps/monitoring/backup-volumes-cronjob.yaml`.
- Remove the DEV-483 bridge `nodeSelector: k3s-worker-2` from
`apps/monitoring/loki-deployment.yaml`. Loki's data protection now
runs via `backup-loki-restic`, which follows the pod via podAffinity
regardless of which node the RWO CSI volume attaches on. The
`Recreate` rollout strategy stays — it is unrelated (avoids the
attach-deadlock during a rollout). Resolves the RWO/nodeSelector
attach race that was blocking DEV-478 weekly OS updates.
- Update `apps/monitoring/README.md` to drop the `backup-volumes`
section, link the restic restore runbook, and record the pin
removal.
- Clean stale coexistence comments in the restic/prometheus CronJob
manifests now that the legacy job is gone.
Cluster-side (already applied out-of-band, since these manifests are
`kubectl apply`-based, not Argo-managed):
- `kubectl -n monitoring delete cronjob backup-volumes` -> NotFound.
- `kubectl -n monitoring delete pvc backup-storage` -> gone; local-path
PV `pvc-d0db0ba9-8f89-4f66-9e65-d573ebe1085a` reclaimed automatically
(Delete policy). Two stale pre-DEV-487 `backup-k8s-resources` job
pods that still referenced the PVC were deleted to release the
`pvc-protection` finalizer.
- `kubectl -n monitoring apply -f loki-deployment.yaml` -> Recreate
rollout, new pod Ready in ~60s, no nodeSelector on the new spec.
- No `VolumeAttachment` for the retired PV.
- Restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
`backup-k8s-resources`, `prometheus-backup`) intact.
Pre-delete snapshots retained on k3s-cp-1 under
`/root/dev489-snapshots-20260816T155411Z/` for post-mortem.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
RollingUpdate deadlocks the Loki rollout because loki-storage-encrypted
is a Hetzner CSI RWO volume — the new pod on worker-2 cannot attach the
PVC while the old pod on worker-1 still holds it. Recreate drains the
old pod first so the CSI detaches the volume cleanly.
Same pattern as the Harbor fix (commit 56e598f).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Bridge fix so DEV-478 weekly OS-update preflight can proceed while
DEV-482 (Option 4: restic -> Hetzner Object Storage) is designed and
rolled out. Captures the previously out-of-band Loki Deployment as a
manifest and adds nodeSelector kubernetes.io/hostname: k3s-worker-2 so
the loki-storage-encrypted RWO CSI volume co-locates with the
backup-volumes CronJob on worker-2, resolving the RWO/nodeSelector race.
This nodeSelector is temporary. It comes back out as part of DEV-482
step 6 when the legacy backup-volumes CronJob is retired.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
