feat(monitoring): retire legacy backup-volumes CronJob + backup-storage PVC (DEV-489)
The restic pipeline (loki/grafana/k8s-resources) is proven end-to-end after the DEV-488 restore drill, so the legacy rsync/tar pipeline into the 100 Gi local-path `backup-storage` PVC is removed. - Delete `apps/monitoring/backup-volumes-cronjob.yaml`. - Remove the DEV-483 bridge `nodeSelector: k3s-worker-2` from `apps/monitoring/loki-deployment.yaml`. Loki's data protection now runs via `backup-loki-restic`, which follows the pod via podAffinity regardless of which node the RWO CSI volume attaches on. The `Recreate` rollout strategy stays — it is unrelated (avoids the attach-deadlock during a rollout). Resolves the RWO/nodeSelector attach race that was blocking DEV-478 weekly OS updates. - Update `apps/monitoring/README.md` to drop the `backup-volumes` section, link the restic restore runbook, and record the pin removal. - Clean stale coexistence comments in the restic/prometheus CronJob manifests now that the legacy job is gone. Cluster-side (already applied out-of-band, since these manifests are `kubectl apply`-based, not Argo-managed): - `kubectl -n monitoring delete cronjob backup-volumes` -> NotFound. - `kubectl -n monitoring delete pvc backup-storage` -> gone; local-path PV `pvc-d0db0ba9-8f89-4f66-9e65-d573ebe1085a` reclaimed automatically (Delete policy). Two stale pre-DEV-487 `backup-k8s-resources` job pods that still referenced the PVC were deleted to release the `pvc-protection` finalizer. - `kubectl -n monitoring apply -f loki-deployment.yaml` -> Recreate rollout, new pod Ready in ~60s, no nodeSelector on the new spec. - No `VolumeAttachment` for the retired PV. - Restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`, `backup-k8s-resources`, `prometheus-backup`) intact. Pre-delete snapshots retained on k3s-cp-1 under `/root/dev489-snapshots-20260816T155411Z/` for post-mortem. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
49b724439b
commit
0c1c05fc50
7 changed files with 13 additions and 133 deletions
|
|
@ -3,11 +3,10 @@
|
|||
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
|
||||
|
||||
- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
|
||||
- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below). Being retired by the restic pipeline in [DEV-482](/DEV/issues/DEV-482) — keep running until step 6 (restore drill passed).
|
||||
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
|
||||
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
|
||||
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
|
||||
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
|
||||
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)).
|
||||
|
||||
The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the destination for `backup-volumes` only. `backup-k8s-resources` no longer writes there — the tarball path was retired in [DEV-487](/DEV/issues/DEV-487). The PVC is scheduled for retirement together with `backup-volumes` once [DEV-482](/DEV/issues/DEV-482) step 6 (restore drill) has passed.
|
||||
Restore procedure for all three restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16) passed — see [DEV-488](/DEV/issues/DEV-488).
|
||||
|
||||
`backup-volumes` pins itself to `k3s-worker-2` via `nodeSelector` because that is the node that holds its destination PVC. `backup-k8s-resources` is no longer node-pinned. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`.
|
||||
The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` and `prometheus-backup` remain unpinned.
|
||||
|
|
|
|||
|
|
@ -4,9 +4,7 @@
|
|||
#
|
||||
# Streams the RWO PVC `grafana-storage` (mounted read-only) into
|
||||
# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana`, a client-side
|
||||
# encrypted restic repository. Deployed in parallel with the legacy
|
||||
# `backup-volumes` CronJob — do not retire that job until DEV-482
|
||||
# step 6 (restore drill passed).
|
||||
# encrypted restic repository.
|
||||
#
|
||||
# `grafana-storage` is a local-path PV anchored on k3s-worker-2, so
|
||||
# the grafana pod is already pinned there; a plain nodeSelector on
|
||||
|
|
|
|||
|
|
@ -14,10 +14,6 @@
|
|||
# 2. `restic` main container (`restic/restic:0.17.3`, matches the
|
||||
# loki/grafana siblings) reads that file on stdin and streams it
|
||||
# into the restic repo with `--stdin-filename cluster.yaml`.
|
||||
#
|
||||
# Deployed in parallel with the legacy `backup-volumes` CronJob — do
|
||||
# not retire that job or the `backup-storage` PVC until DEV-482 step 6
|
||||
# (restore drill) has passed.
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
|
|
|
|||
|
|
@ -4,9 +4,7 @@
|
|||
#
|
||||
# Streams the RWO PVC `loki-storage-encrypted` (mounted read-only)
|
||||
# into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki`, a client-side
|
||||
# encrypted restic repository. Deployed in parallel with the legacy
|
||||
# `backup-volumes` CronJob — do not retire that job until DEV-482
|
||||
# step 6 (restore drill passed).
|
||||
# encrypted restic repository.
|
||||
#
|
||||
# podAffinity co-schedules with the Loki pod (app=loki, topology
|
||||
# kubernetes.io/hostname). RWO permits additional read-only mounts
|
||||
|
|
@ -122,10 +120,10 @@ spec:
|
|||
echo "restic check exit: ${CHECK_STATUS}"
|
||||
|
||||
# Textfile-collector metrics. Written into an emptyDir per
|
||||
# the current pattern used by backup-volumes and
|
||||
# backup-k8s-resources. Once a node-exporter textfile
|
||||
# collector path is wired up, these become scrapeable —
|
||||
# see the follow-up notes in DEV-482.
|
||||
# the current pattern used by the other backup CronJobs.
|
||||
# Once a node-exporter textfile collector path is wired
|
||||
# up, these become scrapeable — see the follow-up notes
|
||||
# in DEV-482.
|
||||
{
|
||||
echo "backup_loki_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
|
||||
echo "backup_loki_timestamp_seconds $(date +%s)"
|
||||
|
|
|
|||
|
|
@ -1,102 +0,0 @@
|
|||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: backup-volumes
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app: backup
|
||||
type: volumes
|
||||
spec:
|
||||
schedule: "0 3 * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: backup
|
||||
type: volumes
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: backup
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
# backup-storage, grafana-storage and loki-storage-encrypted are all
|
||||
# RWO PVCs pinned to k3s-worker-2. Prometheus data lives on
|
||||
# k3s-worker-1 and is intentionally NOT backed up here — see
|
||||
# DEV-464 for the split rationale and the follow-up ticket for
|
||||
# a dedicated Prometheus data backup.
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: k3s-worker-2
|
||||
containers:
|
||||
- name: volume-backup
|
||||
image: alpine:3.19
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
set -e
|
||||
apk add --no-cache rsync
|
||||
|
||||
BACKUP_DATE=$(date +%Y%m%d-%H%M%S)
|
||||
BACKUP_DIR="/backup/volumes-$BACKUP_DATE"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
echo "Starting volume backup to $BACKUP_DIR"
|
||||
|
||||
if [ -d "/source/grafana" ]; then
|
||||
echo "Backing up Grafana data..."
|
||||
rsync -a /source/grafana/ "$BACKUP_DIR/grafana/" || echo "Warning: Grafana backup incomplete"
|
||||
fi
|
||||
|
||||
if [ -d "/source/loki" ]; then
|
||||
echo "Backing up Loki data..."
|
||||
rsync -a /source/loki/ "$BACKUP_DIR/loki/" || echo "Warning: Loki backup incomplete"
|
||||
fi
|
||||
|
||||
cd /backup
|
||||
tar -czf "volumes-backup-$BACKUP_DATE.tar.gz" "$(basename $BACKUP_DIR)"
|
||||
rm -rf "$BACKUP_DIR"
|
||||
|
||||
find /backup -name "volumes-backup-*.tar.gz" -mtime +7 -delete
|
||||
|
||||
BACKUP_SIZE=$(du -sh "/backup/volumes-backup-$BACKUP_DATE.tar.gz" | cut -f1)
|
||||
echo "Volume backup completed successfully: $BACKUP_SIZE"
|
||||
|
||||
echo "backup_volumes_success 1" > /metrics/backup_success.prom
|
||||
echo "backup_volumes_timestamp $(date +%s)" >> /metrics/backup_success.prom
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 512Mi
|
||||
volumeMounts:
|
||||
- mountPath: /backup
|
||||
name: backup-storage
|
||||
- mountPath: /source/grafana
|
||||
name: grafana-data
|
||||
readOnly: true
|
||||
- mountPath: /source/loki
|
||||
name: loki-data
|
||||
readOnly: true
|
||||
- mountPath: /metrics
|
||||
name: metrics
|
||||
volumes:
|
||||
- name: backup-storage
|
||||
persistentVolumeClaim:
|
||||
claimName: backup-storage
|
||||
- name: grafana-data
|
||||
persistentVolumeClaim:
|
||||
claimName: grafana-storage
|
||||
- name: loki-data
|
||||
persistentVolumeClaim:
|
||||
claimName: loki-storage-encrypted
|
||||
- name: metrics
|
||||
emptyDir: {}
|
||||
|
|
@ -21,12 +21,6 @@ spec:
|
|||
labels:
|
||||
app: loki
|
||||
spec:
|
||||
# DEV-483 bridge fix: pin Loki to k3s-worker-2 so the loki-storage-encrypted
|
||||
# RWO CSI volume co-locates with the backup-volumes CronJob (also on worker-2).
|
||||
# Removed as part of DEV-482 step 6 once the legacy backup-volumes CronJob
|
||||
# is retired (Option 4: restic -> Hetzner Object Storage).
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: k3s-worker-2
|
||||
containers:
|
||||
- name: loki
|
||||
image: grafana/loki:2.9.2
|
||||
|
|
|
|||
|
|
@ -2,11 +2,8 @@
|
|||
# Prometheus data backup (DEV-465).
|
||||
#
|
||||
# Prometheus data lives on the RWO PVC `prometheus-data-encrypted` in
|
||||
# namespace `monitoring`. That PVC is mounted by the Prometheus pod which
|
||||
# currently lives on k3s-worker-1, and the Hetzner CSI volume can only be
|
||||
# attached to one node at a time. The shared `backup-volumes` CronJob (see
|
||||
# `apps/monitoring/backup-volumes-cronjob.yaml`) is pinned to k3s-worker-2
|
||||
# (where grafana + loki live) and therefore cannot back up Prometheus.
|
||||
# namespace `monitoring`. That PVC is mounted by the Prometheus pod, and
|
||||
# the Hetzner CSI volume can only be attached to one node at a time.
|
||||
#
|
||||
# This CronJob co-schedules with the Prometheus pod via podAffinity, so it
|
||||
# lands on whichever node currently holds `prometheus-data-encrypted`. The
|
||||
|
|
@ -23,7 +20,7 @@ metadata:
|
|||
app: backup
|
||||
type: prometheus
|
||||
spec:
|
||||
schedule: "30 3 * * *" # daily 03:30, offset from backup-volumes (03:00)
|
||||
schedule: "30 3 * * *" # daily 03:30, offset from the loki/grafana restic runs
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 3
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue