feat(monitoring): backup-k8s-resources → restic --stdin to Hetzner S3 (DEV-487)

Rewrites the backup-k8s-resources CronJob to drop the local-path tarball
path and stream a YAML dump through `restic backup --stdin` into
`hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern:
init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3`
reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops
the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling
loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and
textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` /
`_check_status`).

Deployed in parallel with the legacy `backup-volumes` CronJob — the
`backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6
(restore drill).

Server-side dry-run validated on the k3s control plane.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/monitoring/README.md

@@ -2,12 +2,12 @@
 
 Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
 
-- `backup-k8s-resources-cronjob.yaml` — daily dump of Kubernetes resources into `backup-storage` PVC.
+- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
 - `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below). Being retired by the restic pipeline in [DEV-482](/DEV/issues/DEV-482) — keep running until step 6 (restore drill passed).
 - `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
 - `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
 - `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)).
 
-The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for `backup-k8s-resources` and `backup-volumes`.
+The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the destination for `backup-volumes` only. `backup-k8s-resources` no longer writes there — the tarball path was retired in [DEV-487](/DEV/issues/DEV-487). The PVC is scheduled for retirement together with `backup-volumes` once [DEV-482](/DEV/issues/DEV-482) step 6 (restore drill) has passed.
 
-`backup-k8s-resources` and `backup-volumes` pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used there. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`.
+`backup-volumes` pins itself to `k3s-worker-2` via `nodeSelector` because that is the node that holds its destination PVC. `backup-k8s-resources` is no longer node-pinned. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`.

apps/monitoring/backup-k8s-resources-cronjob.yaml

@@ -1,3 +1,23 @@
+---
+# Kubernetes-resource backup via restic to Hetzner Object Storage
+# (DEV-487, DEV-482 Option 4). Step 4 of the Option 4 rollout.
+#
+# Streams a concatenated YAML dump of cluster-scoped and per-namespace
+# resources through `restic backup --stdin` into
+# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources`. No PVC mount
+# (drops the local-path `backup-storage` dependency), no node pin.
+#
+# Two-container pattern:
+#   1. `kubectl-dump` init container (`alpine/k8s:1.29.4`) writes
+#      /dump/cluster.yaml into an emptyDir. Uses `serviceAccountName:
+#      backup-sa` (unchanged from the legacy job).
+#   2. `restic` main container (`restic/restic:0.17.3`, matches the
+#      loki/grafana siblings) reads that file on stdin and streams it
+#      into the restic repo with `--stdin-filename cluster.yaml`.
+#
+# Deployed in parallel with the legacy `backup-volumes` CronJob — do
+# not retire that job or the `backup-storage` PVC until DEV-482 step 6
+# (restore drill) has passed.
 apiVersion: batch/v1
 kind: CronJob
 metadata:
@@ -6,6 +26,7 @@ metadata:
   labels:
     app: backup
     type: k8s-resources
+    backend: restic
 spec:
   schedule: "0 2 * * *"
   concurrencyPolicy: Forbid
@@ -13,72 +34,159 @@ spec:
   failedJobsHistoryLimit: 3
   jobTemplate:
     metadata:
-      annotations:
-        prometheus.io/scrape: "true"
       labels:
         app: backup
         type: k8s-resources
+        backend: restic
     spec:
       backoffLimit: 2
+      activeDeadlineSeconds: 3600
       template:
         metadata:
           labels:
             app: backup
+            type: k8s-resources
+            backend: restic
         spec:
           restartPolicy: OnFailure
           serviceAccountName: backup-sa
-          # backup-storage PVC (local-path) is bound to k3s-worker-2, so pin here.
+          initContainers:
-          nodeSelector:
+            - name: kubectl-dump
-            kubernetes.io/hostname: k3s-worker-2
-          containers:
-          - name: kubectl-backup
               image: alpine/k8s:1.29.4
               command:
                 - /bin/sh
                 - -c
                 - |
-              set -e
+                  set -eu
-              BACKUP_DIR="/backup/k8s-$(date +%Y%m%d-%H%M%S)"
+                  echo "=== kubectl-dump started at $(date -u +%FT%TZ) ==="
-              mkdir -p "$BACKUP_DIR"
+                  DUMP=/dump/cluster.yaml
+                  : > "${DUMP}"
 
-              echo "Starting Kubernetes resources backup to $BACKUP_DIR"
+                  echo "--- namespaces ---"
+                  kubectl get namespaces -o yaml >> "${DUMP}"
+                  echo "---" >> "${DUMP}"
 
-              kubectl get namespaces -o yaml > "$BACKUP_DIR/namespaces.yaml"
+                  echo "--- cluster-scoped resources ---"
+                  kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \
+                    -o yaml >> "${DUMP}"
+                  echo "---" >> "${DUMP}"
 
+                  echo "--- namespaced resources ---"
                   for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
-                mkdir -p "$BACKUP_DIR/$ns"
+                    echo "  ns=${ns}"
-                kubectl get configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \
+                    kubectl get \
-                  -n "$ns" -o yaml > "$BACKUP_DIR/$ns/resources.yaml" 2>/dev/null || true
+                      configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \
+                      -n "${ns}" -o yaml >> "${DUMP}" 2>/dev/null || true
+                    echo "---" >> "${DUMP}"
                   done
 
-              kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \
+                  echo "dump size: $(wc -c < ${DUMP}) bytes"
-                -o yaml > "$BACKUP_DIR/cluster-resources.yaml"
+                  echo "=== kubectl-dump finished at $(date -u +%FT%TZ) ==="
-
-              cd /backup
-              tar -czf "k8s-backup-$(date +%Y%m%d-%H%M%S).tar.gz" "$(basename $BACKUP_DIR)"
-              rm -rf "$BACKUP_DIR"
-
-              find /backup -name "k8s-backup-*.tar.gz" -mtime +7 -delete
-
-              echo "Backup completed successfully"
-
-              echo "backup_k8s_resources_success 1" > /metrics/backup_success.prom
-              echo "backup_k8s_resources_timestamp $(date +%s)" >> /metrics/backup_success.prom
               resources:
                 requests:
                   cpu: 50m
                   memory: 128Mi
                 limits:
                   cpu: 500m
-                memory: 384Mi
+                  memory: 512Mi
               volumeMounts:
-            - mountPath: /backup
+                - name: dump
-              name: backup-storage
+                  mountPath: /dump
-            - mountPath: /metrics
+          containers:
-              name: metrics
+            - name: restic
+              image: restic/restic:0.17.3
+              env:
+                - name: AWS_ACCESS_KEY_ID
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: access-key
+                - name: AWS_SECRET_ACCESS_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: secret-key
+                - name: RESTIC_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: restic-password
+                - name: S3_ENDPOINT
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: endpoint
+                - name: S3_BUCKET
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: bucket
+                - name: RESTIC_REPOSITORY
+                  value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/k8s-resources"
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  echo "=== backup-k8s-resources-restic started at $(date -u +%FT%TZ) ==="
+                  echo "Repository: ${RESTIC_REPOSITORY}"
+
+                  # First-run tolerance: init if the repo isn't there yet.
+                  if restic snapshots >/dev/null 2>&1; then
+                    echo "Repo exists, skipping init."
+                  else
+                    echo "Repo missing, initialising..."
+                    restic init
+                  fi
+
+                  echo "--- restic backup --stdin cluster.yaml ---"
+                  restic backup --stdin \
+                    --stdin-filename cluster.yaml \
+                    --tag k8s-resources \
+                    --host k3s < /dump/cluster.yaml
+
+                  echo "--- restic forget/prune ---"
+                  restic forget --tag k8s-resources \
+                    --keep-daily 7 \
+                    --keep-weekly 4 \
+                    --keep-monthly 6 \
+                    --prune
+
+                  echo "--- restic check --read-data-subset=5% ---"
+                  CHECK_STATUS=0
+                  restic check --read-data-subset=5% || CHECK_STATUS=$?
+                  echo "restic check exit: ${CHECK_STATUS}"
+
+                  # Textfile-collector metrics; identical wiring to the
+                  # loki/grafana siblings. Scrapeable once node-exporter's
+                  # textfile collector path is enabled — tracked in DEV-482.
+                  {
+                    echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
+                    echo "backup_k8s_resources_timestamp_seconds $(date +%s)"
+                    echo "backup_k8s_resources_check_status ${CHECK_STATUS}"
+                  } > /metrics/backup_k8s_resources.prom
+
+                  echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ==="
+                  exit ${CHECK_STATUS}
+              volumeMounts:
+                - name: dump
+                  mountPath: /dump
+                  readOnly: true
+                - name: metrics
+                  mountPath: /metrics
+                - name: cache
+                  mountPath: /root/.cache/restic
+              resources:
+                requests:
+                  cpu: 100m
+                  memory: 128Mi
+                limits:
+                  cpu: 1500m
+                  memory: 1Gi
           volumes:
-          - name: backup-storage
+            - name: dump
-            persistentVolumeClaim:
+              emptyDir: {}
-              claimName: backup-storage
             - name: metrics
               emptyDir: {}
+            - name: cache
+              emptyDir: {}