From e434fa751b43d4913bdcc9911d0ae7d2992537f7 Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 16 Aug 2026 15:39:06 +0000 Subject: [PATCH] =?UTF-8?q?feat(monitoring):=20backup-k8s-resources=20?= =?UTF-8?q?=E2=86=92=20restic=20--stdin=20to=20Hetzner=20S3=20(DEV-487)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrites the backup-k8s-resources CronJob to drop the local-path tarball path and stream a YAML dump through `restic backup --stdin` into `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern: init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3` reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` / `_check_status`). Deployed in parallel with the legacy `backup-volumes` CronJob — the `backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6 (restore drill). Server-side dry-run validated on the k3s control plane. Co-Authored-By: Paperclip --- apps/monitoring/README.md | 6 +- .../backup-k8s-resources-cronjob.yaml | 206 +++++++++++++----- 2 files changed, 160 insertions(+), 52 deletions(-) diff --git a/apps/monitoring/README.md b/apps/monitoring/README.md index 85efc21..08f3036 100644 --- a/apps/monitoring/README.md +++ b/apps/monitoring/README.md @@ -2,12 +2,12 @@ Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context. -- `backup-k8s-resources-cronjob.yaml` — daily dump of Kubernetes resources into `backup-storage` PVC. +- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)). - `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below). Being retired by the restic pipeline in [DEV-482](/DEV/issues/DEV-482) — keep running until step 6 (restore drill passed). - `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)). - `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)). - `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)). -The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for `backup-k8s-resources` and `backup-volumes`. +The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the destination for `backup-volumes` only. `backup-k8s-resources` no longer writes there — the tarball path was retired in [DEV-487](/DEV/issues/DEV-487). The PVC is scheduled for retirement together with `backup-volumes` once [DEV-482](/DEV/issues/DEV-482) step 6 (restore drill) has passed. -`backup-k8s-resources` and `backup-volumes` pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used there. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`. +`backup-volumes` pins itself to `k3s-worker-2` via `nodeSelector` because that is the node that holds its destination PVC. `backup-k8s-resources` is no longer node-pinned. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`. diff --git a/apps/monitoring/backup-k8s-resources-cronjob.yaml b/apps/monitoring/backup-k8s-resources-cronjob.yaml index 6a6f64a..9e54831 100644 --- a/apps/monitoring/backup-k8s-resources-cronjob.yaml +++ b/apps/monitoring/backup-k8s-resources-cronjob.yaml @@ -1,3 +1,23 @@ +--- +# Kubernetes-resource backup via restic to Hetzner Object Storage +# (DEV-487, DEV-482 Option 4). Step 4 of the Option 4 rollout. +# +# Streams a concatenated YAML dump of cluster-scoped and per-namespace +# resources through `restic backup --stdin` into +# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources`. No PVC mount +# (drops the local-path `backup-storage` dependency), no node pin. +# +# Two-container pattern: +# 1. `kubectl-dump` init container (`alpine/k8s:1.29.4`) writes +# /dump/cluster.yaml into an emptyDir. Uses `serviceAccountName: +# backup-sa` (unchanged from the legacy job). +# 2. `restic` main container (`restic/restic:0.17.3`, matches the +# loki/grafana siblings) reads that file on stdin and streams it +# into the restic repo with `--stdin-filename cluster.yaml`. +# +# Deployed in parallel with the legacy `backup-volumes` CronJob — do +# not retire that job or the `backup-storage` PVC until DEV-482 step 6 +# (restore drill) has passed. apiVersion: batch/v1 kind: CronJob metadata: @@ -6,6 +26,7 @@ metadata: labels: app: backup type: k8s-resources + backend: restic spec: schedule: "0 2 * * *" concurrencyPolicy: Forbid @@ -13,72 +34,159 @@ spec: failedJobsHistoryLimit: 3 jobTemplate: metadata: - annotations: - prometheus.io/scrape: "true" labels: app: backup type: k8s-resources + backend: restic spec: backoffLimit: 2 + activeDeadlineSeconds: 3600 template: metadata: labels: app: backup + type: k8s-resources + backend: restic spec: restartPolicy: OnFailure serviceAccountName: backup-sa - # backup-storage PVC (local-path) is bound to k3s-worker-2, so pin here. - nodeSelector: - kubernetes.io/hostname: k3s-worker-2 + initContainers: + - name: kubectl-dump + image: alpine/k8s:1.29.4 + command: + - /bin/sh + - -c + - | + set -eu + echo "=== kubectl-dump started at $(date -u +%FT%TZ) ===" + DUMP=/dump/cluster.yaml + : > "${DUMP}" + + echo "--- namespaces ---" + kubectl get namespaces -o yaml >> "${DUMP}" + echo "---" >> "${DUMP}" + + echo "--- cluster-scoped resources ---" + kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \ + -o yaml >> "${DUMP}" + echo "---" >> "${DUMP}" + + echo "--- namespaced resources ---" + for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do + echo " ns=${ns}" + kubectl get \ + configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \ + -n "${ns}" -o yaml >> "${DUMP}" 2>/dev/null || true + echo "---" >> "${DUMP}" + done + + echo "dump size: $(wc -c < ${DUMP}) bytes" + echo "=== kubectl-dump finished at $(date -u +%FT%TZ) ===" + resources: + requests: + cpu: 50m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + volumeMounts: + - name: dump + mountPath: /dump containers: - - name: kubectl-backup - image: alpine/k8s:1.29.4 - command: - - /bin/sh - - -c - - | - set -e - BACKUP_DIR="/backup/k8s-$(date +%Y%m%d-%H%M%S)" - mkdir -p "$BACKUP_DIR" + - name: restic + image: restic/restic:0.17.3 + env: + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: access-key + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: secret-key + - name: RESTIC_PASSWORD + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: restic-password + - name: S3_ENDPOINT + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: endpoint + - name: S3_BUCKET + valueFrom: + secretKeyRef: + name: monitoring-s3-backup + key: bucket + - name: RESTIC_REPOSITORY + value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/k8s-resources" + command: + - /bin/sh + - -c + - | + set -eu + echo "=== backup-k8s-resources-restic started at $(date -u +%FT%TZ) ===" + echo "Repository: ${RESTIC_REPOSITORY}" - echo "Starting Kubernetes resources backup to $BACKUP_DIR" + # First-run tolerance: init if the repo isn't there yet. + if restic snapshots >/dev/null 2>&1; then + echo "Repo exists, skipping init." + else + echo "Repo missing, initialising..." + restic init + fi - kubectl get namespaces -o yaml > "$BACKUP_DIR/namespaces.yaml" + echo "--- restic backup --stdin cluster.yaml ---" + restic backup --stdin \ + --stdin-filename cluster.yaml \ + --tag k8s-resources \ + --host k3s < /dump/cluster.yaml - for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do - mkdir -p "$BACKUP_DIR/$ns" - kubectl get configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \ - -n "$ns" -o yaml > "$BACKUP_DIR/$ns/resources.yaml" 2>/dev/null || true - done + echo "--- restic forget/prune ---" + restic forget --tag k8s-resources \ + --keep-daily 7 \ + --keep-weekly 4 \ + --keep-monthly 6 \ + --prune - kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \ - -o yaml > "$BACKUP_DIR/cluster-resources.yaml" + echo "--- restic check --read-data-subset=5% ---" + CHECK_STATUS=0 + restic check --read-data-subset=5% || CHECK_STATUS=$? + echo "restic check exit: ${CHECK_STATUS}" - cd /backup - tar -czf "k8s-backup-$(date +%Y%m%d-%H%M%S).tar.gz" "$(basename $BACKUP_DIR)" - rm -rf "$BACKUP_DIR" + # Textfile-collector metrics; identical wiring to the + # loki/grafana siblings. Scrapeable once node-exporter's + # textfile collector path is enabled — tracked in DEV-482. + { + echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)" + echo "backup_k8s_resources_timestamp_seconds $(date +%s)" + echo "backup_k8s_resources_check_status ${CHECK_STATUS}" + } > /metrics/backup_k8s_resources.prom - find /backup -name "k8s-backup-*.tar.gz" -mtime +7 -delete - - echo "Backup completed successfully" - - echo "backup_k8s_resources_success 1" > /metrics/backup_success.prom - echo "backup_k8s_resources_timestamp $(date +%s)" >> /metrics/backup_success.prom - resources: - requests: - cpu: 50m - memory: 128Mi - limits: - cpu: 500m - memory: 384Mi - volumeMounts: - - mountPath: /backup - name: backup-storage - - mountPath: /metrics - name: metrics + echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ===" + exit ${CHECK_STATUS} + volumeMounts: + - name: dump + mountPath: /dump + readOnly: true + - name: metrics + mountPath: /metrics + - name: cache + mountPath: /root/.cache/restic + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 1500m + memory: 1Gi volumes: - - name: backup-storage - persistentVolumeClaim: - claimName: backup-storage - - name: metrics - emptyDir: {} + - name: dump + emptyDir: {} + - name: metrics + emptyDir: {} + - name: cache + emptyDir: {}