Rewrites the backup-k8s-resources CronJob to drop the local-path tarball
path and stream a YAML dump through `restic backup --stdin` into
`hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern:
init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3`
reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops
the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling
loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and
textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` /
`_check_status`).
Deployed in parallel with the legacy `backup-volumes` CronJob — the
`backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6
(restore drill).
Server-side dry-run validated on the k3s control plane.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add a daily 03:15 UTC restic backup of the grafana-storage PVC to
s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana. Pinned to
k3s-worker-2 via nodeSelector because grafana-storage is a
local-path PV anchored there — no podAffinity needed. Same restic
retention as the loki sibling (7d/4w/6m + prune + 5% check) with
metrics backup_grafana_{success,timestamp_seconds,check_status}
written to the emptyDir textfile path. Deployed in parallel with
the legacy backup-volumes CronJob (Option 4 rollout, DEV-482).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Step 2 of the DEV-482 Option 4 rollout. Adds a restic-based Loki
backup that streams the loki-storage-encrypted PVC (mounted RO) to
s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki, client-side encrypted by
restic. Co-schedules with the Loki pod via podAffinity so it lands on
whichever worker holds the RWO VolumeAttachment.
Verified with a manual run in-cluster:
- snapshot 98c6fee6 written, listable via a fresh restic pod
- restic check --read-data-subset=5% clean
- resource requests dropped from the plan's 200m/256Mi to 100m/128Mi
because worker-2 has ~150m free CPU (Loki + Grafana + backup-* live
there); limits stay generous for pack/check bursts.
Runs in parallel with the legacy backup-volumes CronJob — DEV-482
step 6 will retire that job only after the restore drill passes.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
After the DEV-464 split, `monitoring/backup-volumes` only backs up
grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving
Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner
Cloud volume attached to whichever node currently runs the Prometheus
pod (typically k3s-worker-1), so it cannot join the shared backup-volumes
job without provoking Multi-Attach errors.
This introduces a dedicated `monitoring/prometheus-backup` CronJob that:
- Streams `prometheus-data-encrypted` to Hetzner S3 via rclone
(`basicstack-backup/prometheus/prometheus-<DATE>/`).
- Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO
PVC always attaches on the same node.
- Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline.
- Retains 7 days of dated backups (rclone delete --min-age 7d).
- Tolerates the expected TSDB compaction race (Prometheus deletes old
block dirs mid-copy): rclone's non-zero exit from those transient
errors is captured, then success is validated by comparing dest
bytes to source bytes (>= 80% and > 100 MiB floor).
S3 credentials are the same Hetzner Object Storage account used by
`opencloud-backup` and `stalwart-backup`, resealed for the `monitoring`
namespace as `SealedSecret monitoring-s3-backup`.
Verified with a manual job on k3s-worker-1 (see DEV-465 for logs).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Create the missing forgejo/platform-backup-data PVC that forgejo-backup
references (20Gi hcloud-volumes-encrypted).
- Record monitoring/backup-k8s-resources with a k3s-worker-2 nodeSelector
(backup-storage PVC is local-path pinned there), lower memory request
(128Mi) so it fits worker-2 pressure, and switch to alpine/k8s image
(bitnami/kubectl is no longer resolvable).
- Rewrite monitoring/backup-volumes to only back up grafana + loki
co-located with backup-storage on k3s-worker-2. Prometheus data
lives on k3s-worker-1 and is intentionally excluded here; a
dedicated Prometheus data backup follows in a separate ticket.
The three CronJobs previously left Pending/ContainerCreating pods that
blocked the OS-update health guard in DEV-463.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
