Add key restic-password to SealedSecret monitoring-s3-backup so the
upcoming restic CronJobs (DEV-485/486/487) can mount it as
RESTIC_PASSWORD without any further secret rotation. Value is a fresh
32-byte openssl-random base64 password, sealed strict-scope against
the running sealed-secrets-controller. Server-side dry-run of the
new SealedSecret reports "configured".
Bucket write probe against s3://basicstack-backup/ succeeded from an
in-cluster amazon/aws-cli pod using the existing access-key/secret-key
in the same SealedSecret (ls, put, stat, delete).
Recovery-key copy for Passbolt is held root-only on the control plane
at /root/dev-484/restic-password.txt pending the human-side Passbolt
paste; that step tracks as follow-up.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
After the DEV-464 split, `monitoring/backup-volumes` only backs up
grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving
Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner
Cloud volume attached to whichever node currently runs the Prometheus
pod (typically k3s-worker-1), so it cannot join the shared backup-volumes
job without provoking Multi-Attach errors.
This introduces a dedicated `monitoring/prometheus-backup` CronJob that:
- Streams `prometheus-data-encrypted` to Hetzner S3 via rclone
(`basicstack-backup/prometheus/prometheus-<DATE>/`).
- Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO
PVC always attaches on the same node.
- Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline.
- Retains 7 days of dated backups (rclone delete --min-age 7d).
- Tolerates the expected TSDB compaction race (Prometheus deletes old
block dirs mid-copy): rclone's non-zero exit from those transient
errors is captured, then success is validated by comparing dest
bytes to source bytes (>= 80% and > 100 MiB floor).
S3 credentials are the same Hetzner Object Storage account used by
`opencloud-backup` and `stalwart-backup`, resealed for the `monitoring`
namespace as `SealedSecret monitoring-s3-backup`.
Verified with a manual job on k3s-worker-1 (see DEV-465 for logs).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
