basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
stack.basicstack.de/apps/monitoring/README.md

14 lines
2.9 KiB
Markdown
Raw Normal View History

fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464) - Create the missing forgejo/platform-backup-data PVC that forgejo-backup references (20Gi hcloud-volumes-encrypted). - Record monitoring/backup-k8s-resources with a k3s-worker-2 nodeSelector (backup-storage PVC is local-path pinned there), lower memory request (128Mi) so it fits worker-2 pressure, and switch to alpine/k8s image (bitnami/kubectl is no longer resolvable). - Rewrite monitoring/backup-volumes to only back up grafana + loki co-located with backup-storage on k3s-worker-2. Prometheus data lives on k3s-worker-1 and is intentionally excluded here; a dedicated Prometheus data backup follows in a separate ticket. The three CronJobs previously left Pending/ContainerCreating pods that blocked the OS-update health guard in DEV-463. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-09 16:50:51 +00:00
# monitoring — backup CronJobs
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
feat(monitoring): backup-k8s-resources → restic --stdin to Hetzner S3 (DEV-487) Rewrites the backup-k8s-resources CronJob to drop the local-path tarball path and stream a YAML dump through `restic backup --stdin` into `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern: init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3` reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` / `_check_status`). Deployed in parallel with the legacy `backup-volumes` CronJob — the `backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6 (restore drill). Server-side dry-run validated on the k3s control plane. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 15:39:06 +00:00
- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
feat(monitoring): backup-loki-restic CronJob → Hetzner S3 (DEV-485) Step 2 of the DEV-482 Option 4 rollout. Adds a restic-based Loki backup that streams the loki-storage-encrypted PVC (mounted RO) to s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki, client-side encrypted by restic. Co-schedules with the Loki pod via podAffinity so it lands on whichever worker holds the RWO VolumeAttachment. Verified with a manual run in-cluster: - snapshot 98c6fee6 written, listable via a fresh restic pod - restic check --read-data-subset=5% clean - resource requests dropped from the plan's 200m/256Mi to 100m/128Mi because worker-2 has ~150m free CPU (Loki + Grafana + backup-* live there); limits stay generous for pack/check bursts. Runs in parallel with the legacy backup-volumes CronJob — DEV-482 step 6 will retire that job only after the restore drill passes. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 15:27:40 +00:00
- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below). Being retired by the restic pipeline in [DEV-482](/DEV/issues/DEV-482) — keep running until step 6 (restore drill passed).
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
feat(monitoring): backup-grafana-restic CronJob → Hetzner S3 (DEV-486) Add a daily 03:15 UTC restic backup of the grafana-storage PVC to s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana. Pinned to k3s-worker-2 via nodeSelector because grafana-storage is a local-path PV anchored there — no podAffinity needed. Same restic retention as the loki sibling (7d/4w/6m + prune + 5% check) with metrics backup_grafana_{success,timestamp_seconds,check_status} written to the emptyDir textfile path. Deployed in parallel with the legacy backup-volumes CronJob (Option 4 rollout, DEV-482). Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 15:34:37 +00:00
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
Add dedicated Prometheus data backup CronJob (DEV-465) After the DEV-464 split, `monitoring/backup-volumes` only backs up grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner Cloud volume attached to whichever node currently runs the Prometheus pod (typically k3s-worker-1), so it cannot join the shared backup-volumes job without provoking Multi-Attach errors. This introduces a dedicated `monitoring/prometheus-backup` CronJob that: - Streams `prometheus-data-encrypted` to Hetzner S3 via rclone (`basicstack-backup/prometheus/prometheus-<DATE>/`). - Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO PVC always attaches on the same node. - Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline. - Retains 7 days of dated backups (rclone delete --min-age 7d). - Tolerates the expected TSDB compaction race (Prometheus deletes old block dirs mid-copy): rclone's non-zero exit from those transient errors is captured, then success is validated by comparing dest bytes to source bytes (>= 80% and > 100 MiB floor). S3 credentials are the same Hetzner Object Storage account used by `opencloud-backup` and `stalwart-backup`, resealed for the `monitoring` namespace as `SealedSecret monitoring-s3-backup`. Verified with a manual job on k3s-worker-1 (see DEV-465 for logs). Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-09 17:18:21 +00:00
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)).
fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464) - Create the missing forgejo/platform-backup-data PVC that forgejo-backup references (20Gi hcloud-volumes-encrypted). - Record monitoring/backup-k8s-resources with a k3s-worker-2 nodeSelector (backup-storage PVC is local-path pinned there), lower memory request (128Mi) so it fits worker-2 pressure, and switch to alpine/k8s image (bitnami/kubectl is no longer resolvable). - Rewrite monitoring/backup-volumes to only back up grafana + loki co-located with backup-storage on k3s-worker-2. Prometheus data lives on k3s-worker-1 and is intentionally excluded here; a dedicated Prometheus data backup follows in a separate ticket. The three CronJobs previously left Pending/ContainerCreating pods that blocked the OS-update health guard in DEV-463. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-09 16:50:51 +00:00
feat(monitoring): backup-k8s-resources → restic --stdin to Hetzner S3 (DEV-487) Rewrites the backup-k8s-resources CronJob to drop the local-path tarball path and stream a YAML dump through `restic backup --stdin` into `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern: init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3` reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` / `_check_status`). Deployed in parallel with the legacy `backup-volumes` CronJob — the `backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6 (restore drill). Server-side dry-run validated on the k3s control plane. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 15:39:06 +00:00
The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the destination for `backup-volumes` only. `backup-k8s-resources` no longer writes there — the tarball path was retired in [DEV-487](/DEV/issues/DEV-487). The PVC is scheduled for retirement together with `backup-volumes` once [DEV-482](/DEV/issues/DEV-482) step 6 (restore drill) has passed.
fix(backups): Repair three broken CronJobs blocking weekly OS updates (DEV-464) - Create the missing forgejo/platform-backup-data PVC that forgejo-backup references (20Gi hcloud-volumes-encrypted). - Record monitoring/backup-k8s-resources with a k3s-worker-2 nodeSelector (backup-storage PVC is local-path pinned there), lower memory request (128Mi) so it fits worker-2 pressure, and switch to alpine/k8s image (bitnami/kubectl is no longer resolvable). - Rewrite monitoring/backup-volumes to only back up grafana + loki co-located with backup-storage on k3s-worker-2. Prometheus data lives on k3s-worker-1 and is intentionally excluded here; a dedicated Prometheus data backup follows in a separate ticket. The three CronJobs previously left Pending/ContainerCreating pods that blocked the OS-update health guard in DEV-463. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-09 16:50:51 +00:00
feat(monitoring): backup-k8s-resources → restic --stdin to Hetzner S3 (DEV-487) Rewrites the backup-k8s-resources CronJob to drop the local-path tarball path and stream a YAML dump through `restic backup --stdin` into `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern: init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3` reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` / `_check_status`). Deployed in parallel with the legacy `backup-volumes` CronJob — the `backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6 (restore drill). Server-side dry-run validated on the k3s control plane. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-08-16 15:39:06 +00:00
`backup-volumes` pins itself to `k3s-worker-2` via `nodeSelector` because that is the node that holds its destination PVC. `backup-k8s-resources` is no longer node-pinned. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`.
Reference in a new issue Copy permalink