Commit graph

3 commits

Author SHA1 Message Date
CTO (Paperclip Agent)
ac9f36b1e7 feat(velero): monthly restore-drill CronJob for openwebui (DEV-575)
- CronJob at 0 4 1 * * restores the latest openwebui-manifests backup into
  a transient restore-test-<ts> namespace, diffs core resource-kind counts
  against the live openwebui namespace, and deletes the target namespace.
- ServiceAccount + ClusterRole scoped to velero.io + namespace + configmap
  ops. Script enforces the 'restore-test-' name prefix before deleting a
  namespace. Cluster-wide 'view' bound for the diff.
- Failure surfaces via kube_job_status_failed (DEV-574 alert stack) and via
  ConfigMap velero-restore-drill-last-result written on every run.
- Optional Paperclip webhook Secret is sealed but unreachable from Hetzner
  right now (home Paperclip is LAN-only); wired as optional secretKeyRef so
  the drill runs regardless. Connectivity is a follow-up.
- Argo Application velero-restore-drill points at velero/restore-drill/.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-12 18:03:37 +00:00
CTO (Paperclip Agent)
6a66924595 feat(velero): ServiceMonitor + PrometheusRule for backup alerts (DEV-574)
Adds Prometheus scrape of Velero's :8085 metrics and four alerts
(VeleroBackupFailed, VeleroBackupPartial, VeleroBackupMissed,
VeleroStorageLocationUnavailable) labeled team=infra severity=warning.

VeleroStorageLocationUnavailable substitutes for the spec's
VeleroRepoUnhealthy since Velero v1.18.1 does not expose
velero_backup_repository_available (see comment in the rule).

Wired via a new ArgoCD Application (velero-monitoring) that syncs
only servicemonitor.yaml + prometheusrule.yaml from basicstack-repo/velero/.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-12 17:58:45 +00:00
CTO (Paperclip Agent)
ec3e9624b3 feat(velero): schedules + PVC exclude annotations (DEV-573)
- velero/schedules/: cluster-all @ 03:00 (30d TTL, manifests only),
  per-namespace <ns>-manifests @ 03:15+n min (30 in-scope namespaces,
  30d TTL, no fs backup), and per-namespace <ns>-data @ 03:45+n min
  for the whitelist (pocket-id, vaultwarden, vikunja, passbolt;
  14d TTL, defaultVolumesToFsBackup=true).
- All schedules exclude velero, kube-system, kube-node-lease,
  kube-public so the Kopia password Secret never lands in S3.
- apps/app-velero-schedules.yaml: sibling Argo app syncing that
  directory into namespace velero with prune+selfHeal.
- opencloud, harbor(registry+trivy): pod-template annotation
  backup.velero.io/backup-volumes-excludes as a safety net so blob-
  heavy PVCs are never fs-backed-up even under a manual defaultVolumes
  run. nextcloud/monitoring workloads are not Argo-tracked yet — they
  were patched live in the same commit but need a follow-up to source-
  control them.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-12 17:35:49 +00:00