feat(monitoring): pull restic image from Harbor mirror (DEV-493)

Point the three monitoring restic CronJobs at the Harbor-hosted
mirror at harbor.basicstack.de/library/restic:0.17.3 so the backup
pipeline is not gated by Docker Hub throttling or upstream retag.

- Mirrored docker.io/restic/restic:0.17.3 into the public
  `library` project via a one-shot crane copy Job. Digest in
  Harbor matches the upstream index (sha256:8f5a62b4…).
- Updated the three CronJobs
  (backup-loki-restic, backup-grafana-restic,
  backup-k8s-resources) to pull from Harbor.
- Documented the tag-bump procedure in
  docs/monitoring/restic-restore.md (§ Tag-bump procedure) and
  pointed the restore-drill pod at the same Harbor image so the
  restore round-trip stays image-consistent with the writers.

Refs: [DEV-493](/DEV/issues/DEV-493), parent [DEV-482](/DEV/issues/DEV-482).

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/monitoring/backup-grafana-restic-cronjob.yaml

@@ -44,7 +44,10 @@ spec:
             kubernetes.io/hostname: k3s-worker-2
           containers:
             - name: restic
-              image: restic/restic:0.17.3
+              image: harbor.basicstack.de/library/restic:0.17.3
+              # Mirrored from docker.io/restic/restic:0.17.3 (DEV-493) —
+              # deterministic ingress via Harbor. Retag procedure in
+              # docs/monitoring/restic-restore.md § "Tag-bump procedure".
               env:
                 - name: AWS_ACCESS_KEY_ID
                   valueFrom:

apps/monitoring/backup-k8s-resources-cronjob.yaml

@@ -90,7 +90,10 @@ spec:
                   mountPath: /dump
           containers:
             - name: restic
-              image: restic/restic:0.17.3
+              image: harbor.basicstack.de/library/restic:0.17.3
+              # Mirrored from docker.io/restic/restic:0.17.3 (DEV-493) —
+              # deterministic ingress via Harbor. Retag procedure in
+              # docs/monitoring/restic-restore.md § "Tag-bump procedure".
               env:
                 - name: AWS_ACCESS_KEY_ID
                   valueFrom:

apps/monitoring/backup-loki-restic-cronjob.yaml

@@ -53,7 +53,10 @@ spec:
                   topologyKey: kubernetes.io/hostname
           containers:
             - name: restic
-              image: restic/restic:0.17.3
+              image: harbor.basicstack.de/library/restic:0.17.3
+              # Mirrored from docker.io/restic/restic:0.17.3 (DEV-493) —
+              # deterministic ingress via Harbor. Retag procedure in
+              # docs/monitoring/restic-restore.md § "Tag-bump procedure".
               env:
                 - name: AWS_ACCESS_KEY_ID
                   valueFrom:

docs/monitoring/restic-restore.md

@@ -13,13 +13,14 @@ event, or (b) run the quarterly restore drill that keeps the
 
 ## Repo layout
 
-Single bucket, three repo prefixes:
+Single bucket, four repo prefixes:
 
 | repo prefix                                    | source                                        | writer CronJob         | tag             |
-| ---------------------------------------------- | ---------------------------------- | ---------------------- | --------------- |
+| ---------------------------------------------- | --------------------------------------------- | ---------------------- | --------------- |
 | `restic/loki`                                  | PVC `loki-storage-encrypted`                  | `backup-loki-restic`   | `loki`          |
 | `restic/grafana`                               | PVC `grafana-storage` (worker-2)              | `backup-grafana-restic`| `grafana`       |
 | `restic/k8s-resources`                         | `kubectl get -o yaml` (stdin)                 | `backup-k8s-resources` | `k8s-resources` |
+| `restic/prometheus`                            | PVC `prometheus-data-encrypted` (excl. WAL)   | `prometheus-backup`    | `prometheus`    |
 
 Password + endpoint + bucket + AWS creds live in `SealedSecret`
 `monitoring-s3-backup` (namespace `monitoring`). Keys:
@@ -28,8 +29,7 @@ Password + endpoint + bucket + AWS creds live in `SealedSecret`
   locally, mirrored to Passbolt entry `restic / monitoring backups`.
 - `access-key`, `secret-key` — Hetzner S3 credentials.
 - `endpoint` — e.g. `https://hel1.your-objectstorage.com`.
-- `bucket` — Hetzner bucket name (also used by the legacy Prometheus
+- `bucket` — Hetzner bucket name (all four restic repos share it).
-  rclone job).
 
 Encryption is done **client-side by restic**. Hetzner Object Storage
 has no SSE-S3 / SSE-KMS ([FAQ](https://docs.hetzner.com/storage/object-storage/faq/general/));
@@ -70,7 +70,7 @@ spec:
   restartPolicy: Never
   containers:
     - name: restic
-      image: restic/restic:0.17.3            # matches CronJob image
+      image: harbor.basicstack.de/library/restic:0.17.3  # matches CronJob image (DEV-493)
       command: ["sleep", "3600"]
       env:
         - { name: AWS_ACCESS_KEY_ID,     valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: access-key } } }
@@ -127,6 +127,41 @@ restic restore latest --target /work/grafana --tag grafana
 
 Follows the loki pattern (restored tree under `/work/grafana/source/`).
 
+### Prometheus
+
+```sh
+export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus"
+restic snapshots
+restic restore latest --target /work/prometheus --tag prometheus
+find /work/prometheus/source -maxdepth 1 -type d | head
+```
+
+Restored tree lands under `/work/prometheus/source/…`. **WAL and
+`chunks_head/` are excluded on purpose** — see the "compaction race"
+notes at the top of `apps/monitoring/prometheus-backup-cronjob.yaml`.
+Expect the last ~15 s of ingested samples to be lost on recovery; the
+compacted 2h/24h blocks are complete.
+
+**Integrity check — `promtool tsdb analyze` (required for the DEV-492
+drill).** The snapshot must load cleanly through Prometheus's own
+verifier before we call the drill green.
+
+```sh
+kubectl -n restore-drill run promtool-check --rm -it --restart=Never \
+  --image=prom/prometheus:v2.53.1 \
+  --overrides='{"spec":{"containers":[{"name":"promtool-check","image":"prom/prometheus:v2.53.1","command":["sh","-c","for b in /work/prometheus/source/*/; do echo \"--- $b\"; promtool tsdb analyze /work/prometheus/source \"$(basename $b)\" || exit 1; done"],"volumeMounts":[{"name":"work","mountPath":"/work"}]}],"volumes":[{"name":"work","emptyDir":{}}]}}'
+```
+
+Simpler drill path (the restore pod already has restic; `promtool` is
+not shipped in `restic/restic`, so run it from a `prom/prometheus`
+sidecar or an interactive pod that mounts the same `emptyDir` — see
+DEV-492 drill entry for the concrete two-pod recipe used on the
+first run).
+
+Exit 0 on every block = pass. A non-zero exit on any block means the
+compaction-race mitigation slipped and the snapshot is corrupt.
+Re-run the CronJob and re-drill.
+
 ## Integrity checks
 
 Two layers.
@@ -198,6 +233,145 @@ restic key remove <old-id>
 
 Rotate annually or immediately on suspected compromise.
 
+## Restic image (Harbor mirror)
+
+All three restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
+`backup-k8s-resources`) reference a Harbor-hosted copy of upstream to
+keep the backup pipeline off the Docker Hub pull path and immune to
+upstream retagging:
+
+- Manifest reference: `harbor.basicstack.de/library/restic:0.17.3`
+- Upstream: `docker.io/restic/restic:0.17.3`
+- Ticket: [DEV-493](/DEV/issues/DEV-493)
+
+The `library` project is public, so no `imagePullSecret` is required
+on the CronJob pods.
+
+### Tag-bump procedure
+
+Run this when we want to move restic to a new pinned tag (e.g. 0.17.3
+→ 0.17.4). Do the mirror push **before** editing manifests so Argo
+cannot roll pods onto an unmirrored tag.
+
+1. **Pick and verify the upstream tag.** Confirm the tag exists on
+   Docker Hub and (ideally) read the upstream restic release notes for
+   breaking changes:
+
+   ```sh
+   curl -s "https://hub.docker.com/v2/repositories/restic/restic/tags/0.17.4"
+   ```
+
+2. **Mirror the image to Harbor** with a one-shot in-cluster crane
+   Job. Requires an admin (or `library`-scoped robot) Harbor
+   credential; the Job auth secret is throwaway.
+
+   ```sh
+   # from the CTO workstation with kubectl + Harbor admin access
+   NEW_TAG=0.17.4
+   ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
+     -o jsonpath='{.data.harborAdminPassword}' | base64 -d)
+   AUTH=$(printf 'admin:%s' "$ADMIN_PW" | base64 -w0)
+   cat >/tmp/dc.json <<JSON
+   {"auths":{"harbor.basicstack.de":{"auth":"$AUTH"}}}
+   JSON
+   kubectl -n harbor create secret generic crane-mirror-cfg \
+     --from-file=config.json=/tmp/dc.json \
+     --dry-run=client -o yaml | kubectl apply -f -
+   rm /tmp/dc.json
+
+   cat <<YAML | kubectl apply -f -
+   apiVersion: batch/v1
+   kind: Job
+   metadata:
+     name: crane-mirror-restic-${NEW_TAG//./-}
+     namespace: harbor
+   spec:
+     backoffLimit: 1
+     ttlSecondsAfterFinished: 600
+     template:
+       spec:
+         restartPolicy: Never
+         containers:
+           - name: crane
+             image: gcr.io/go-containerregistry/crane:v0.21.9
+             env: [{name: DOCKER_CONFIG, value: /docker}]
+             args:
+               - copy
+               - docker.io/restic/restic:${NEW_TAG}
+               - harbor.basicstack.de/library/restic:${NEW_TAG}
+             volumeMounts:
+               - {name: docker-config, mountPath: /docker, readOnly: true}
+         volumes:
+           - name: docker-config
+             secret: {secretName: crane-mirror-cfg}
+   YAML
+
+   kubectl -n harbor wait --for=condition=complete \
+     job/crane-mirror-restic-${NEW_TAG//./-} --timeout=5m
+   kubectl -n harbor logs job/crane-mirror-restic-${NEW_TAG//./-} | tail -20
+   ```
+
+3. **Sanity-check the artifact** — the Harbor digest must match the
+   digest crane just pushed, and an anonymous pull must resolve:
+
+   ```sh
+   ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
+     -o jsonpath='{.data.harborAdminPassword}' | base64 -d)
+   curl -sk -u "admin:$ADMIN_PW" \
+     "https://harbor.basicstack.de/api/v2.0/projects/library/repositories/restic/artifacts" \
+     | jq -r '.[] | "digest=\(.digest) tags=\((.tags//[])|map(.name)|join(","))"'
+   ```
+
+4. **Clean up the mirror secret and Job** (the Job also TTLs itself
+   in 10 min):
+
+   ```sh
+   kubectl -n harbor delete secret crane-mirror-cfg
+   kubectl -n harbor delete job crane-mirror-restic-${NEW_TAG//./-} \
+     --ignore-not-found
+   ```
+
+5. **Bump the manifest.** Update the three files under `apps/monitoring/`
+   in the `basicstack-repo`:
+
+   - `backup-loki-restic-cronjob.yaml`
+   - `backup-grafana-restic-cronjob.yaml`
+   - `backup-k8s-resources-cronjob.yaml`
+
+   Each references `harbor.basicstack.de/library/restic:<tag>` under
+   the `restic` container. Bump `<tag>` in all three, commit, push.
+
+6. **Let Argo sync** — the `monitoring` Application picks up the new
+   manifests. Watch for successful reconciliation:
+
+   ```sh
+   kubectl -n argocd get application monitoring \
+     -o jsonpath='{.status.sync.status}{"  "}{.status.health.status}{"\n"}'
+   ```
+
+7. **Trigger one CronJob run to prove the Harbor pull is green.**
+   Create a manual Job from any of the CronJobs (loki is fine) and
+   inspect its logs:
+
+   ```sh
+   kubectl -n monitoring create job --from=cronjob/backup-loki-restic \
+     smoke-restic-$(date +%Y%m%d-%H%M%S)
+   ```
+
+   The Pod should pull from `harbor.basicstack.de/library/restic:<tag>`
+   (`kubectl describe pod ...` → Events → `Pulling image …`) and
+   finish with `restic backup /source` output ending in a snapshot id.
+
+8. **Update this doc's "restore pod" template** if the new tag is
+   incompatible with the existing restore pod command — the drill pod
+   image must match the CronJob image, otherwise the restore-drill
+   won't round-trip.
+
+**Rollback:** if the new tag misbehaves, edit the three manifests back
+to the last-known-good tag, commit, and let Argo re-sync. The old
+tag's manifest remains in Harbor until it's explicitly deleted, so
+rollback is a manifest change only.
+
 ## Drill log
 
 Every drill appends to this section. Include: