basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feat(monitoring): pull restic image from Harbor mirror (DEV-493)

Browse source 
Point the three monitoring restic CronJobs at the Harbor-hosted
mirror at harbor.basicstack.de/library/restic:0.17.3 so the backup
pipeline is not gated by Docker Hub throttling or upstream retag.

- Mirrored docker.io/restic/restic:0.17.3 into the public
  `library` project via a one-shot crane copy Job. Digest in
  Harbor matches the upstream index (sha256:8f5a62b4…).
- Updated the three CronJobs
  (backup-loki-restic, backup-grafana-restic,
  backup-k8s-resources) to pull from Harbor.
- Documented the tag-bump procedure in
  docs/monitoring/restic-restore.md (§ Tag-bump procedure) and
  pointed the restore-drill pod at the same Harbor image so the
  restore round-trip stays image-consistent with the writers.

Refs: [DEV-493](/DEV/issues/DEV-493), parent [DEV-482](/DEV/issues/DEV-482).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-08-16 16:20:06 +00:00
parent 550d1276f7
commit b1680431c6
4 changed files with 195 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
apps/monitoring/backup-grafana-restic-cronjob.yaml
View file

@ -44,7 +44,10 @@ spec:
kubernetes.io/hostname: k3s-worker-2
containers:
- name: restic
image: restic/restic:0.17.3
image: harbor.basicstack.de/library/restic:0.17.3
# Mirrored from docker.io/restic/restic:0.17.3 (DEV-493) —
# deterministic ingress via Harbor. Retag procedure in
# docs/monitoring/restic-restore.md § "Tag-bump procedure".
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:

5
apps/monitoring/backup-k8s-resources-cronjob.yaml
View file

@ -90,7 +90,10 @@ spec:
mountPath: /dump
containers:
- name: restic
image: restic/restic:0.17.3
image: harbor.basicstack.de/library/restic:0.17.3
# Mirrored from docker.io/restic/restic:0.17.3 (DEV-493) —
# deterministic ingress via Harbor. Retag procedure in
# docs/monitoring/restic-restore.md § "Tag-bump procedure".
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:

5
apps/monitoring/backup-loki-restic-cronjob.yaml
View file

@ -53,7 +53,10 @@ spec:
topologyKey: kubernetes.io/hostname
containers:
- name: restic
image: restic/restic:0.17.3
image: harbor.basicstack.de/library/restic:0.17.3
# Mirrored from docker.io/restic/restic:0.17.3 (DEV-493) —
# deterministic ingress via Harbor. Retag procedure in
# docs/monitoring/restic-restore.md § "Tag-bump procedure".
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:

184
docs/monitoring/restic-restore.md
View file

@ -13,13 +13,14 @@ event, or (b) run the quarterly restore drill that keeps the
## Repo layout
Single bucket, three repo prefixes:
Single bucket, four repo prefixes:
| repo prefix | source | writer CronJob | tag |
| ---------------------------------------------- | ---------------------------------- | ---------------------- | --------------- |
| ---------------------------------------------- | --------------------------------------------- | ---------------------- | --------------- |
| `restic/loki` | PVC `loki-storage-encrypted` | `backup-loki-restic` | `loki` |
| `restic/grafana` | PVC `grafana-storage` (worker-2) | `backup-grafana-restic`| `grafana` |
| `restic/k8s-resources` | `kubectl get -o yaml` (stdin) | `backup-k8s-resources` | `k8s-resources` |
| `restic/prometheus` | PVC `prometheus-data-encrypted` (excl. WAL) | `prometheus-backup` | `prometheus` |
Password + endpoint + bucket + AWS creds live in `SealedSecret`
`monitoring-s3-backup` (namespace `monitoring`). Keys:
@ -28,8 +29,7 @@ Password + endpoint + bucket + AWS creds live in `SealedSecret`
locally, mirrored to Passbolt entry `restic / monitoring backups`.
- `access-key`, `secret-key` — Hetzner S3 credentials.
- `endpoint` — e.g. `https://hel1.your-objectstorage.com`.
- `bucket` — Hetzner bucket name (also used by the legacy Prometheus
rclone job).
- `bucket` — Hetzner bucket name (all four restic repos share it).
Encryption is done **client-side by restic**. Hetzner Object Storage
has no SSE-S3 / SSE-KMS ([FAQ](https://docs.hetzner.com/storage/object-storage/faq/general/));
@ -70,7 +70,7 @@ spec:
restartPolicy: Never
containers:
- name: restic
image: restic/restic:0.17.3 # matches CronJob image
image: harbor.basicstack.de/library/restic:0.17.3 # matches CronJob image (DEV-493)
command: ["sleep", "3600"]
env:
- { name: AWS_ACCESS_KEY_ID, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: access-key } } }
@ -127,6 +127,41 @@ restic restore latest --target /work/grafana --tag grafana
Follows the loki pattern (restored tree under `/work/grafana/source/`).
### Prometheus
```sh
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus"
restic snapshots
restic restore latest --target /work/prometheus --tag prometheus
find /work/prometheus/source -maxdepth 1 -type d | head
```
Restored tree lands under `/work/prometheus/source/…`. **WAL and
`chunks_head/` are excluded on purpose** — see the "compaction race"
notes at the top of `apps/monitoring/prometheus-backup-cronjob.yaml`.
Expect the last ~15 s of ingested samples to be lost on recovery; the
compacted 2h/24h blocks are complete.
**Integrity check — `promtool tsdb analyze` (required for the DEV-492
drill).** The snapshot must load cleanly through Prometheus's own
verifier before we call the drill green.
```sh
kubectl -n restore-drill run promtool-check --rm -it --restart=Never \
--image=prom/prometheus:v2.53.1 \
--overrides='{"spec":{"containers":[{"name":"promtool-check","image":"prom/prometheus:v2.53.1","command":["sh","-c","for b in /work/prometheus/source/*/; do echo \"--- $b\"; promtool tsdb analyze /work/prometheus/source \"$(basename $b)\" || exit 1; done"],"volumeMounts":[{"name":"work","mountPath":"/work"}]}],"volumes":[{"name":"work","emptyDir":{}}]}}'
```
Simpler drill path (the restore pod already has restic; `promtool` is
not shipped in `restic/restic`, so run it from a `prom/prometheus`
sidecar or an interactive pod that mounts the same `emptyDir` — see
DEV-492 drill entry for the concrete two-pod recipe used on the
first run).
Exit 0 on every block = pass. A non-zero exit on any block means the
compaction-race mitigation slipped and the snapshot is corrupt.
Re-run the CronJob and re-drill.
## Integrity checks
Two layers.
@ -198,6 +233,145 @@ restic key remove <old-id>
Rotate annually or immediately on suspected compromise.
## Restic image (Harbor mirror)
All three restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
`backup-k8s-resources`) reference a Harbor-hosted copy of upstream to
keep the backup pipeline off the Docker Hub pull path and immune to
upstream retagging:
- Manifest reference: `harbor.basicstack.de/library/restic:0.17.3`
- Upstream: `docker.io/restic/restic:0.17.3`
- Ticket: [DEV-493](/DEV/issues/DEV-493)
The `library` project is public, so no `imagePullSecret` is required
on the CronJob pods.
### Tag-bump procedure
Run this when we want to move restic to a new pinned tag (e.g. 0.17.3
→ 0.17.4). Do the mirror push **before** editing manifests so Argo
cannot roll pods onto an unmirrored tag.
1. **Pick and verify the upstream tag.** Confirm the tag exists on
Docker Hub and (ideally) read the upstream restic release notes for
breaking changes:
```sh
curl -s "https://hub.docker.com/v2/repositories/restic/restic/tags/0.17.4"
```
2. **Mirror the image to Harbor** with a one-shot in-cluster crane
Job. Requires an admin (or `library`-scoped robot) Harbor
credential; the Job auth secret is throwaway.
```sh
# from the CTO workstation with kubectl + Harbor admin access
NEW_TAG=0.17.4
ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
-o jsonpath='{.data.harborAdminPassword}' | base64 -d)
AUTH=$(printf 'admin:%s' "$ADMIN_PW" | base64 -w0)
cat >/tmp/dc.json <<JSON
{"auths":{"harbor.basicstack.de":{"auth":"$AUTH"}}}
JSON
kubectl -n harbor create secret generic crane-mirror-cfg \
--from-file=config.json=/tmp/dc.json \
--dry-run=client -o yaml | kubectl apply -f -
rm /tmp/dc.json
cat <<YAML | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
name: crane-mirror-restic-${NEW_TAG//./-}
namespace: harbor
spec:
backoffLimit: 1
ttlSecondsAfterFinished: 600
template:
spec:
restartPolicy: Never
containers:
- name: crane
image: gcr.io/go-containerregistry/crane:v0.21.9
env: [{name: DOCKER_CONFIG, value: /docker}]
args:
- copy
- docker.io/restic/restic:${NEW_TAG}
- harbor.basicstack.de/library/restic:${NEW_TAG}
volumeMounts:
- {name: docker-config, mountPath: /docker, readOnly: true}
volumes:
- name: docker-config
secret: {secretName: crane-mirror-cfg}
YAML
kubectl -n harbor wait --for=condition=complete \
job/crane-mirror-restic-${NEW_TAG//./-} --timeout=5m
kubectl -n harbor logs job/crane-mirror-restic-${NEW_TAG//./-} | tail -20
```
3. **Sanity-check the artifact** — the Harbor digest must match the
digest crane just pushed, and an anonymous pull must resolve:
```sh
ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
-o jsonpath='{.data.harborAdminPassword}' | base64 -d)
curl -sk -u "admin:$ADMIN_PW" \
"https://harbor.basicstack.de/api/v2.0/projects/library/repositories/restic/artifacts" \
| jq -r '.[] | "digest=\(.digest) tags=\((.tags//[])|map(.name)|join(","))"'
```
4. **Clean up the mirror secret and Job** (the Job also TTLs itself
in 10 min):
```sh
kubectl -n harbor delete secret crane-mirror-cfg
kubectl -n harbor delete job crane-mirror-restic-${NEW_TAG//./-} \
--ignore-not-found
```
5. **Bump the manifest.** Update the three files under `apps/monitoring/`
in the `basicstack-repo`:
- `backup-loki-restic-cronjob.yaml`
- `backup-grafana-restic-cronjob.yaml`
- `backup-k8s-resources-cronjob.yaml`
Each references `harbor.basicstack.de/library/restic:<tag>` under
the `restic` container. Bump `<tag>` in all three, commit, push.
6. **Let Argo sync** — the `monitoring` Application picks up the new
manifests. Watch for successful reconciliation:
```sh
kubectl -n argocd get application monitoring \
-o jsonpath='{.status.sync.status}{" "}{.status.health.status}{"\n"}'
```
7. **Trigger one CronJob run to prove the Harbor pull is green.**
Create a manual Job from any of the CronJobs (loki is fine) and
inspect its logs:
```sh
kubectl -n monitoring create job --from=cronjob/backup-loki-restic \
smoke-restic-$(date +%Y%m%d-%H%M%S)
```
The Pod should pull from `harbor.basicstack.de/library/restic:<tag>`
(`kubectl describe pod ...` → Events → `Pulling image …`) and
finish with `restic backup /source` output ending in a snapshot id.
8. **Update this doc's "restore pod" template** if the new tag is
incompatible with the existing restore pod command — the drill pod
image must match the CronJob image, otherwise the restore-drill
won't round-trip.
**Rollback:** if the new tag misbehaves, edit the three manifests back
to the last-known-good tag, commit, and let Argo re-sync. The old
tag's manifest remains in Harbor until it's explicitly deleted, so
rollback is a manifest change only.
## Drill log
Every drill appends to this section. Include: