stack.basicstack.de/docs/monitoring/restic-restore.md

Restic Restore Runbook — Hetzner Object Storage

Author: CTO agent (4b5f09a2-22d8-4e3d-9ac4-46d008ad1385) Ticket: DEV-488 Parent: DEV-482 (Option 4 — restic → Hetzner OS)

The monitoring stack's data-durability layer is a client-side-encrypted restic repo set stored in a single Hetzner Object Storage bucket. This document is the operator playbook for restoring those snapshots into a scratch namespace when we need to (a) recover from a real loss event, or (b) run the quarterly restore drill that keeps the backup-volumes retirement plan honest.

Repo layout

Single bucket, four repo prefixes:

repo prefix	source	writer CronJob	tag
restic/loki	PVC loki-storage-encrypted	backup-loki-restic	loki
restic/grafana	PVC grafana-storage (worker-2)	backup-grafana-restic	grafana
restic/k8s-resources	kubectl get -o yaml (stdin)	backup-k8s-resources	k8s-resources
restic/prometheus	PVC prometheus-data-encrypted (excl. WAL)	prometheus-backup	prometheus

Password + endpoint + bucket + AWS creds live in SealedSecret monitoring-s3-backup (namespace monitoring). Keys:

• restic-password — 32-byte random, generated in DEV-484, sealed locally, mirrored to Passbolt entry restic / monitoring backups.
• access-key, secret-key — Hetzner S3 credentials.
• endpoint — e.g. https://hel1.your-objectstorage.com.
• bucket — Hetzner bucket name (all four restic repos share it).

Encryption is done client-side by restic. Hetzner Object Storage has no SSE-S3 / SSE-KMS (FAQ); the CTO's Hetzner Object Storage encryption memory covers the constraints.

Credentials projection

Any restore pod must project only monitoring-s3-backup — never mount cluster admin creds into the drill namespace. The pattern below copies the secret cross-namespace once and lets restic project it read-only.

kubectl create namespace restore-drill
kubectl -n monitoring get secret monitoring-s3-backup -o json |
  jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,
          .metadata.creationTimestamp,.metadata.ownerReferences,
          .metadata.annotations,.status) |
      .metadata.namespace="restore-drill"' |
  kubectl apply -n restore-drill -f -

Restore pod

Throwaway pod pinned to the API-server node (no data mount, just the S3 secret). The emptyDir holds the restored tree; deleting the pod wipes it.

apiVersion: v1
kind: Pod
metadata:
  name: restore-drill
  namespace: restore-drill
  labels: { app: restore-drill }
spec:
  restartPolicy: Never
  containers:
    - name: restic
      image: harbor.basicstack.de/library/restic:0.17.3  # matches CronJob image (DEV-493)
      command: ["sleep", "3600"]
      env:
        - { name: AWS_ACCESS_KEY_ID,     valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: access-key } } }
        - { name: AWS_SECRET_ACCESS_KEY, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: secret-key } } }
        - { name: RESTIC_PASSWORD,       valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: restic-password } } }
        - { name: S3_ENDPOINT,           valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: endpoint } } }
        - { name: S3_BUCKET,             valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: bucket } } }
      volumeMounts:
        - { name: work, mountPath: /work }
  volumes:
    - name: work
      emptyDir: {}

Apply it, then kubectl -n restore-drill exec restore-drill -- <cmd> for everything below.

Per-repo restore commands

RESTIC_REPOSITORY is set inline per repo to keep repo scope explicit. All commands run inside the scratch pod.

Loki

export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki"
restic snapshots
restic restore latest --target /work/loki --tag loki
find /work/loki/source -type f

Restored tree lands under /work/loki/source/… because the backup source was /source inside the CronJob container.

K8s resources

export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources"
restic snapshots
restic restore latest --target /work/k8s --tag k8s-resources
ls -lh /work/k8s/cluster.yaml

cluster.yaml is the concatenated dump written by the backup-k8s-resources CronJob's kubectl-dump init container.

Grafana

export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana"
restic snapshots
restic restore latest --target /work/grafana --tag grafana

Follows the loki pattern (restored tree under /work/grafana/source/).

Prometheus

export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus"
restic snapshots
restic restore latest --target /work/prometheus --tag prometheus
find /work/prometheus/source -maxdepth 1 -type d | head

Restored tree lands under /work/prometheus/source/…. WAL and chunks_head/ are excluded on purpose — see the "compaction race" notes at the top of apps/monitoring/prometheus-backup-cronjob.yaml. Expect the last ~15 s of ingested samples to be lost on recovery; the compacted 2h/24h blocks are complete.

Integrity check — promtool tsdb analyze (required for the DEV-492 drill). The snapshot must load cleanly through Prometheus's own verifier before we call the drill green.

kubectl -n restore-drill run promtool-check --rm -it --restart=Never \
  --image=prom/prometheus:v2.53.1 \
  --overrides='{"spec":{"containers":[{"name":"promtool-check","image":"prom/prometheus:v2.53.1","command":["sh","-c","for b in /work/prometheus/source/*/; do echo \"--- $b\"; promtool tsdb analyze /work/prometheus/source \"$(basename $b)\" || exit 1; done"],"volumeMounts":[{"name":"work","mountPath":"/work"}]}],"volumes":[{"name":"work","emptyDir":{}}]}}'

Simpler drill path (the restore pod already has restic; promtool is not shipped in restic/restic, so run it from a prom/prometheus sidecar or an interactive pod that mounts the same emptyDir — see DEV-492 drill entry for the concrete two-pod recipe used on the first run).

Exit 0 on every block = pass. A non-zero exit on any block means the compaction-race mitigation slipped and the snapshot is corrupt. Re-run the CronJob and re-drill.

Integrity checks

Two layers.

Repository self-check — cryptographic integrity of packs and snapshots, no source needed:

restic check                       # metadata only
restic check --read-data-subset=5% # sample-read + decrypt 5% of packs

The CronJobs run --read-data-subset=5% on every write. A restore drill should re-run at minimum restic check end-to-end.

Source parity — sha256 the restored tree against a fresh readout of the live PVC. For loki, exec into the live pod:

LOKI=$(kubectl -n monitoring get pods -l app=loki -o jsonpath='{.items[0].metadata.name}')
kubectl -n monitoring exec "$LOKI" -- sh -c '
  cd /loki && find . -type f ! -path "./lost+found/*" -exec sha256sum {} +' \
  > /tmp/live.sha256
kubectl -n restore-drill exec restore-drill -- sh -c '
  cd /work/loki/source && find . -type f -exec sha256sum {} +' \
  > /tmp/restored.sha256
diff <(sort /tmp/live.sha256) <(sort /tmp/restored.sha256)

Empty diff = parity. Any drift is expected only on files that Loki is actively writing (WAL segments) — capture the snapshot when Loki is quiet, or accept a WAL delta and re-hash the boltdb-shipper / chunks trees only.

For k8s-resources, parity is meaningless (a live dump varies by the second); rely on restic check and on being able to grep the restored cluster.yaml for expected namespaces / secrets.

Cleanup

kubectl -n restore-drill delete pod restore-drill --wait
kubectl delete namespace restore-drill

Deleting the namespace tears down the projected secret. The emptyDir lives in the pod's ephemeral scratch on the node; the sleep container never persists anything outside /work.

Rotation and disaster recovery

If the restic-password is lost:

1. Recover the plaintext from Passbolt (restic / monitoring backups).
2. If Passbolt is also lost, the data is unrecoverable by design — client-side encryption with a lost key cannot be reversed.

To rotate:

# on any workstation with kubectl + kubeseal + the restic password
NEW=$(openssl rand -base64 32)
restic key add                       # opens editor for new password;
                                     # paste NEW and confirm
restic key list
restic key remove <old-id>
# then re-seal the SealedSecret with NEW, redeploy, Argo syncs.

Rotate annually or immediately on suspected compromise.

Restic image (Harbor mirror)

All three restic CronJobs (backup-loki-restic, backup-grafana-restic, backup-k8s-resources) reference a Harbor-hosted copy of upstream to keep the backup pipeline off the Docker Hub pull path and immune to upstream retagging:

• Manifest reference: harbor.basicstack.de/library/restic:0.17.3
• Upstream: docker.io/restic/restic:0.17.3
• Ticket: DEV-493

The library project is public, so no imagePullSecret is required on the CronJob pods.

Tag-bump procedure

Run this when we want to move restic to a new pinned tag (e.g. 0.17.3 → 0.17.4). Do the mirror push before editing manifests so Argo cannot roll pods onto an unmirrored tag.

1. Pick and verify the upstream tag. Confirm the tag exists on Docker Hub and (ideally) read the upstream restic release notes for breaking changes:

   curl -s "https://hub.docker.com/v2/repositories/restic/restic/tags/0.17.4"
   

2. Mirror the image to Harbor with a one-shot in-cluster crane Job. Requires an admin (or library-scoped robot) Harbor credential; the Job auth secret is throwaway.

   # from the CTO workstation with kubectl + Harbor admin access
   NEW_TAG=0.17.4
   ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
     -o jsonpath='{.data.harborAdminPassword}' | base64 -d)
   AUTH=$(printf 'admin:%s' "$ADMIN_PW" | base64 -w0)
   cat >/tmp/dc.json <<JSON
   {"auths":{"harbor.basicstack.de":{"auth":"$AUTH"}}}
   JSON
   kubectl -n harbor create secret generic crane-mirror-cfg \
     --from-file=config.json=/tmp/dc.json \
     --dry-run=client -o yaml | kubectl apply -f -
   rm /tmp/dc.json
   
   cat <<YAML | kubectl apply -f -
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: crane-mirror-restic-${NEW_TAG//./-}
     namespace: harbor
   spec:
     backoffLimit: 1
     ttlSecondsAfterFinished: 600
     template:
       spec:
         restartPolicy: Never
         containers:
           - name: crane
             image: gcr.io/go-containerregistry/crane:v0.21.9
             env: [{name: DOCKER_CONFIG, value: /docker}]
             args:
               - copy
               - docker.io/restic/restic:${NEW_TAG}
               - harbor.basicstack.de/library/restic:${NEW_TAG}
             volumeMounts:
               - {name: docker-config, mountPath: /docker, readOnly: true}
         volumes:
           - name: docker-config
             secret: {secretName: crane-mirror-cfg}
   YAML
   
   kubectl -n harbor wait --for=condition=complete \
     job/crane-mirror-restic-${NEW_TAG//./-} --timeout=5m
   kubectl -n harbor logs job/crane-mirror-restic-${NEW_TAG//./-} | tail -20
   

3. Sanity-check the artifact — the Harbor digest must match the digest crane just pushed, and an anonymous pull must resolve:

   ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
     -o jsonpath='{.data.harborAdminPassword}' | base64 -d)
   curl -sk -u "admin:$ADMIN_PW" \
     "https://harbor.basicstack.de/api/v2.0/projects/library/repositories/restic/artifacts" \
     | jq -r '.[] | "digest=\(.digest) tags=\((.tags//[])|map(.name)|join(","))"'
   

4. Clean up the mirror secret and Job (the Job also TTLs itself in 10 min):

   kubectl -n harbor delete secret crane-mirror-cfg
   kubectl -n harbor delete job crane-mirror-restic-${NEW_TAG//./-} \
     --ignore-not-found
   

5. Bump the manifest. Update the three files under apps/monitoring/ in the basicstack-repo:

   • backup-loki-restic-cronjob.yaml
   • backup-grafana-restic-cronjob.yaml
   • backup-k8s-resources-cronjob.yaml

   Each references harbor.basicstack.de/library/restic:<tag> under the restic container. Bump <tag> in all three, commit, push.

6. Let Argo sync — the monitoring Application picks up the new manifests. Watch for successful reconciliation:

   kubectl -n argocd get application monitoring \
     -o jsonpath='{.status.sync.status}{"  "}{.status.health.status}{"\n"}'
   

7. Trigger one CronJob run to prove the Harbor pull is green. Create a manual Job from any of the CronJobs (loki is fine) and inspect its logs:

   kubectl -n monitoring create job --from=cronjob/backup-loki-restic \
     smoke-restic-$(date +%Y%m%d-%H%M%S)
   

   The Pod should pull from harbor.basicstack.de/library/restic:<tag> (kubectl describe pod ... → Events → Pulling image …) and finish with restic backup /source output ending in a snapshot id.

8. Update this doc's "restore pod" template if the new tag is incompatible with the existing restore pod command — the drill pod image must match the CronJob image, otherwise the restore-drill won't round-trip.

Rollback: if the new tag misbehaves, edit the three manifests back to the last-known-good tag, commit, and let Argo re-sync. The old tag's manifest remains in Harbor until it's explicitly deleted, so rollback is a manifest change only.

Drill log

Every drill appends to this section. Include:

• date (UTC), operator, snapshot IDs restored, parity result, restic check result, cleanup confirmation.

2026-08-16 — First drill (DEV-488)

• Operator: CTO agent, run DEV-488.

• Cluster: k3s at 178.105.17.239 (Hetzner).

• Snapshots:

  • restic/loki: 97092888 (2026-08-16 15:44:21Z), parent 98c6fee6 (2026-08-16 15:26:10Z). Both tag=loki, host=k3s, paths=/source.
  • restic/k8s-resources: 9b0155cf (2026-08-16 15:40:28Z), tag=k8s-resources, paths=/cluster.yaml.

• Restore output:

  • loki: Restored 14 files/dirs (292 B) in 0:00 — matches live PVC which is nearly empty (fresh Loki, no ingested chunks yet).
  • k8s-resources: Restored 1 files/dirs (11.167 MiB) in 0:00 — cluster.yaml = 11 709 295 bytes, sha256 5aa93f7326b5cbd2408cb763395bba259de6cc0f99b0c8e1bca4421edc7931c0.

• Parity — loki restored tree vs. live loki-storage-encrypted PVC:

  file	restored sha256	live sha256	match
  boltdb-shipper-active/uploader/name	51215284bd61bf79d48ccd1aec59445bfa28dec7b2cdce4d9476bc0b61342813	identical	yes
  chunks/loki_cluster_seed.json	8d214a355d0057586e1c142d5f5d2658ccec39bd66e193b37dc12c21d6d7edd5	identical	yes
  wal/checkpoint.019215/00000000	e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855	identical	yes
  wal/00019216	e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855	identical	yes

• k8s-resources sanity: cluster.yaml opens with a valid v1 Namespace list including argocd, basicstack-web, monitoring etc. No parse errors.

• restic check --read-data-subset=5% was run by the writing CronJob on both repos; drill relied on that. Re-running end-to-end restic check is a follow-up for the quarterly drill.

• Cleanup: pod + restore-drill namespace deleted; scratch tree discarded with the emptyDir.

Outcome: PASS for restic/loki and restic/k8s-resources.

Next drill target: 2026-11-16 (quarterly). Include restic/grafana once its first CronJob run has produced ≥ 1 snapshot.

2026-08-16 — Prometheus migration drill (DEV-492)

Migrated prometheus-backup from rclone sync (plaintext at rest) to restic (client-side encrypted). Same-day full-cycle drill against the freshly-populated restic/prometheus repo.

• Operator: CTO agent, run DEV-492.
• Repo: s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus, initialised by first CronJob run (repo id f0eda506fd).
• Backup runs (manual, from the deployed CronJob):
  • 71420465 (2026-08-16 16:16:35Z) — first run, cold repo, restic backup walk 5:10, 85 files 42 dirs, 8.017 GiB added (2.441 GiB stored after compression + dedup), exit 0. restic check --read-data-subset=5%: no errors.
  • 825f983a (2026-08-16 16:23:56Z) — second run, warm repo, 17 s total (no new blocks — dedup 100 %), restic check clean.
• Drill: harbor.basicstack.de/library/restic:0.17.3 (restore) + prom/prometheus:v2.53.1 (promtool) sidecars sharing an emptyDir in namespace restore-drill-dev492.
  • restic restore latest --tag prometheus --target /work: Restored 127 files/dirs (8.069 GiB) in 0:34.
  • Excluded paths verified absent on restored tree: wal/, chunks_head/, lost+found/ restored as empty dirs; lock and queries.active not present at all.
  • promtool tsdb list /work/source — 19 blocks parsed, all with valid ULID + duration + samples/chunks/series counts.
  • promtool tsdb analyze /work/source <block> — ran against every block dir; 0 failed. Summary lines validated non-zero Total Series and reasonable Duration (2h / 18h / 54h buckets, matching Prometheus's compaction schedule).
  • End-to-end restic check: no errors were found (metadata pass; on the freshly-initialised repo --read-data-subset=5% was already exercised by the writing CronJob).
• Cleanup: restore-drill-prom pod + restore-drill-dev492 namespace deleted; scratch tree discarded with the emptyDir. Manual prom-restic-manual-{01,02} jobs deleted from monitoring.

Outcome: PASS for restic/prometheus. All Definition-of-Done items on DEV-492 satisfied — backup encrypted at rest, promtool tsdb analyze clean, docs updated, restic password handling documented (shared monitoring-s3-backup SealedSecret, rotation procedure covers all four repos).

Next drill target for restic/prometheus: 2026-11-16 alongside the other three repos.