stack.basicstack.de/docs/monitoring/restic-restore.md

# Restic Restore Runbook — Hetzner Object Storage

Author: CTO agent (`4b5f09a2-22d8-4e3d-9ac4-46d008ad1385`)
Ticket: [DEV-488](/DEV/issues/DEV-488)
Parent: [DEV-482](/DEV/issues/DEV-482) (Option 4 — restic → Hetzner OS)

The monitoring stack's data-durability layer is a client-side-encrypted
restic repo set stored in a single Hetzner Object Storage bucket. This
document is the operator playbook for **restoring** those snapshots
into a scratch namespace when we need to (a) recover from a real loss
event, or (b) run the quarterly restore drill that keeps the
[`backup-volumes` retirement plan](/DEV/issues/DEV-489) honest.

## Repo layout

Single bucket, four repo prefixes:

| repo prefix                                    | source                                        | writer CronJob         | tag             |
| ---------------------------------------------- | --------------------------------------------- | ---------------------- | --------------- |
| `restic/loki`                                  | PVC `loki-storage-encrypted`                  | `backup-loki-restic`   | `loki`          |
| `restic/grafana`                               | PVC `grafana-storage` (worker-2)              | `backup-grafana-restic`| `grafana`       |
| `restic/k8s-resources`                         | `kubectl get -o yaml` (stdin)                 | `backup-k8s-resources` | `k8s-resources` |
| `restic/prometheus`                            | PVC `prometheus-data-encrypted` (excl. WAL)   | `prometheus-backup`    | `prometheus`    |

Password + endpoint + bucket + AWS creds live in `SealedSecret`
`monitoring-s3-backup` (namespace `monitoring`). Keys:

- `restic-password` — 32-byte random, generated in DEV-484, sealed
  locally, mirrored to Passbolt entry `restic / monitoring backups`.
- `access-key`, `secret-key` — Hetzner S3 credentials.
- `endpoint` — e.g. `https://hel1.your-objectstorage.com`.
- `bucket` — Hetzner bucket name (all four restic repos share it).

Encryption is done **client-side by restic**. Hetzner Object Storage
has no SSE-S3 / SSE-KMS ([FAQ](https://docs.hetzner.com/storage/object-storage/faq/general/));
the CTO's [Hetzner Object Storage encryption memory](../../memory/hetzner-object-storage-encryption.md)
covers the constraints.

## Credentials projection

Any restore pod must project **only** `monitoring-s3-backup` — never
mount cluster admin creds into the drill namespace. The pattern below
copies the secret cross-namespace once and lets restic project it
read-only.

```bash
kubectl create namespace restore-drill
kubectl -n monitoring get secret monitoring-s3-backup -o json |
  jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,
          .metadata.creationTimestamp,.metadata.ownerReferences,
          .metadata.annotations,.status) |
      .metadata.namespace="restore-drill"' |
  kubectl apply -n restore-drill -f -
```

## Restore pod

Throwaway pod pinned to the API-server node (no data mount, just the
S3 secret). The `emptyDir` holds the restored tree; deleting the pod
wipes it.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: restore-drill
  namespace: restore-drill
  labels: { app: restore-drill }
spec:
  restartPolicy: Never
  containers:
    - name: restic
      image: harbor.basicstack.de/library/restic:0.17.3  # matches CronJob image (DEV-493)
      command: ["sleep", "3600"]
      env:
        - { name: AWS_ACCESS_KEY_ID,     valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: access-key } } }
        - { name: AWS_SECRET_ACCESS_KEY, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: secret-key } } }
        - { name: RESTIC_PASSWORD,       valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: restic-password } } }
        - { name: S3_ENDPOINT,           valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: endpoint } } }
        - { name: S3_BUCKET,             valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: bucket } } }
      volumeMounts:
        - { name: work, mountPath: /work }
  volumes:
    - name: work
      emptyDir: {}
```

Apply it, then `kubectl -n restore-drill exec restore-drill -- <cmd>`
for everything below.

## Per-repo restore commands

`RESTIC_REPOSITORY` is set inline per repo to keep repo scope explicit.
All commands run inside the scratch pod.

### Loki

```sh
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki"
restic snapshots
restic restore latest --target /work/loki --tag loki
find /work/loki/source -type f
```

Restored tree lands under `/work/loki/source/…` because the backup
source was `/source` inside the CronJob container.

### K8s resources

```sh
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources"
restic snapshots
restic restore latest --target /work/k8s --tag k8s-resources
ls -lh /work/k8s/cluster.yaml
```

`cluster.yaml` is the concatenated dump written by the
`backup-k8s-resources` CronJob's `kubectl-dump` init container.

### Grafana

```sh
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana"
restic snapshots
restic restore latest --target /work/grafana --tag grafana
```

Follows the loki pattern (restored tree under `/work/grafana/source/`).

### Prometheus

```sh
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus"
restic snapshots
restic restore latest --target /work/prometheus --tag prometheus
find /work/prometheus/source -maxdepth 1 -type d | head
```

Restored tree lands under `/work/prometheus/source/…`. **WAL and
`chunks_head/` are excluded on purpose** — see the "compaction race"
notes at the top of `apps/monitoring/prometheus-backup-cronjob.yaml`.
Expect the last ~15 s of ingested samples to be lost on recovery; the
compacted 2h/24h blocks are complete.

**Integrity check — `promtool tsdb analyze` (required for the DEV-492
drill).** The snapshot must load cleanly through Prometheus's own
verifier before we call the drill green.

```sh
kubectl -n restore-drill run promtool-check --rm -it --restart=Never \
  --image=prom/prometheus:v2.53.1 \
  --overrides='{"spec":{"containers":[{"name":"promtool-check","image":"prom/prometheus:v2.53.1","command":["sh","-c","for b in /work/prometheus/source/*/; do echo \"--- $b\"; promtool tsdb analyze /work/prometheus/source \"$(basename $b)\" || exit 1; done"],"volumeMounts":[{"name":"work","mountPath":"/work"}]}],"volumes":[{"name":"work","emptyDir":{}}]}}'
```

Simpler drill path (the restore pod already has restic; `promtool` is
not shipped in `restic/restic`, so run it from a `prom/prometheus`
sidecar or an interactive pod that mounts the same `emptyDir` — see
DEV-492 drill entry for the concrete two-pod recipe used on the
first run).

Exit 0 on every block = pass. A non-zero exit on any block means the
compaction-race mitigation slipped and the snapshot is corrupt.
Re-run the CronJob and re-drill.

## Integrity checks

Two layers.

**Repository self-check** — cryptographic integrity of packs and
snapshots, no source needed:

```sh
restic check                       # metadata only
restic check --read-data-subset=5% # sample-read + decrypt 5% of packs
```

The CronJobs run `--read-data-subset=5%` on every write. A restore
drill should re-run at minimum `restic check` end-to-end.

**Source parity** — sha256 the restored tree against a fresh readout
of the live PVC. For loki, exec into the live pod:

```sh
LOKI=$(kubectl -n monitoring get pods -l app=loki -o jsonpath='{.items[0].metadata.name}')
kubectl -n monitoring exec "$LOKI" -- sh -c '
  cd /loki && find . -type f ! -path "./lost+found/*" -exec sha256sum {} +' \
  > /tmp/live.sha256
kubectl -n restore-drill exec restore-drill -- sh -c '
  cd /work/loki/source && find . -type f -exec sha256sum {} +' \
  > /tmp/restored.sha256
diff <(sort /tmp/live.sha256) <(sort /tmp/restored.sha256)
```

Empty diff = parity. Any drift is expected only on files that Loki is
actively writing (WAL segments) — capture the snapshot when Loki is
quiet, or accept a WAL delta and re-hash the boltdb-shipper /
chunks trees only.

For k8s-resources, parity is meaningless (a live dump varies by the
second); rely on `restic check` and on being able to `grep` the
restored `cluster.yaml` for expected namespaces / secrets.

## Cleanup

```sh
kubectl -n restore-drill delete pod restore-drill --wait
kubectl delete namespace restore-drill
```

Deleting the namespace tears down the projected secret. The `emptyDir`
lives in the pod's ephemeral scratch on the node; the sleep container
never persists anything outside `/work`.

## Rotation and disaster recovery

If the `restic-password` is lost:

1. Recover the plaintext from Passbolt (`restic / monitoring backups`).
2. If Passbolt is also lost, the data is unrecoverable by design —
   client-side encryption with a lost key cannot be reversed.

To rotate:

```sh
# on any workstation with kubectl + kubeseal + the restic password
NEW=$(openssl rand -base64 32)
restic key add                       # opens editor for new password;
                                     # paste NEW and confirm
restic key list
restic key remove <old-id>
# then re-seal the SealedSecret with NEW, redeploy, Argo syncs.
```

Rotate annually or immediately on suspected compromise.

## Restic image (Harbor mirror)

All three restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
`backup-k8s-resources`) reference a Harbor-hosted copy of upstream to
keep the backup pipeline off the Docker Hub pull path and immune to
upstream retagging:

- Manifest reference: `harbor.basicstack.de/library/restic:0.17.3`
- Upstream: `docker.io/restic/restic:0.17.3`
- Ticket: [DEV-493](/DEV/issues/DEV-493)

The `library` project is public, so no `imagePullSecret` is required
on the CronJob pods.

### Tag-bump procedure

Run this when we want to move restic to a new pinned tag (e.g. 0.17.3
→ 0.17.4). Do the mirror push **before** editing manifests so Argo
cannot roll pods onto an unmirrored tag.

1. **Pick and verify the upstream tag.** Confirm the tag exists on
   Docker Hub and (ideally) read the upstream restic release notes for
   breaking changes:

   ```sh
   curl -s "https://hub.docker.com/v2/repositories/restic/restic/tags/0.17.4"
   ```

2. **Mirror the image to Harbor** with a one-shot in-cluster crane
   Job. Requires an admin (or `library`-scoped robot) Harbor
   credential; the Job auth secret is throwaway.

   ```sh
   # from the CTO workstation with kubectl + Harbor admin access
   NEW_TAG=0.17.4
   ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
     -o jsonpath='{.data.harborAdminPassword}' | base64 -d)
   AUTH=$(printf 'admin:%s' "$ADMIN_PW" | base64 -w0)
   cat >/tmp/dc.json <<JSON
   {"auths":{"harbor.basicstack.de":{"auth":"$AUTH"}}}
   JSON
   kubectl -n harbor create secret generic crane-mirror-cfg \
     --from-file=config.json=/tmp/dc.json \
     --dry-run=client -o yaml | kubectl apply -f -
   rm /tmp/dc.json

   cat <<YAML | kubectl apply -f -
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: crane-mirror-restic-${NEW_TAG//./-}
     namespace: harbor
   spec:
     backoffLimit: 1
     ttlSecondsAfterFinished: 600
     template:
       spec:
         restartPolicy: Never
         containers:
           - name: crane
             image: gcr.io/go-containerregistry/crane:v0.21.9
             env: [{name: DOCKER_CONFIG, value: /docker}]
             args:
               - copy
               - docker.io/restic/restic:${NEW_TAG}
               - harbor.basicstack.de/library/restic:${NEW_TAG}
             volumeMounts:
               - {name: docker-config, mountPath: /docker, readOnly: true}
         volumes:
           - name: docker-config
             secret: {secretName: crane-mirror-cfg}
   YAML

   kubectl -n harbor wait --for=condition=complete \
     job/crane-mirror-restic-${NEW_TAG//./-} --timeout=5m
   kubectl -n harbor logs job/crane-mirror-restic-${NEW_TAG//./-} | tail -20
   ```

3. **Sanity-check the artifact** — the Harbor digest must match the
   digest crane just pushed, and an anonymous pull must resolve:

   ```sh
   ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
     -o jsonpath='{.data.harborAdminPassword}' | base64 -d)
   curl -sk -u "admin:$ADMIN_PW" \
     "https://harbor.basicstack.de/api/v2.0/projects/library/repositories/restic/artifacts" \
     | jq -r '.[] | "digest=\(.digest) tags=\((.tags//[])|map(.name)|join(","))"'
   ```

4. **Clean up the mirror secret and Job** (the Job also TTLs itself
   in 10 min):

   ```sh
   kubectl -n harbor delete secret crane-mirror-cfg
   kubectl -n harbor delete job crane-mirror-restic-${NEW_TAG//./-} \
     --ignore-not-found
   ```

5. **Bump the manifest.** Update the three files under `apps/monitoring/`
   in the `basicstack-repo`:

   - `backup-loki-restic-cronjob.yaml`
   - `backup-grafana-restic-cronjob.yaml`
   - `backup-k8s-resources-cronjob.yaml`

   Each references `harbor.basicstack.de/library/restic:<tag>` under
   the `restic` container. Bump `<tag>` in all three, commit, push.

6. **Let Argo sync** — the `monitoring` Application picks up the new
   manifests. Watch for successful reconciliation:

   ```sh
   kubectl -n argocd get application monitoring \
     -o jsonpath='{.status.sync.status}{"  "}{.status.health.status}{"\n"}'
   ```

7. **Trigger one CronJob run to prove the Harbor pull is green.**
   Create a manual Job from any of the CronJobs (loki is fine) and
   inspect its logs:

   ```sh
   kubectl -n monitoring create job --from=cronjob/backup-loki-restic \
     smoke-restic-$(date +%Y%m%d-%H%M%S)
   ```

   The Pod should pull from `harbor.basicstack.de/library/restic:<tag>`
   (`kubectl describe pod ...` → Events → `Pulling image …`) and
   finish with `restic backup /source` output ending in a snapshot id.

8. **Update this doc's "restore pod" template** if the new tag is
   incompatible with the existing restore pod command — the drill pod
   image must match the CronJob image, otherwise the restore-drill
   won't round-trip.

**Rollback:** if the new tag misbehaves, edit the three manifests back
to the last-known-good tag, commit, and let Argo re-sync. The old
tag's manifest remains in Harbor until it's explicitly deleted, so
rollback is a manifest change only.

## Drill log

Every drill appends to this section. Include:

- date (UTC), operator, snapshot IDs restored, parity result,
  restic check result, cleanup confirmation.

### 2026-08-16 — First drill (DEV-488)

- Operator: CTO agent, run [DEV-488](/DEV/issues/DEV-488).
- Cluster: k3s at `178.105.17.239` (Hetzner).
- Snapshots:
  - `restic/loki`: `97092888` (2026-08-16 15:44:21Z), parent `98c6fee6`
    (2026-08-16 15:26:10Z). Both tag=`loki`, host=`k3s`, paths=`/source`.
  - `restic/k8s-resources`: `9b0155cf` (2026-08-16 15:40:28Z),
    tag=`k8s-resources`, paths=`/cluster.yaml`.
- Restore output:
  - loki: `Restored 14 files/dirs (292 B) in 0:00` — matches live PVC
    which is nearly empty (fresh Loki, no ingested chunks yet).
  - k8s-resources: `Restored 1 files/dirs (11.167 MiB) in 0:00` —
    `cluster.yaml` = 11 709 295 bytes, sha256
    `5aa93f7326b5cbd2408cb763395bba259de6cc0f99b0c8e1bca4421edc7931c0`.
- Parity — loki restored tree vs. live loki-storage-encrypted PVC:

  | file                                     | restored sha256                                                    | live sha256 | match |
  | ---------------------------------------- | ------------------------------------------------------------------ | ----------- | ----- |
  | `boltdb-shipper-active/uploader/name`    | `51215284bd61bf79d48ccd1aec59445bfa28dec7b2cdce4d9476bc0b61342813` | identical   | yes   |
  | `chunks/loki_cluster_seed.json`          | `8d214a355d0057586e1c142d5f5d2658ccec39bd66e193b37dc12c21d6d7edd5` | identical   | yes   |
  | `wal/checkpoint.019215/00000000`         | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | identical   | yes   |
  | `wal/00019216`                           | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | identical   | yes   |

- k8s-resources sanity: `cluster.yaml` opens with a valid v1
  Namespace list including `argocd`, `basicstack-web`, `monitoring`
  etc. No parse errors.
- `restic check --read-data-subset=5%` was run by the writing
  CronJob on both repos; drill relied on that. Re-running end-to-end
  `restic check` is a follow-up for the quarterly drill.
- Cleanup: pod + `restore-drill` namespace deleted; scratch tree
  discarded with the emptyDir.

**Outcome: PASS** for `restic/loki` and `restic/k8s-resources`.

Next drill target: 2026-11-16 (quarterly). Include `restic/grafana`
once its first CronJob run has produced ≥ 1 snapshot.

### 2026-08-16 — Prometheus migration drill (DEV-492)

Migrated `prometheus-backup` from `rclone sync` (plaintext at rest) to
restic (client-side encrypted). Same-day full-cycle drill against the
freshly-populated `restic/prometheus` repo.

- Operator: CTO agent, run [DEV-492](/DEV/issues/DEV-492).
- Repo: `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, initialised
  by first CronJob run (repo id `f0eda506fd`).
- Backup runs (manual, from the deployed CronJob):
  - `71420465` (2026-08-16 16:16:35Z) — first run, cold repo,
    `restic backup` walk 5:10, 85 files 42 dirs, 8.017 GiB added
    (2.441 GiB stored after compression + dedup), exit 0. `restic
    check --read-data-subset=5%`: no errors.
  - `825f983a` (2026-08-16 16:23:56Z) — second run, warm repo, 17 s
    total (no new blocks — dedup 100 %), `restic check` clean.
- Drill: `harbor.basicstack.de/library/restic:0.17.3` (restore) +
  `prom/prometheus:v2.53.1` (promtool) sidecars sharing an `emptyDir`
  in namespace `restore-drill-dev492`.
  - `restic restore latest --tag prometheus --target /work`:
    `Restored 127 files/dirs (8.069 GiB) in 0:34`.
  - Excluded paths verified absent on restored tree: `wal/`,
    `chunks_head/`, `lost+found/` restored as empty dirs; `lock` and
    `queries.active` not present at all.
  - `promtool tsdb list /work/source` — 19 blocks parsed, all with
    valid ULID + duration + samples/chunks/series counts.
  - `promtool tsdb analyze /work/source <block>` — ran against every
    block dir; 0 failed. Summary lines validated non-zero `Total
    Series` and reasonable `Duration` (2h / 18h / 54h buckets, matching
    Prometheus's compaction schedule).
  - End-to-end `restic check`: `no errors were found` (metadata pass;
    on the freshly-initialised repo `--read-data-subset=5%` was
    already exercised by the writing CronJob).
- Cleanup: `restore-drill-prom` pod + `restore-drill-dev492` namespace
  deleted; scratch tree discarded with the `emptyDir`. Manual
  `prom-restic-manual-{01,02}` jobs deleted from `monitoring`.

**Outcome: PASS** for `restic/prometheus`. All Definition-of-Done
items on [DEV-492](/DEV/issues/DEV-492) satisfied — backup encrypted
at rest, `promtool tsdb analyze` clean, docs updated, restic password
handling documented (shared `monitoring-s3-backup` SealedSecret,
rotation procedure covers all four repos).

Next drill target for `restic/prometheus`: 2026-11-16 alongside the
other three repos.