e69e3d8eb4
Mirrored docker.io/restic/restic:0.19.1 to harbor.basicstack.de/library/restic:0.19.1 (crane in-cluster Job). Updated all CronJob pins and the restore-drill/tag-bump docs. Digest: sha256:136600b6ff6843d61d355f7f71f460a166429f35de6fd11b568fece3c9a4d510 Co-Authored-By: Paperclip <noreply@paperclip.ing>
464 lines
19 KiB
Markdown
464 lines
19 KiB
Markdown
# Restic Restore Runbook — Hetzner Object Storage
|
|
|
|
Author: CTO agent (`4b5f09a2-22d8-4e3d-9ac4-46d008ad1385`)
|
|
Ticket: [DEV-488](/DEV/issues/DEV-488)
|
|
Parent: [DEV-482](/DEV/issues/DEV-482) (Option 4 — restic → Hetzner OS)
|
|
|
|
The monitoring stack's data-durability layer is a client-side-encrypted
|
|
restic repo set stored in a single Hetzner Object Storage bucket. This
|
|
document is the operator playbook for **restoring** those snapshots
|
|
into a scratch namespace when we need to (a) recover from a real loss
|
|
event, or (b) run the quarterly restore drill that keeps the
|
|
[`backup-volumes` retirement plan](/DEV/issues/DEV-489) honest.
|
|
|
|
## Repo layout
|
|
|
|
Single bucket, four repo prefixes:
|
|
|
|
| repo prefix | source | writer CronJob | tag |
|
|
| ---------------------------------------------- | --------------------------------------------- | ---------------------- | --------------- |
|
|
| `restic/loki` | PVC `loki-storage-encrypted` | `backup-loki-restic` | `loki` |
|
|
| `restic/grafana` | PVC `grafana-storage` (worker-2) | `backup-grafana-restic`| `grafana` |
|
|
| `restic/k8s-resources` | `kubectl get -o yaml` (stdin) | `backup-k8s-resources` | `k8s-resources` |
|
|
| `restic/prometheus` | PVC `prometheus-data-encrypted` (excl. WAL) | `prometheus-backup` | `prometheus` |
|
|
|
|
Password + endpoint + bucket + AWS creds live in `SealedSecret`
|
|
`monitoring-s3-backup` (namespace `monitoring`). Keys:
|
|
|
|
- `restic-password` — 32-byte random, generated in DEV-484, sealed
|
|
locally, mirrored to Passbolt entry `restic / monitoring backups`.
|
|
- `access-key`, `secret-key` — Hetzner S3 credentials.
|
|
- `endpoint` — e.g. `https://hel1.your-objectstorage.com`.
|
|
- `bucket` — Hetzner bucket name (all four restic repos share it).
|
|
|
|
Encryption is done **client-side by restic**. Hetzner Object Storage
|
|
has no SSE-S3 / SSE-KMS ([FAQ](https://docs.hetzner.com/storage/object-storage/faq/general/));
|
|
the CTO's [Hetzner Object Storage encryption memory](../../memory/hetzner-object-storage-encryption.md)
|
|
covers the constraints.
|
|
|
|
## Credentials projection
|
|
|
|
Any restore pod must project **only** `monitoring-s3-backup` — never
|
|
mount cluster admin creds into the drill namespace. The pattern below
|
|
copies the secret cross-namespace once and lets restic project it
|
|
read-only.
|
|
|
|
```bash
|
|
kubectl create namespace restore-drill
|
|
kubectl -n monitoring get secret monitoring-s3-backup -o json |
|
|
jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,
|
|
.metadata.creationTimestamp,.metadata.ownerReferences,
|
|
.metadata.annotations,.status) |
|
|
.metadata.namespace="restore-drill"' |
|
|
kubectl apply -n restore-drill -f -
|
|
```
|
|
|
|
## Restore pod
|
|
|
|
Throwaway pod pinned to the API-server node (no data mount, just the
|
|
S3 secret). The `emptyDir` holds the restored tree; deleting the pod
|
|
wipes it.
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: restore-drill
|
|
namespace: restore-drill
|
|
labels: { app: restore-drill }
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: restic
|
|
image: harbor.basicstack.de/library/restic:0.19.1 # matches CronJob image (DEV-493, bumped in DEV-541)
|
|
command: ["sleep", "3600"]
|
|
env:
|
|
- { name: AWS_ACCESS_KEY_ID, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: access-key } } }
|
|
- { name: AWS_SECRET_ACCESS_KEY, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: secret-key } } }
|
|
- { name: RESTIC_PASSWORD, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: restic-password } } }
|
|
- { name: S3_ENDPOINT, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: endpoint } } }
|
|
- { name: S3_BUCKET, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: bucket } } }
|
|
volumeMounts:
|
|
- { name: work, mountPath: /work }
|
|
volumes:
|
|
- name: work
|
|
emptyDir: {}
|
|
```
|
|
|
|
Apply it, then `kubectl -n restore-drill exec restore-drill -- <cmd>`
|
|
for everything below.
|
|
|
|
## Per-repo restore commands
|
|
|
|
`RESTIC_REPOSITORY` is set inline per repo to keep repo scope explicit.
|
|
All commands run inside the scratch pod.
|
|
|
|
### Loki
|
|
|
|
```sh
|
|
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki"
|
|
restic snapshots
|
|
restic restore latest --target /work/loki --tag loki
|
|
find /work/loki/source -type f
|
|
```
|
|
|
|
Restored tree lands under `/work/loki/source/…` because the backup
|
|
source was `/source` inside the CronJob container.
|
|
|
|
### K8s resources
|
|
|
|
```sh
|
|
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources"
|
|
restic snapshots
|
|
restic restore latest --target /work/k8s --tag k8s-resources
|
|
ls -lh /work/k8s/cluster.yaml
|
|
```
|
|
|
|
`cluster.yaml` is the concatenated dump written by the
|
|
`backup-k8s-resources` CronJob's `kubectl-dump` init container.
|
|
|
|
### Grafana
|
|
|
|
```sh
|
|
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana"
|
|
restic snapshots
|
|
restic restore latest --target /work/grafana --tag grafana
|
|
```
|
|
|
|
Follows the loki pattern (restored tree under `/work/grafana/source/`).
|
|
|
|
### Prometheus
|
|
|
|
```sh
|
|
export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus"
|
|
restic snapshots
|
|
restic restore latest --target /work/prometheus --tag prometheus
|
|
find /work/prometheus/source -maxdepth 1 -type d | head
|
|
```
|
|
|
|
Restored tree lands under `/work/prometheus/source/…`. **WAL and
|
|
`chunks_head/` are excluded on purpose** — see the "compaction race"
|
|
notes at the top of `apps/monitoring/prometheus-backup-cronjob.yaml`.
|
|
Expect the last ~15 s of ingested samples to be lost on recovery; the
|
|
compacted 2h/24h blocks are complete.
|
|
|
|
**Integrity check — `promtool tsdb analyze` (required for the DEV-492
|
|
drill).** The snapshot must load cleanly through Prometheus's own
|
|
verifier before we call the drill green.
|
|
|
|
```sh
|
|
kubectl -n restore-drill run promtool-check --rm -it --restart=Never \
|
|
--image=prom/prometheus:v2.53.1 \
|
|
--overrides='{"spec":{"containers":[{"name":"promtool-check","image":"prom/prometheus:v2.53.1","command":["sh","-c","for b in /work/prometheus/source/*/; do echo \"--- $b\"; promtool tsdb analyze /work/prometheus/source \"$(basename $b)\" || exit 1; done"],"volumeMounts":[{"name":"work","mountPath":"/work"}]}],"volumes":[{"name":"work","emptyDir":{}}]}}'
|
|
```
|
|
|
|
Simpler drill path (the restore pod already has restic; `promtool` is
|
|
not shipped in `restic/restic`, so run it from a `prom/prometheus`
|
|
sidecar or an interactive pod that mounts the same `emptyDir` — see
|
|
DEV-492 drill entry for the concrete two-pod recipe used on the
|
|
first run).
|
|
|
|
Exit 0 on every block = pass. A non-zero exit on any block means the
|
|
compaction-race mitigation slipped and the snapshot is corrupt.
|
|
Re-run the CronJob and re-drill.
|
|
|
|
## Integrity checks
|
|
|
|
Two layers.
|
|
|
|
**Repository self-check** — cryptographic integrity of packs and
|
|
snapshots, no source needed:
|
|
|
|
```sh
|
|
restic check # metadata only
|
|
restic check --read-data-subset=5% # sample-read + decrypt 5% of packs
|
|
```
|
|
|
|
The CronJobs run `--read-data-subset=5%` on every write. A restore
|
|
drill should re-run at minimum `restic check` end-to-end.
|
|
|
|
**Source parity** — sha256 the restored tree against a fresh readout
|
|
of the live PVC. For loki, exec into the live pod:
|
|
|
|
```sh
|
|
LOKI=$(kubectl -n monitoring get pods -l app=loki -o jsonpath='{.items[0].metadata.name}')
|
|
kubectl -n monitoring exec "$LOKI" -- sh -c '
|
|
cd /loki && find . -type f ! -path "./lost+found/*" -exec sha256sum {} +' \
|
|
> /tmp/live.sha256
|
|
kubectl -n restore-drill exec restore-drill -- sh -c '
|
|
cd /work/loki/source && find . -type f -exec sha256sum {} +' \
|
|
> /tmp/restored.sha256
|
|
diff <(sort /tmp/live.sha256) <(sort /tmp/restored.sha256)
|
|
```
|
|
|
|
Empty diff = parity. Any drift is expected only on files that Loki is
|
|
actively writing (WAL segments) — capture the snapshot when Loki is
|
|
quiet, or accept a WAL delta and re-hash the boltdb-shipper /
|
|
chunks trees only.
|
|
|
|
For k8s-resources, parity is meaningless (a live dump varies by the
|
|
second); rely on `restic check` and on being able to `grep` the
|
|
restored `cluster.yaml` for expected namespaces / secrets.
|
|
|
|
## Cleanup
|
|
|
|
```sh
|
|
kubectl -n restore-drill delete pod restore-drill --wait
|
|
kubectl delete namespace restore-drill
|
|
```
|
|
|
|
Deleting the namespace tears down the projected secret. The `emptyDir`
|
|
lives in the pod's ephemeral scratch on the node; the sleep container
|
|
never persists anything outside `/work`.
|
|
|
|
## Rotation and disaster recovery
|
|
|
|
If the `restic-password` is lost:
|
|
|
|
1. Recover the plaintext from Passbolt (`restic / monitoring backups`).
|
|
2. If Passbolt is also lost, the data is unrecoverable by design —
|
|
client-side encryption with a lost key cannot be reversed.
|
|
|
|
To rotate:
|
|
|
|
```sh
|
|
# on any workstation with kubectl + kubeseal + the restic password
|
|
NEW=$(openssl rand -base64 32)
|
|
restic key add # opens editor for new password;
|
|
# paste NEW and confirm
|
|
restic key list
|
|
restic key remove <old-id>
|
|
# then re-seal the SealedSecret with NEW, redeploy, Argo syncs.
|
|
```
|
|
|
|
Rotate annually or immediately on suspected compromise.
|
|
|
|
## Restic image (Harbor mirror)
|
|
|
|
All three restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
|
|
`backup-k8s-resources`) reference a Harbor-hosted copy of upstream to
|
|
keep the backup pipeline off the Docker Hub pull path and immune to
|
|
upstream retagging:
|
|
|
|
- Manifest reference: `harbor.basicstack.de/library/restic:0.19.1`
|
|
- Upstream: `docker.io/restic/restic:0.19.1`
|
|
- Ticket: [DEV-493](/DEV/issues/DEV-493) (bumped to 0.19.1 in [DEV-541](/DEV/issues/DEV-541))
|
|
|
|
The `library` project is public, so no `imagePullSecret` is required
|
|
on the CronJob pods.
|
|
|
|
### Tag-bump procedure
|
|
|
|
Run this when we want to move restic to a new pinned tag (e.g. 0.17.3
|
|
→ 0.17.4). Do the mirror push **before** editing manifests so Argo
|
|
cannot roll pods onto an unmirrored tag.
|
|
|
|
1. **Pick and verify the upstream tag.** Confirm the tag exists on
|
|
Docker Hub and (ideally) read the upstream restic release notes for
|
|
breaking changes:
|
|
|
|
```sh
|
|
curl -s "https://hub.docker.com/v2/repositories/restic/restic/tags/0.17.4"
|
|
```
|
|
|
|
2. **Mirror the image to Harbor** with a one-shot in-cluster crane
|
|
Job. Requires an admin (or `library`-scoped robot) Harbor
|
|
credential; the Job auth secret is throwaway.
|
|
|
|
```sh
|
|
# from the CTO workstation with kubectl + Harbor admin access
|
|
NEW_TAG=0.17.4
|
|
ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
|
|
-o jsonpath='{.data.harborAdminPassword}' | base64 -d)
|
|
AUTH=$(printf 'admin:%s' "$ADMIN_PW" | base64 -w0)
|
|
cat >/tmp/dc.json <<JSON
|
|
{"auths":{"harbor.basicstack.de":{"auth":"$AUTH"}}}
|
|
JSON
|
|
kubectl -n harbor create secret generic crane-mirror-cfg \
|
|
--from-file=config.json=/tmp/dc.json \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
rm /tmp/dc.json
|
|
|
|
cat <<YAML | kubectl apply -f -
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: crane-mirror-restic-${NEW_TAG//./-}
|
|
namespace: harbor
|
|
spec:
|
|
backoffLimit: 1
|
|
ttlSecondsAfterFinished: 600
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: crane
|
|
image: gcr.io/go-containerregistry/crane:v0.21.9
|
|
env: [{name: DOCKER_CONFIG, value: /docker}]
|
|
args:
|
|
- copy
|
|
- docker.io/restic/restic:${NEW_TAG}
|
|
- harbor.basicstack.de/library/restic:${NEW_TAG}
|
|
volumeMounts:
|
|
- {name: docker-config, mountPath: /docker, readOnly: true}
|
|
volumes:
|
|
- name: docker-config
|
|
secret: {secretName: crane-mirror-cfg}
|
|
YAML
|
|
|
|
kubectl -n harbor wait --for=condition=complete \
|
|
job/crane-mirror-restic-${NEW_TAG//./-} --timeout=5m
|
|
kubectl -n harbor logs job/crane-mirror-restic-${NEW_TAG//./-} | tail -20
|
|
```
|
|
|
|
3. **Sanity-check the artifact** — the Harbor digest must match the
|
|
digest crane just pushed, and an anonymous pull must resolve:
|
|
|
|
```sh
|
|
ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
|
|
-o jsonpath='{.data.harborAdminPassword}' | base64 -d)
|
|
curl -sk -u "admin:$ADMIN_PW" \
|
|
"https://harbor.basicstack.de/api/v2.0/projects/library/repositories/restic/artifacts" \
|
|
| jq -r '.[] | "digest=\(.digest) tags=\((.tags//[])|map(.name)|join(","))"'
|
|
```
|
|
|
|
4. **Clean up the mirror secret and Job** (the Job also TTLs itself
|
|
in 10 min):
|
|
|
|
```sh
|
|
kubectl -n harbor delete secret crane-mirror-cfg
|
|
kubectl -n harbor delete job crane-mirror-restic-${NEW_TAG//./-} \
|
|
--ignore-not-found
|
|
```
|
|
|
|
5. **Bump the manifest.** Update the three files under `apps/monitoring/`
|
|
in the `basicstack-repo`:
|
|
|
|
- `backup-loki-restic-cronjob.yaml`
|
|
- `backup-grafana-restic-cronjob.yaml`
|
|
- `backup-k8s-resources-cronjob.yaml`
|
|
|
|
Each references `harbor.basicstack.de/library/restic:<tag>` under
|
|
the `restic` container. Bump `<tag>` in all three, commit, push.
|
|
|
|
6. **Let Argo sync** — the `monitoring` Application picks up the new
|
|
manifests. Watch for successful reconciliation:
|
|
|
|
```sh
|
|
kubectl -n argocd get application monitoring \
|
|
-o jsonpath='{.status.sync.status}{" "}{.status.health.status}{"\n"}'
|
|
```
|
|
|
|
7. **Trigger one CronJob run to prove the Harbor pull is green.**
|
|
Create a manual Job from any of the CronJobs (loki is fine) and
|
|
inspect its logs:
|
|
|
|
```sh
|
|
kubectl -n monitoring create job --from=cronjob/backup-loki-restic \
|
|
smoke-restic-$(date +%Y%m%d-%H%M%S)
|
|
```
|
|
|
|
The Pod should pull from `harbor.basicstack.de/library/restic:<tag>`
|
|
(`kubectl describe pod ...` → Events → `Pulling image …`) and
|
|
finish with `restic backup /source` output ending in a snapshot id.
|
|
|
|
8. **Update this doc's "restore pod" template** if the new tag is
|
|
incompatible with the existing restore pod command — the drill pod
|
|
image must match the CronJob image, otherwise the restore-drill
|
|
won't round-trip.
|
|
|
|
**Rollback:** if the new tag misbehaves, edit the three manifests back
|
|
to the last-known-good tag, commit, and let Argo re-sync. The old
|
|
tag's manifest remains in Harbor until it's explicitly deleted, so
|
|
rollback is a manifest change only.
|
|
|
|
## Drill log
|
|
|
|
Every drill appends to this section. Include:
|
|
|
|
- date (UTC), operator, snapshot IDs restored, parity result,
|
|
restic check result, cleanup confirmation.
|
|
|
|
### 2026-08-16 — First drill (DEV-488)
|
|
|
|
- Operator: CTO agent, run [DEV-488](/DEV/issues/DEV-488).
|
|
- Cluster: k3s at `178.105.17.239` (Hetzner).
|
|
- Snapshots:
|
|
- `restic/loki`: `97092888` (2026-08-16 15:44:21Z), parent `98c6fee6`
|
|
(2026-08-16 15:26:10Z). Both tag=`loki`, host=`k3s`, paths=`/source`.
|
|
- `restic/k8s-resources`: `9b0155cf` (2026-08-16 15:40:28Z),
|
|
tag=`k8s-resources`, paths=`/cluster.yaml`.
|
|
- Restore output:
|
|
- loki: `Restored 14 files/dirs (292 B) in 0:00` — matches live PVC
|
|
which is nearly empty (fresh Loki, no ingested chunks yet).
|
|
- k8s-resources: `Restored 1 files/dirs (11.167 MiB) in 0:00` —
|
|
`cluster.yaml` = 11 709 295 bytes, sha256
|
|
`5aa93f7326b5cbd2408cb763395bba259de6cc0f99b0c8e1bca4421edc7931c0`.
|
|
- Parity — loki restored tree vs. live loki-storage-encrypted PVC:
|
|
|
|
| file | restored sha256 | live sha256 | match |
|
|
| ---------------------------------------- | ------------------------------------------------------------------ | ----------- | ----- |
|
|
| `boltdb-shipper-active/uploader/name` | `51215284bd61bf79d48ccd1aec59445bfa28dec7b2cdce4d9476bc0b61342813` | identical | yes |
|
|
| `chunks/loki_cluster_seed.json` | `8d214a355d0057586e1c142d5f5d2658ccec39bd66e193b37dc12c21d6d7edd5` | identical | yes |
|
|
| `wal/checkpoint.019215/00000000` | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | identical | yes |
|
|
| `wal/00019216` | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | identical | yes |
|
|
|
|
- k8s-resources sanity: `cluster.yaml` opens with a valid v1
|
|
Namespace list including `argocd`, `basicstack-web`, `monitoring`
|
|
etc. No parse errors.
|
|
- `restic check --read-data-subset=5%` was run by the writing
|
|
CronJob on both repos; drill relied on that. Re-running end-to-end
|
|
`restic check` is a follow-up for the quarterly drill.
|
|
- Cleanup: pod + `restore-drill` namespace deleted; scratch tree
|
|
discarded with the emptyDir.
|
|
|
|
**Outcome: PASS** for `restic/loki` and `restic/k8s-resources`.
|
|
|
|
Next drill target: 2026-11-16 (quarterly). Include `restic/grafana`
|
|
once its first CronJob run has produced ≥ 1 snapshot.
|
|
|
|
### 2026-08-16 — Prometheus migration drill (DEV-492)
|
|
|
|
Migrated `prometheus-backup` from `rclone sync` (plaintext at rest) to
|
|
restic (client-side encrypted). Same-day full-cycle drill against the
|
|
freshly-populated `restic/prometheus` repo.
|
|
|
|
- Operator: CTO agent, run [DEV-492](/DEV/issues/DEV-492).
|
|
- Repo: `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, initialised
|
|
by first CronJob run (repo id `f0eda506fd`).
|
|
- Backup runs (manual, from the deployed CronJob):
|
|
- `71420465` (2026-08-16 16:16:35Z) — first run, cold repo,
|
|
`restic backup` walk 5:10, 85 files 42 dirs, 8.017 GiB added
|
|
(2.441 GiB stored after compression + dedup), exit 0. `restic
|
|
check --read-data-subset=5%`: no errors.
|
|
- `825f983a` (2026-08-16 16:23:56Z) — second run, warm repo, 17 s
|
|
total (no new blocks — dedup 100 %), `restic check` clean.
|
|
- Drill: `harbor.basicstack.de/library/restic:0.17.3` (restore) +
|
|
`prom/prometheus:v2.53.1` (promtool) sidecars sharing an `emptyDir`
|
|
in namespace `restore-drill-dev492`.
|
|
- `restic restore latest --tag prometheus --target /work`:
|
|
`Restored 127 files/dirs (8.069 GiB) in 0:34`.
|
|
- Excluded paths verified absent on restored tree: `wal/`,
|
|
`chunks_head/`, `lost+found/` restored as empty dirs; `lock` and
|
|
`queries.active` not present at all.
|
|
- `promtool tsdb list /work/source` — 19 blocks parsed, all with
|
|
valid ULID + duration + samples/chunks/series counts.
|
|
- `promtool tsdb analyze /work/source <block>` — ran against every
|
|
block dir; 0 failed. Summary lines validated non-zero `Total
|
|
Series` and reasonable `Duration` (2h / 18h / 54h buckets, matching
|
|
Prometheus's compaction schedule).
|
|
- End-to-end `restic check`: `no errors were found` (metadata pass;
|
|
on the freshly-initialised repo `--read-data-subset=5%` was
|
|
already exercised by the writing CronJob).
|
|
- Cleanup: `restore-drill-prom` pod + `restore-drill-dev492` namespace
|
|
deleted; scratch tree discarded with the `emptyDir`. Manual
|
|
`prom-restic-manual-{01,02}` jobs deleted from `monitoring`.
|
|
|
|
**Outcome: PASS** for `restic/prometheus`. All Definition-of-Done
|
|
items on [DEV-492](/DEV/issues/DEV-492) satisfied — backup encrypted
|
|
at rest, `promtool tsdb analyze` clean, docs updated, restic password
|
|
handling documented (shared `monitoring-s3-backup` SealedSecret,
|
|
rotation procedure covers all four repos).
|
|
|
|
Next drill target for `restic/prometheus`: 2026-11-16 alongside the
|
|
other three repos.
|