feat(monitoring): backup-k8s-resources → restic --stdin to Hetzner S3 (DEV-487)
Rewrites the backup-k8s-resources CronJob to drop the local-path tarball
path and stream a YAML dump through `restic backup --stdin` into
`hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern:
init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3`
reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops
the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling
loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and
textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` /
`_check_status`).
Deployed in parallel with the legacy `backup-volumes` CronJob — the
`backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6
(restore drill).
Server-side dry-run validated on the k3s control plane.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
d495a5936f
commit
e434fa751b
2 changed files with 160 additions and 52 deletions
|
|
@ -2,12 +2,12 @@
|
|||
|
||||
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
|
||||
|
||||
- `backup-k8s-resources-cronjob.yaml` — daily dump of Kubernetes resources into `backup-storage` PVC.
|
||||
- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.17.3` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
|
||||
- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data is NOT included here — it lives on a different node (see below). Being retired by the restic pipeline in [DEV-482](/DEV/issues/DEV-482) — keep running until step 6 (restore drill passed).
|
||||
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
|
||||
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed in parallel with `backup-volumes` per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
|
||||
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — dedicated Prometheus data backup that streams `prometheus-data-encrypted` to Hetzner S3 via rclone. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node ([DEV-465](/DEV/issues/DEV-465)).
|
||||
|
||||
The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for `backup-k8s-resources` and `backup-volumes`.
|
||||
The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the destination for `backup-volumes` only. `backup-k8s-resources` no longer writes there — the tarball path was retired in [DEV-487](/DEV/issues/DEV-487). The PVC is scheduled for retirement together with `backup-volumes` once [DEV-482](/DEV/issues/DEV-482) step 6 (restore drill) has passed.
|
||||
|
||||
`backup-k8s-resources` and `backup-volumes` pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used there. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`.
|
||||
`backup-volumes` pins itself to `k3s-worker-2` via `nodeSelector` because that is the node that holds its destination PVC. `backup-k8s-resources` is no longer node-pinned. `prometheus-backup` follows the Prometheus pod via `podAffinity`, writing to Hetzner S3 (`hetzner-s3:basicstack-backup/prometheus/`) so it stays independent of `backup-storage`.
|
||||
|
|
|
|||
|
|
@ -1,3 +1,23 @@
|
|||
---
|
||||
# Kubernetes-resource backup via restic to Hetzner Object Storage
|
||||
# (DEV-487, DEV-482 Option 4). Step 4 of the Option 4 rollout.
|
||||
#
|
||||
# Streams a concatenated YAML dump of cluster-scoped and per-namespace
|
||||
# resources through `restic backup --stdin` into
|
||||
# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources`. No PVC mount
|
||||
# (drops the local-path `backup-storage` dependency), no node pin.
|
||||
#
|
||||
# Two-container pattern:
|
||||
# 1. `kubectl-dump` init container (`alpine/k8s:1.29.4`) writes
|
||||
# /dump/cluster.yaml into an emptyDir. Uses `serviceAccountName:
|
||||
# backup-sa` (unchanged from the legacy job).
|
||||
# 2. `restic` main container (`restic/restic:0.17.3`, matches the
|
||||
# loki/grafana siblings) reads that file on stdin and streams it
|
||||
# into the restic repo with `--stdin-filename cluster.yaml`.
|
||||
#
|
||||
# Deployed in parallel with the legacy `backup-volumes` CronJob — do
|
||||
# not retire that job or the `backup-storage` PVC until DEV-482 step 6
|
||||
# (restore drill) has passed.
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
|
|
@ -6,6 +26,7 @@ metadata:
|
|||
labels:
|
||||
app: backup
|
||||
type: k8s-resources
|
||||
backend: restic
|
||||
spec:
|
||||
schedule: "0 2 * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
|
|
@ -13,72 +34,159 @@ spec:
|
|||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: backup
|
||||
type: k8s-resources
|
||||
backend: restic
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 3600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: backup
|
||||
type: k8s-resources
|
||||
backend: restic
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
serviceAccountName: backup-sa
|
||||
# backup-storage PVC (local-path) is bound to k3s-worker-2, so pin here.
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: k3s-worker-2
|
||||
containers:
|
||||
- name: kubectl-backup
|
||||
initContainers:
|
||||
- name: kubectl-dump
|
||||
image: alpine/k8s:1.29.4
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
set -e
|
||||
BACKUP_DIR="/backup/k8s-$(date +%Y%m%d-%H%M%S)"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
set -eu
|
||||
echo "=== kubectl-dump started at $(date -u +%FT%TZ) ==="
|
||||
DUMP=/dump/cluster.yaml
|
||||
: > "${DUMP}"
|
||||
|
||||
echo "Starting Kubernetes resources backup to $BACKUP_DIR"
|
||||
echo "--- namespaces ---"
|
||||
kubectl get namespaces -o yaml >> "${DUMP}"
|
||||
echo "---" >> "${DUMP}"
|
||||
|
||||
kubectl get namespaces -o yaml > "$BACKUP_DIR/namespaces.yaml"
|
||||
echo "--- cluster-scoped resources ---"
|
||||
kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \
|
||||
-o yaml >> "${DUMP}"
|
||||
echo "---" >> "${DUMP}"
|
||||
|
||||
echo "--- namespaced resources ---"
|
||||
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
|
||||
mkdir -p "$BACKUP_DIR/$ns"
|
||||
kubectl get configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \
|
||||
-n "$ns" -o yaml > "$BACKUP_DIR/$ns/resources.yaml" 2>/dev/null || true
|
||||
echo " ns=${ns}"
|
||||
kubectl get \
|
||||
configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \
|
||||
-n "${ns}" -o yaml >> "${DUMP}" 2>/dev/null || true
|
||||
echo "---" >> "${DUMP}"
|
||||
done
|
||||
|
||||
kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \
|
||||
-o yaml > "$BACKUP_DIR/cluster-resources.yaml"
|
||||
|
||||
cd /backup
|
||||
tar -czf "k8s-backup-$(date +%Y%m%d-%H%M%S).tar.gz" "$(basename $BACKUP_DIR)"
|
||||
rm -rf "$BACKUP_DIR"
|
||||
|
||||
find /backup -name "k8s-backup-*.tar.gz" -mtime +7 -delete
|
||||
|
||||
echo "Backup completed successfully"
|
||||
|
||||
echo "backup_k8s_resources_success 1" > /metrics/backup_success.prom
|
||||
echo "backup_k8s_resources_timestamp $(date +%s)" >> /metrics/backup_success.prom
|
||||
echo "dump size: $(wc -c < ${DUMP}) bytes"
|
||||
echo "=== kubectl-dump finished at $(date -u +%FT%TZ) ==="
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 384Mi
|
||||
memory: 512Mi
|
||||
volumeMounts:
|
||||
- mountPath: /backup
|
||||
name: backup-storage
|
||||
- mountPath: /metrics
|
||||
name: metrics
|
||||
- name: dump
|
||||
mountPath: /dump
|
||||
containers:
|
||||
- name: restic
|
||||
image: restic/restic:0.17.3
|
||||
env:
|
||||
- name: AWS_ACCESS_KEY_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: access-key
|
||||
- name: AWS_SECRET_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: secret-key
|
||||
- name: RESTIC_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: restic-password
|
||||
- name: S3_ENDPOINT
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: endpoint
|
||||
- name: S3_BUCKET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: monitoring-s3-backup
|
||||
key: bucket
|
||||
- name: RESTIC_REPOSITORY
|
||||
value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/k8s-resources"
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
echo "=== backup-k8s-resources-restic started at $(date -u +%FT%TZ) ==="
|
||||
echo "Repository: ${RESTIC_REPOSITORY}"
|
||||
|
||||
# First-run tolerance: init if the repo isn't there yet.
|
||||
if restic snapshots >/dev/null 2>&1; then
|
||||
echo "Repo exists, skipping init."
|
||||
else
|
||||
echo "Repo missing, initialising..."
|
||||
restic init
|
||||
fi
|
||||
|
||||
echo "--- restic backup --stdin cluster.yaml ---"
|
||||
restic backup --stdin \
|
||||
--stdin-filename cluster.yaml \
|
||||
--tag k8s-resources \
|
||||
--host k3s < /dump/cluster.yaml
|
||||
|
||||
echo "--- restic forget/prune ---"
|
||||
restic forget --tag k8s-resources \
|
||||
--keep-daily 7 \
|
||||
--keep-weekly 4 \
|
||||
--keep-monthly 6 \
|
||||
--prune
|
||||
|
||||
echo "--- restic check --read-data-subset=5% ---"
|
||||
CHECK_STATUS=0
|
||||
restic check --read-data-subset=5% || CHECK_STATUS=$?
|
||||
echo "restic check exit: ${CHECK_STATUS}"
|
||||
|
||||
# Textfile-collector metrics; identical wiring to the
|
||||
# loki/grafana siblings. Scrapeable once node-exporter's
|
||||
# textfile collector path is enabled — tracked in DEV-482.
|
||||
{
|
||||
echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
|
||||
echo "backup_k8s_resources_timestamp_seconds $(date +%s)"
|
||||
echo "backup_k8s_resources_check_status ${CHECK_STATUS}"
|
||||
} > /metrics/backup_k8s_resources.prom
|
||||
|
||||
echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ==="
|
||||
exit ${CHECK_STATUS}
|
||||
volumeMounts:
|
||||
- name: dump
|
||||
mountPath: /dump
|
||||
readOnly: true
|
||||
- name: metrics
|
||||
mountPath: /metrics
|
||||
- name: cache
|
||||
mountPath: /root/.cache/restic
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 1500m
|
||||
memory: 1Gi
|
||||
volumes:
|
||||
- name: backup-storage
|
||||
persistentVolumeClaim:
|
||||
claimName: backup-storage
|
||||
- name: dump
|
||||
emptyDir: {}
|
||||
- name: metrics
|
||||
emptyDir: {}
|
||||
- name: cache
|
||||
emptyDir: {}
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue