chore(stalwart): bump v0.16.11 -> v0.16.18 (DEV-542)

Transport-only patch bump. No auth/OIDC/config changes.
Snapshots (config.json, DB dump, live YAML) attached to DEV-542.
Rollback: git revert; ArgoCD resyncs; PVCs retained.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

apps/app-argocd.yaml

@@ -3,16 +3,27 @@ kind: Application
 metadata:
   name: argocd
   namespace: argocd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
 spec:
   project: default
-  source:
-    repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
-    targetRevision: main
-    path: apps/argocd
   destination:
     server: https://kubernetes.default.svc
     namespace: argocd
+  sources:
+    - repoURL: https://argoproj.github.io/argo-helm
+      chart: argo-cd
+      targetRevision: 10.4.0
+      helm:
+        releaseName: argocd
+        valueFiles:
+          - $values/apps/argocd/values.yaml
+    - repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
+      targetRevision: main
+      path: apps/argocd
+      ref: values
   syncPolicy:
     syncOptions:
       - CreateNamespace=true
       - ServerSideApply=true
+      - ApplyOutOfSyncOnly=true

apps/argocd/README.md

@@ -2,19 +2,28 @@
 
 This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster.
 
+Argo CD itself is installed from the community Helm chart (`argoproj/argo-helm`, chart `argo-cd`). This directory holds the chart values file plus a small kustomize wrapper for the ingress and sealed secrets that stay in git.
+
 ## Files
 
-- `argocd-install.yaml` - Auto-generated Argo CD installation manifest (DO NOT EDIT DIRECTLY)
-- `kustomization.yaml` - Kustomize overlay that adds resource limits and other customizations
-- `argocd-ingress.yaml` - Ingress configuration for Argo CD UI
-- `argocd-oidc-secret-sealed.yaml` - Sealed secret for OIDC integration
-- `repo-*.yaml` - Sealed secrets for Git repository access
+- `values.yaml` - Helm values for the `argo-cd` chart (image tag, OIDC/RBAC config, resource limits, ingress disabled).
+- `kustomization.yaml` - Kustomize wrapper for the ingress + sealed secrets (does NOT install Argo CD itself).
+- `argocd-ingress.yaml` - Ingress configuration for the Argo CD UI (Traefik + cert-manager `letsencrypt-prod`).
+- `argocd-oidc-secret-sealed.yaml` - Sealed secret for Pocket ID OIDC integration.
+- `repo-*.yaml` - Sealed secrets for Git repository access.
 
-## Resource Limits
+## How the install is wired
 
-**IMPORTANT**: Resource limits were added after DEV-281 (resource exhaustion incident on 2026-07-12).
+The root [`../app-argocd.yaml`](../app-argocd.yaml) is an Argo CD `Application` with two sources:
 
-All Argo CD components now have memory limits to prevent OOM incidents:
+1. The public Helm chart at `https://argoproj.github.io/argo-helm`, chart `argo-cd`, `targetRevision` pinned in git.
+2. This repo (`ref: values`) providing the `values.yaml` used by source (1) AND the ingress/sealed secrets applied via `kustomize`.
+
+Once bootstrapped, Argo CD manages its own install by syncing this Application.
+
+## Resource limits
+
+Memory limits were added after DEV-281 (resource exhaustion incident on 2026-07-12) and are now driven by `values.yaml`:
 
 | Component | Memory Limit | Memory Request |
 |-----------|--------------|----------------|
@@ -25,46 +34,51 @@ All Argo CD components now have memory limits to prevent OOM incidents:
 | notifications-controller | 128Mi | 64Mi |
 | applicationset-controller | 256Mi | 128Mi |
 
-These limits are based on observed usage patterns and provide headroom while preventing unlimited memory consumption.
-
 ## Deployment
 
-### Option 1: Apply with kustomize (RECOMMENDED)
+### Steady state (managed by Argo CD)
+
+Once the cluster is bootstrapped, changes to this directory are picked up by the root `argocd` Application on the next sync. No manual `kubectl apply` is required.
+
+### First-time / disaster-recovery bootstrap
+
+Argo CD cannot install itself while it is gone. Bootstrap with helm, then hand ownership back:
 
 ```bash
-kubectl apply -k apps/argocd/
+helm repo add argo https://argoproj.github.io/argo-helm
+helm repo update
+helm install argocd argo/argo-cd \
+  --version 10.4.0 \
+  --namespace argocd --create-namespace \
+  --values apps/argocd/values.yaml \
+  --wait --timeout 10m
+
+kubectl apply -k apps/argocd/          # ingress + sealed secrets
+kubectl apply -f apps/app-argocd.yaml  # hand ownership back to GitOps
 ```
 
-This will apply the base manifests plus all patches defined in `kustomization.yaml`.
-
-### Option 2: Direct apply (not recommended)
-
-```bash
-kubectl apply -f apps/argocd/argocd-install.yaml
-kubectl apply -f apps/argocd/argocd-ingress.yaml
-# etc.
-```
-
-**Note**: This skips the resource limit patches and is NOT recommended.
-
 ## Updating Argo CD
 
-When updating to a new Argo CD version:
+Bump the chart and the image tag in a single PR:
 
-1. Download the new install manifest:
+1. Refresh the local helm repo cache and check what's available:
    ```bash
-   curl -sSL https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml > argocd-install.yaml
+   helm repo update
+   helm search repo argo/argo-cd --versions | head
    ```
 
-2. Apply with kustomize (resource limits will be automatically applied):
+2. Bump both fields together, keeping them in sync with the chart's `appVersion`:
+   - `apps/app-argocd.yaml` -> `spec.sources[0].targetRevision` (chart version, e.g. `10.4.0`)
+   - `apps/argocd/values.yaml` -> `global.image.tag` (app version, e.g. `v3.5.1`)
+
+3. Optionally render locally to sanity-check the output before opening the PR:
    ```bash
-   kubectl apply -k apps/argocd/
+   helm template argocd argo/argo-cd \
+     --version <new-chart-version> \
+     -f apps/argocd/values.yaml -n argocd | less
    ```
 
-3. Verify resource limits are in place:
-   ```bash
-   kubectl get statefulset,deployment -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'
-   ```
+4. Open the PR. After merge, Argo CD syncs itself onto the new version.
 
 ## Troubleshooting
 
@@ -74,22 +88,21 @@ When updating to a new Argo CD version:
 kubectl top pods -n argocd
 ```
 
-### Check if resource limits are applied
+### Check applied resource limits
 
 ```bash
-kubectl get deployment,statefulset -n argocd -o json | jq '.items[] | {name: .metadata.name, limits: .spec.template.spec.containers[0].resources.limits}'
+kubectl get deployment,statefulset -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'
 ```
 
-### Rollback if needed
-
-If there are issues after applying resource limits:
+### Rollback if a chart upgrade misbehaves
 
 ```bash
-# Remove limits from a specific component
-kubectl patch deployment -n argocd argocd-server --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/resources"}]'
+helm -n argocd history argocd
+helm -n argocd rollback argocd <previous-revision>
 ```
 
 ## History
 
-- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281)
-- **2026-07-11**: Initial deployment
+- **2026-08-23**: Switched to the community Helm chart, bumped to `v3.5.1` / chart `10.4.0` (DEV-519).
+- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281).
+- **2026-07-11**: Initial deployment (vendored `install.yaml`).

apps/argocd/argocd-oidc-secret-sealed.yaml

@@ -16,4 +16,6 @@ spec:
       creationTimestamp: null
       name: argocd-oidc-secret
       namespace: argocd
+      labels:
+        app.kubernetes.io/part-of: argocd
     type: Opaque

apps/argocd/kustomization.yaml

@@ -4,82 +4,7 @@ kind: Kustomization
 namespace: argocd
 
 resources:
-  - argocd-install.yaml
   - argocd-ingress.yaml
   - argocd-oidc-secret-sealed.yaml
   - repo-basicstack-org-secret-sealed.yaml
   - repo-stack-basicstack-de-secret-sealed.yaml
-
-patches:
-  # Add memory limits to prevent OOM incidents (DEV-281)
-  - target:
-      kind: StatefulSet
-      name: argocd-application-controller
-    patch: |-
-      - op: add
-        path: /spec/template/spec/containers/0/resources
-        value:
-          limits:
-            memory: 512Mi
-          requests:
-            memory: 256Mi
-
-  - target:
-      kind: Deployment
-      name: argocd-repo-server
-    patch: |-
-      - op: add
-        path: /spec/template/spec/containers/0/resources
-        value:
-          limits:
-            memory: 512Mi
-          requests:
-            memory: 256Mi
-
-  - target:
-      kind: Deployment
-      name: argocd-redis
-    patch: |-
-      - op: add
-        path: /spec/template/spec/containers/0/resources
-        value:
-          limits:
-            memory: 256Mi
-          requests:
-            memory: 128Mi
-
-  - target:
-      kind: Deployment
-      name: argocd-server
-    patch: |-
-      - op: add
-        path: /spec/template/spec/containers/0/resources
-        value:
-          limits:
-            memory: 256Mi
-          requests:
-            memory: 128Mi
-
-  - target:
-      kind: Deployment
-      name: argocd-notifications-controller
-    patch: |-
-      - op: add
-        path: /spec/template/spec/containers/0/resources
-        value:
-          limits:
-            memory: 128Mi
-          requests:
-            memory: 64Mi
-
-  - target:
-      kind: Deployment
-      name: argocd-applicationset-controller
-    patch: |-
-      - op: add
-        path: /spec/template/spec/containers/0/resources
-        value:
-          limits:
-            memory: 256Mi
-          requests:
-            memory: 128Mi

apps/argocd/values.yaml

@@ -0,0 +1,87 @@
+global:
+  image:
+    tag: v3.5.1
+
+configs:
+  cm:
+    url: https://argo.basicstack.de
+    application.instanceLabelKey: argocd.argoproj.io/instance
+    resource.exclusions: |
+      - apiGroups: [cilium.io]
+        kinds: [CiliumIdentity, CiliumEndpoint, CiliumEndpointSlice]
+      - apiGroups: [kyverno.io, reports.kyverno.io, wgpolicyk8s.io]
+        kinds: [PolicyReport, ClusterPolicyReport, EphemeralReport,
+                ClusterEphemeralReport, AdmissionReport, ClusterAdmissionReport,
+                BackgroundScanReport, ClusterBackgroundScanReport, UpdateRequest]
+    oidc.config: |
+      name: Pocket ID
+      issuer: https://auth.basicstack.de
+      clientID: $argocd-oidc-secret:oidc.pocketid.clientId
+      clientSecret: $argocd-oidc-secret:oidc.pocketid.clientSecret
+      requestedScopes: [openid, profile, email, groups]
+      requestedIDTokenClaims:
+        groups: {essential: true}
+
+  rbac:
+    policy.default: role:readonly
+    policy.csv: |
+      g, argo_admins, role:admin
+      p, role:admin, applications, *, */*, allow
+      p, role:admin, clusters, *, *, allow
+      p, role:admin, repositories, *, *, allow
+      p, role:admin, projects, *, *, allow
+      p, role:admin, accounts, *, *, allow
+      p, role:admin, gpgkeys, *, *, allow
+      p, role:admin, certificates, *, *, allow
+      p, role:admin, exec, *, *, allow
+
+  params:
+    server.insecure: "true"
+
+  ssh:
+    extraHosts: |
+      forgejo.forgejo.svc.cluster.local ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCnFlDTGVmri7geuOfZ1UvHzCRi1U5BqCq+9MwkrbGkZCmBmUt3ek95JBZzurLvR/0rE624B9tB1AvSJIEIQ1W1YXD4ydiDaHX6J0sip6lEfqhREx0y91C15zHgi9jN0UKidse1g0xlLHpwePUmz8/2BJcLHJuwbZSdUu7+uhDIYtIJ5+3trX3IABNeluRA5TkspPAC0ViEaz4saWgWQWjKL8dsb3jIR94DiWAVpQnhaCBEILkIStJrmWl0O5B10Jr3KWy18szr9UVd8edCkoEXriCR8gx69jHdmuem5WlZPGvsK5Adf0mXE8S3rdyHqWOkDRs6Wlwd9p/1nDY8c8wtE3vqhefCpt1BwpTys9PMmgUfG0at/W+NdvalRqgQz26Bso8Tf7hcfyA/B69U2pvjA0tbgdIGJ5kLeCzpz9kpCYn8wSIuNIJ86BpyfnjRYFCYJVc6Ls86i8j3fEZAjEX7bmbeDBHQnyyH+rjq+Llo8aUd2Uf7HoSA93EjeuWq/Ta+YbWEWrp9Mrd48jnypxHXsiwDzqkm+YGQbXHfasiarxji/eQ5UMMG8hCxpLp1lJLhGN2th4eCLkpwchFr5jZGWgyZCth1WzQGj7NHvDTxKRU6n4MfsEX1B6GF0D60qtM5vZynpmO902mkn6wtxo+pCkDMroj668a62zw3rSS5Bw==
+
+controller:
+  resources:
+    requests:
+      memory: 256Mi
+    limits:
+      memory: 512Mi
+
+repoServer:
+  resources:
+    requests:
+      memory: 256Mi
+    limits:
+      memory: 512Mi
+
+server:
+  resources:
+    requests:
+      memory: 128Mi
+    limits:
+      memory: 256Mi
+  ingress:
+    enabled: false
+
+redis:
+  resources:
+    requests:
+      memory: 128Mi
+    limits:
+      memory: 256Mi
+
+notifications:
+  resources:
+    requests:
+      memory: 64Mi
+    limits:
+      memory: 128Mi
+
+applicationSet:
+  resources:
+    requests:
+      memory: 128Mi
+    limits:
+      memory: 256Mi

apps/bookstack/bookstack-deployment.yaml

@@ -100,6 +100,8 @@ metadata:
   namespace: bookstack
 spec:
   replicas: 1
+  strategy:
+    type: Recreate
   selector:
     matchLabels:
       app: bookstack
@@ -200,7 +202,7 @@ spec:
             cpu: "1000m"
         livenessProbe:
           httpGet:
-            path: /
+            path: /status
             port: 80
           initialDelaySeconds: 60
           periodSeconds: 10
@@ -208,7 +210,7 @@ spec:
           failureThreshold: 6
         readinessProbe:
           httpGet:
-            path: /
+            path: /status
             port: 80
           initialDelaySeconds: 30
           periodSeconds: 5

apps/directus/directus-deployment.yaml

@@ -35,7 +35,7 @@ spec:
         fsGroup: 1000
       containers:
       - name: directus
-        image: directus/directus:12.1.1
+        image: directus/directus:12.3.0
         ports:
         - name: http
           containerPort: 8055

apps/dozzle/deployment.yaml

@@ -18,7 +18,7 @@ spec:
       serviceAccountName: dozzle
       containers:
       - name: dozzle
-        image: amir20/dozzle:v10.6.10
+        image: amir20/dozzle:v10.7.3
         ports:
         - containerPort: 8080
           name: dozzle-http

apps/forgejo-runner/README.md

@@ -50,17 +50,40 @@ kubectl apply -f apps/app-forgejo-runner.yaml
 
 ## Runner Configuration
 
-The runner is deployed as a StatefulSet with Docker-in-Docker (dind) sidecar for proper isolation and state management.
+The runner is deployed as a Deployment (single replica) that bind-mounts the
+host's `/var/run/docker.sock` to execute job containers. This requires the
+Docker Engine (package `docker.io`) to be installed and running on the target
+node — see **Node prerequisites** below.
 
 Configuration:
-- **Deployment type**: StatefulSet (stable pod identity, persistent storage)
-- **Docker execution**: Docker-in-Docker sidecar (privileged init container)
+- **Deployment type**: Deployment (replicas=1)
+- **Docker execution**: Host Docker socket (`/var/run/docker.sock`)
 - **Concurrent jobs**: 2 (configurable via config.yaml)
 - **Labels**: ubuntu-latest:docker://node:24-bookworm, ubuntu-22.04:docker://node:24-bookworm
 - **Forgejo URL**: https://forgejo.basicstack.de (external URL for proper webhook/API access)
-- **Persistent volumes**:
-  - runner-data (1Gi): Runner registration and config
-  - docker-data (20Gi): Docker image cache
+- **Node selector**: `basicstack.de/docker=true` — schedules only on nodes with the
+  Docker Engine installed. See **Node prerequisites** below.
+
+## Node prerequisites (required)
+
+The runner uses the host's Docker daemon. Every worker node that should be
+eligible to run the runner MUST have `docker.io` installed, the `docker`
+systemd unit enabled, and be labeled `basicstack.de/docker=true`. Bootstrap a
+worker with:
+
+```bash
+ssh root@<node>
+DEBIAN_FRONTEND=noninteractive apt-get update
+DEBIAN_FRONTEND=noninteractive apt-get install -y docker.io
+systemctl enable --now docker
+# from the control plane:
+kubectl label node <node-name> basicstack.de/docker=true --overwrite
+```
+
+Rolling OS updates that reimage a node will remove Docker. Re-run the bootstrap
+after any node reimage. Without Docker, the runner pod fails to mount
+`/var/run/docker.sock` (hostPath type check for `Socket` fails); the
+nodeSelector prevents that scheduling mistake by pinning to labeled nodes.
 
 ## Troubleshooting
 

apps/forgejo-runner/forgejo-runner-deployment.yaml

@@ -14,6 +14,8 @@ spec:
         app: forgejo-runner
     spec:
       serviceAccountName: forgejo-runner
+      nodeSelector:
+        basicstack.de/docker: "true"
       containers:
       - name: runner
         image: code.forgejo.org/forgejo/runner:4.0.1

apps/forgejo/forgejo-backup-cronjob.yaml

@@ -1,3 +1,18 @@
+---
+# Forgejo pg_dump → restic → Hetzner Object Storage (DEV-514).
+#
+# Replaces the legacy volume-based backup that wrote pg_dump files to
+# an RWO PVC (`platform-backup-data`). The hcloud volume backing that
+# PVC was deleted externally on 2026-08-17; rather than re-provision
+# the same legacy pattern, this migrates to the restic→S3 pipeline
+# already used for loki/grafana/prometheus (DEV-485…DEV-492).
+#
+# initContainer runs pg_dump -F c into an emptyDir; main container
+# runs `restic backup --stdin-from-command` isn't used here because
+# the dump has to complete before restic starts (need a proper exit
+# code, and the custom-format dump is not resumable). Instead we
+# stage the dump on emptyDir and let restic dedup it into
+# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/forgejo`.
 apiVersion: batch/v1
 kind: CronJob
 metadata:
@@ -5,56 +20,54 @@ metadata:
   namespace: forgejo
   labels:
     app: forgejo-backup
+    backend: restic
 spec:
   schedule: "0 3 * * *"
   concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 7
+  successfulJobsHistoryLimit: 3
   failedJobsHistoryLimit: 3
   jobTemplate:
+    metadata:
+      labels:
+        app: forgejo-backup
+        backend: restic
     spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 3600
       template:
+        metadata:
+          labels:
+            app: forgejo-backup
+            backend: restic
         spec:
           restartPolicy: OnFailure
-          containers:
-          - name: forgejo-backup
+          initContainers:
+            - name: pgdump
               image: postgres:16-alpine
               imagePullPolicy: IfNotPresent
               command:
                 - /bin/sh
-            - -lc
+                - -c
                 - |
-              set -euo pipefail
-              ts=$(date -u +%Y%m%dT%H%M%SZ)
-              BACKUP_DIR="/data/forgejo-${ts}"
-              mkdir -p "${BACKUP_DIR}"
-
-              echo "[${ts}] Starting Forgejo backup..."
-
-              # Backup PostgreSQL database
-              echo "[$(date -u +%H:%M:%S)] Dumping PostgreSQL database..."
+                  set -eu
+                  echo "=== forgejo pg_dump started at $(date -u +%FT%TZ) ==="
+                  mkdir -p /dump
                   PGPASSWORD="${POSTGRES_PASSWORD}" pg_dump \
                     -h forgejo-postgres.forgejo.svc.cluster.local \
                     -U "${POSTGRES_USER}" \
                     -d "${POSTGRES_DB}" \
                     -F c \
-                -f "${BACKUP_DIR}/forgejo-db.dump"
-
-              echo "[$(date -u +%H:%M:%S)] Database backup complete: $(ls -lh ${BACKUP_DIR}/forgejo-db.dump | awk '{print $5}')"
-
-              # Create backup manifest
-              echo "backup_timestamp=${ts}" > "${BACKUP_DIR}/manifest.txt"
-              echo "type=postgresql_custom_dump" >> "${BACKUP_DIR}/manifest.txt"
-              echo "database=forgejo" >> "${BACKUP_DIR}/manifest.txt"
-              echo "restore_cmd=PGPASSWORD=<pass> pg_restore -h <host> -U forgejo -d forgejo -F c forgejo-db.dump" >> "${BACKUP_DIR}/manifest.txt"
-
-              # Cleanup old backups (keep 14 days)
-              find /data -maxdepth 1 -type d -name "forgejo-*" -mtime +14 -exec rm -rf {} + 2>/dev/null || true
-
-              echo "[$(date -u +%H:%M:%S)] Backup complete. Files:"
-              ls -lh "${BACKUP_DIR}/"
-
-              echo "[$(date -u +%H:%M:%S)] All backups in storage:"
-              ls -1 /data | grep "forgejo-"
+                    -f /dump/forgejo-db.dump
+                  echo "pg_dump size: $(wc -c < /dump/forgejo-db.dump) bytes"
+                  # Manifest lets a restic-restore consumer identify the
+                  # dump without touching the binary.
+                  {
+                    echo "backup_timestamp=$(date -u +%Y%m%dT%H%M%SZ)"
+                    echo "type=postgresql_custom_dump"
+                    echo "database=${POSTGRES_DB}"
+                    echo "restore_cmd=PGPASSWORD=<pass> pg_restore -h <host> -U ${POSTGRES_USER} -d ${POSTGRES_DB} -F c forgejo-db.dump"
+                  } > /dump/manifest.txt
+                  echo "=== forgejo pg_dump finished at $(date -u +%FT%TZ) ==="
               env:
                 - name: POSTGRES_USER
                   valueFrom:
@@ -72,9 +85,120 @@ spec:
                       name: forgejo-postgres-secret
                       key: POSTGRES_DB
               volumeMounts:
-            - name: backup-data
-              mountPath: /data
+                - name: dump
+                  mountPath: /dump
+              resources:
+                requests:
+                  cpu: 100m
+                  memory: 128Mi
+                limits:
+                  cpu: 1000m
+                  memory: 512Mi
+          containers:
+            - name: restic
+              image: harbor.basicstack.de/library/restic:0.19.1
+              env:
+                - name: AWS_ACCESS_KEY_ID
+                  valueFrom:
+                    secretKeyRef:
+                      name: forgejo-s3-backup
+                      key: access-key
+                - name: AWS_SECRET_ACCESS_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: forgejo-s3-backup
+                      key: secret-key
+                - name: RESTIC_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: forgejo-s3-backup
+                      key: restic-password
+                - name: S3_ENDPOINT
+                  valueFrom:
+                    secretKeyRef:
+                      name: forgejo-s3-backup
+                      key: endpoint
+                - name: S3_BUCKET
+                  valueFrom:
+                    secretKeyRef:
+                      name: forgejo-s3-backup
+                      key: bucket
+                - name: RESTIC_REPOSITORY
+                  value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/forgejo"
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  echo "=== backup-forgejo-restic started at $(date -u +%FT%TZ) ==="
+                  echo "Repository: ${RESTIC_REPOSITORY}"
+
+                  if restic snapshots >/dev/null 2>&1; then
+                    echo "Repo exists, skipping init."
+                  else
+                    echo "Repo missing, initialising..."
+                    restic init
+                  fi
+
+                  echo "--- restic backup /source ---"
+                  restic backup /source \
+                    --tag forgejo \
+                    --host k3s
+
+                  echo "--- restic forget/prune ---"
+                  restic forget --tag forgejo \
+                    --keep-daily 7 \
+                    --keep-weekly 4 \
+                    --keep-monthly 6 \
+                    --prune
+
+                  echo "--- restic check --read-data-subset=5% ---"
+                  CHECK_STATUS=0
+                  restic check --read-data-subset=5% || CHECK_STATUS=$?
+                  echo "restic check exit: ${CHECK_STATUS}"
+
+                  echo "--- restic stats (repo size) ---"
+                  REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
+                    | grep -oE '"total_size":[0-9]+' \
+                    | head -1 \
+                    | cut -d: -f2)
+                  REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
+                  echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
+
+                  # Textfile-collector metrics, same wiring as
+                  # loki/grafana siblings (DEV-494). Atomic write.
+                  {
+                    echo "backup_forgejo_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
+                    echo "backup_forgejo_timestamp_seconds $(date +%s)"
+                    echo "backup_forgejo_check_status ${CHECK_STATUS}"
+                    echo "restic_repo_size_bytes{repo=\"forgejo\"} ${REPO_SIZE_BYTES}"
+                  } > /metrics/backup_forgejo.prom.tmp
+                  mv /metrics/backup_forgejo.prom.tmp /metrics/backup_forgejo.prom
+
+                  echo "=== backup-forgejo-restic finished at $(date -u +%FT%TZ) ==="
+                  exit ${CHECK_STATUS}
+              volumeMounts:
+                - name: dump
+                  mountPath: /source
+                  readOnly: true
+                - name: metrics
+                  mountPath: /metrics
+                - name: cache
+                  mountPath: /root/.cache/restic
+              resources:
+                requests:
+                  cpu: 100m
+                  memory: 128Mi
+                limits:
+                  cpu: 1500m
+                  memory: 1Gi
           volumes:
-          - name: backup-data
-            persistentVolumeClaim:
-              claimName: platform-backup-data
+            - name: dump
+              emptyDir:
+                sizeLimit: 5Gi
+            - name: metrics
+              hostPath:
+                path: /var/lib/node_exporter/textfile_collector
+                type: DirectoryOrCreate
+            - name: cache
+              emptyDir: {}

apps/forgejo/forgejo-s3-backup-sealed.yaml

@@ -0,0 +1,19 @@
+---
+apiVersion: bitnami.com/v1alpha1
+kind: SealedSecret
+metadata:
+  creationTimestamp: null
+  name: forgejo-s3-backup
+  namespace: forgejo
+spec:
+  encryptedData:
+    access-key: AgAu/XawJy0uhi3l8vwtd+loShP2ITnbOEZ/9eu6dkypz7l1MmFJhnEGq8Sy1lsg7BKj6epYRI40kDJIAG6KMBD5i7FvLYCvbFuPJaS3yehA8b1ECreHXvx/VYRUZL4K3WtiPO0fXmUNkxAqIcfHOcHtOZ7YcgSSyTw4DaXzngtek4xcTQlisyIVeBgc1dLPY+vHGn10BGkU5R/LnsUQqgGdp6u24Y+m7OrQGqMH8xwtZR7siEEkasu4IWG0QcY+okcMeiCi8mgAF64rdeZnIeEWqRnKHXxwEurtm20NmmCeRR8OgZfWDesgasThsnNSlRySd4EKlP3DMz8Yyluz/LlSkPt24brnP4qKuEoRpCHHOWgOizX8hkkS+Al7Rnpmxv2pt5CGlRw9s/5uPY81PAN9WLIgjs+XIylvaJBSW06nw3v+MjCUr5jfimW9riTLN32cBSurdpNMzlgb5+DJzEvaGNEG+jSjjFj2eSLci23UWySh7oFdmXeBp3kDWCrrNlJOYqJEwAAunTwvtM/1XknW2qJyyJrD7hbWN7SX4Ymb5Q5q18fMGH+IzB/Si1jdBli8pi8B2Gd/jo6RSmhXUheO3xsB2sXYc09GuSBBBBNzr9MsRL3mvUVr22pay+YPcCbwbwtdR0qmrdhEHAychYSJpNGuwGvs0AthRsNGN6hr1/IJqnnSRkGdJPr6k+T3v7XJzouYy+8Dh8r6oPW3hOhHo4HpJw==
+    bucket: AgAyQVCK+wE0wWxwVjx5DVnI2falTGNzsUrolv1YyoG2Bx6msYe2G00ovxCMJ6/dWws1Q/Sf6yoiFT0IByYflQFhY/ABzXRi7/gFxCWyYU/7yTo7w1Ue0+eEnBKFtkP/ZhIvtfubo6GDDtbmshpp4UtzPq7kB6qvE6oCFyf6awQD2pe/GKX8RZaDx5TCbxPxhedLkYdc7srR/8rFZBnQiXm/2ohtDOwvLA+RVOpnCW/bsVLvLYQmXyhszMv2M3zqXwM9ODYzQ4Y7lIw6T3Hsmel/1hHWJtN8/mlq+Q3Z/NbzhVeEWs8/xbK/ShC9RwZUgBWxAiW04HvwcxVYs60KaJ4oOMLtO4Iko4+iPTNonjra2wmziEpUtsotvtReRjwHdvk1bBn3q6uqzNaJFv9wLc/RVCvKSpqE5KOxmdbke9EDUs4wQHVQCAfFfwnzzpcF/Rf6ML3TUWc+U0W6UjCfVTWwROznpi0E18tA07apx+xm1y+MyuVYnSpZv++vXetnCjA+KFvqZSon/WnKQrmSO2RFR07FcMs6gQ+EOSaRXBPCnEcnF2IJUbJLdFsgpQbzYG6zo3m2KmxN7gaGrTUdhNg/N7LklnNbJML9KRa7e2Rv8t4NGoigOpZG6+soIijZOSfEJF6owN6MP3oecgUgNsnQ5ftKK/RznNaDtIeZ5KByHlQ7CsZ0MIkHBmoEkDt64xWNEoW4NrWPSxas6pOQGZMPMA==
+    endpoint: AgBhXqp3TdUU6GxsexRIoeHNwjvxnd2v4Any9WGs7gtyZvfdGIuWHIQ+JpsuRDrEq+BvTaX49D0QYoqnoQ6ZgmmGUI7kJxc+Ombcqo/8PAsOmzfdUaz1NcLAeR/UTGu8BDxAQcySSFax2dyK/zkY3jcsmuUCJNJLB57EZwtwx4fyz+ota+zNBE7pY+sDpWkXKHHtWrvNkD7tSLYHQuS3uzATidRixngL+iaH8rQX3ZK6KaK4H4ZdFG3/1Fw0wjLt8+yVBntBnh/bCL9/JLWQWIiHtooQnKXDdEwYQHWhrktti1KDzy/S++o4XVcRG1alzsQ0vV7uzSZAh4xCmnzximJxeJKawzFXRgFUcoEPrOI/nVedYrAU0DzkuwPdqtQDi/fGoqzZPx3PYhB9wv2PJsLg20ZlFcfHYUCad3O6OmO+VlrpYYcUe4oStkOygrmK5bv89fQ4Tov7BB14+RQIo9ogKmLzZu3xStqIVK+/Y8OUC64zyo7FpPMoYKtRW5oawu8pxo7Gtb+27BMMXQBiMEPNf38C4aj3GGJOvm4CFxVHV510/7o2hSW2ITt5X2RzNOEzULJOHD3DvO8wdWzvpBykRFnZ8JTYv77Kbx7bd9Ylx62VXm2a+Bd0K1Nc/sNJE63pglVhdl05oLYsUTrcSsNoJ6Dz3pMKeue3pYrn9KNurG0s+BFfO6I2Aa4/L8iAggXBzf9G1ALSkn7A30C9rMeyV+/1n+q8GpdPqq4ltqLOHVwIZw==
+    restic-password: AgB1JGFnSDe3N5ZaiT9xShfzUQiyDtGwH2wotYO5Ekk9nqAbS4xTNBXu/1Z1+cB8zVUP80hrPHHbVXjNu47AcLI+Bn8Ozn9fKIZFsnVHXqicG0Y+WHRMpQxuMijVEMl48oUNE+vw+xRhYTzibVWoxur+VEzyPCn8rLUX07tcNhY+jYptNVBYlVVXDrHcOs0s9wpvIlSWmc9RsINMu+6FaedXAs3nKY4gUpWxmNLEpLG8V7D2mNJZHk1adghTN9p8RMGcLtNylDY9W1Sdnziga/NG0WulZf+4IWV6e7wc2oBCSekTofCZuCva8HazJ1EIeqBvt2GuXoG1CjjbPNb2UwZeH21Iol1W0C5So1VFguxHZq5WL44mqD79NLlnSAMmQMSvxf7gXQVXEdtT5UlF9nYCkuQ+K+mT+8iRMy1YSCQSQ9kkD36unkrlYcLXKoPLYxHcoo7fGZ0MPIW8J+P77Hp9Z8jNPCyCaeSJYApCxybO8ujbUGdIr2FCP1z/JDP4/zZat9MkTA6G3H9kDNso7AJBHL/NfkWImvQnveLGj0CuhPpf2zwOylCqTeXrgoN8zztO2/qVDa3VRQbznap0njr0qzVrvOfTxYoiRmkGh3PXZ2mb/NqbeIvCFhj2VyVgavdMchlIbM3fCzXGEdCa2RN+ZYVtcFe/b6wR/qf7lrMaPHsTZiMfYuMJoC9F9wdbkOttT9c36Sob4x+aQYLs7/+XGCJdbw672qot6x6P7GHjY8zVKlMYLSZjKcoD48c=
+    secret-key: AgAHz7lfqTLG0Kldt/IZGW/o8KKazICHGbdVklXs3Gzs6Mc2KaG3nggkZDRWwY6noochg4Wc9mDDMGY3XIQnEy3PXs/rjQgwym0Dbvc54CvjWUcfmxCJyIqY1uCUxoXp/WAEV18lpWFSkVx+5U9wSccYVmkOJXNJ/MgH0vzJPaNW6xpYDg1XRC4nj72vwqyvZ/gM/LEX41NOpPjBROLGql3uBk5UfzNZG4dBun0aaPiYx5KKAbz8JAbK0umvOOOctTPSv1hIhyv9xJRRB67MW/J8T6mB2YfezsisXSSOAVNuE2JWrafHFxKq/hSNQHSrQRKjt18k6vfxmfkICbgQUZl/7AnoKi/f+76Feu+3wuMT+XPsaJN+iWb3jNzcb1T+Aiv4rGyIA7uCB6FNOc9Ngf5ngDQJw0nDNHDSnUoowgcE96kuPefU1rmn2V+JJcwoYdzYE3Ds48C0rNT+WSIit4551SiyWuj7iCRLhyEf5vhojxkuoyCN+EC1a50XwIkt223EBmy6FFPeBxql6HyJbOE313BPsQU6enPrTKNFd7iZYqJXq1k+SRuGwJhMeo1BBNhmVqxFGOpyxkSVs3VPybeXd82Zk/xF0SWKyBp3XQnLB39ZmT83Z5jwrYfkZmWkkdRvCME6NmlQtftRDDk3efGbgw7bz4Eq0d08KPCezsaDSAGNewonYy3UOiv/CNQg5hQ5G6R55y7+kEMV4nNkjGj0qF1hrZURS0mV4I/7H0OwXpDYiVetfaYI
+  template:
+    metadata:
+      creationTimestamp: null
+      name: forgejo-s3-backup
+      namespace: forgejo

apps/forgejo/platform-backup-data-pvc.yaml

@@ -1,14 +0,0 @@
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: platform-backup-data
-  namespace: forgejo
-  labels:
-    app: forgejo-backup
-spec:
-  accessModes:
-  - ReadWriteOnce
-  storageClassName: hcloud-volumes-encrypted
-  resources:
-    requests:
-      storage: 20Gi

apps/headlamp/deployment.yaml

@@ -16,7 +16,7 @@ spec:
       serviceAccountName: headlamp-admin
       containers:
       - name: headlamp
-        image: ghcr.io/headlamp-k8s/headlamp:v0.43.0
+        image: ghcr.io/headlamp-k8s/headlamp:v0.45.0
         imagePullPolicy: IfNotPresent
         args:
         - "-in-cluster"

apps/monitoring/README.md

@@ -2,9 +2,57 @@
 
 Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
 
-- `backup-k8s-resources-cronjob.yaml` — daily dump of Kubernetes resources into `backup-storage` PVC.
-- `backup-volumes-cronjob.yaml` — daily rsync/tar of Grafana + Loki PVCs into `backup-storage`. Prometheus data backup is **not** included here; it needs a separate on-node backup (tracked as a follow-up because Prometheus is on a different node than `backup-storage`).
+- `backup-k8s-resources-cronjob.yaml` — daily dump of cluster-scoped and per-namespace Kubernetes resources, streamed through `restic backup --stdin` to `hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses `serviceAccountName: backup-sa` and no PVC mount (init container `alpine/k8s:1.29.4` writes an emptyDir, main container `restic/restic:0.19.1` reads it on stdin). Rewritten from the local-path tarball per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-487](/DEV/issues/DEV-487)).
+- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
+- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
+- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block.
+- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the four restic repos (loki/grafana/k8s-resources/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)).
 
-The `backup-storage` PVC (100Gi, local-path, bound to k3s-worker-2) is the shared destination for both jobs.
+## Shared SealedSecret
 
-Both CronJobs pin themselves to `k3s-worker-2` via `nodeSelector` because that is the node that holds all destination + source PVCs used here.
+All four restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys:
+
+| Key              | Purpose                                                                                                          |
+|------------------|------------------------------------------------------------------------------------------------------------------|
+| `access-key`     | Hetzner Object Storage access key ID                                                                             |
+| `secret-key`     | Hetzner Object Storage secret access key                                                                         |
+| `endpoint`       | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`)                                                        |
+| `bucket`         | Bucket name (single bucket, per-prefix repos)                                                                    |
+| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all four repos (loki/grafana/k8s-resources/prometheus) — rotating rewrites the key file on every repo. |
+
+## Restore
+
+Restore procedure for all four restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16, loki/k8s-resources) passed — see [DEV-488](/DEV/issues/DEV-488). Prometheus repo added and drilled same day — see [DEV-492](/DEV/issues/DEV-492).
+
+## Emitted metrics (textfile-collector format)
+
+Each restic CronJob writes `/metrics/backup_<kind>.prom` (atomic — `.prom.tmp` + `mv`) into a `hostPath` volume mounted at the node-exporter textfile-collector directory (`/var/lib/node_exporter/textfile_collector`). The kube-prometheus-stack node-exporter DaemonSet has `--collector.textfile.directory=/host/textfile_collector` enabled ([DEV-494](/DEV/issues/DEV-494), applied via [`apps/observability/patches/node-exporter-textfile-collector.yaml`](../observability/patches/node-exporter-textfile-collector.yaml)) and surfaces those samples in Prometheus.
+
+| Metric                                     | Emitted by                                                                       |
+|--------------------------------------------|----------------------------------------------------------------------------------|
+| `backup_<kind>_success` (0/1)              | `backup-{loki,grafana,k8s-resources}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` |
+| `backup_<kind>_timestamp_seconds`          | same                                                                             |
+| `backup_<kind>_check_status` (exit code)   | same — from `restic check --read-data-subset=5%`                                 |
+| `backup_prometheus_backup_status`          | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) |
+| `restic_repo_size_bytes{repo="<kind>"}`    | same — from `restic stats --json --mode raw-data` (added in DEV-490)             |
+
+**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods, `k8s-resources` is unpinned), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.
+
+## Alerts
+
+`backup-restic-alerts.yaml` defines nine alerts (all `severity: warning`):
+
+- `BackupLokiStale` / `BackupGrafanaStale` / `BackupK8sResourcesStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
+- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupK8sResourcesCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
+- `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions.
+
+All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally:
+
+```bash
+awk '/^spec:/{f=1;next} f{sub(/^  /,"");print}' \
+  apps/monitoring/backup-restic-alerts.yaml > /tmp/backup-restic-rules.yaml
+promtool check rules /tmp/backup-restic-rules.yaml
+promtool test rules  apps/monitoring/backup-restic-alerts.test.yaml
+```
+
+The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `backup-k8s-resources` has no PVC dep and stays unpinned. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers.

apps/monitoring/backup-grafana-restic-cronjob.yaml

@@ -0,0 +1,160 @@
+---
+# Grafana data backup via restic to Hetzner Object Storage (DEV-486,
+# DEV-482 Option 4). Step 3 of the Option 4 rollout.
+#
+# Streams the RWO PVC `grafana-storage` (mounted read-only) into
+# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana`, a client-side
+# encrypted restic repository.
+#
+# `grafana-storage` is a local-path PV anchored on k3s-worker-2, so
+# the grafana pod is already pinned there; a plain nodeSelector on
+# the same host is enough (no podAffinity like the loki job needed).
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: backup-grafana-restic
+  namespace: monitoring
+  labels:
+    app: backup
+    type: grafana
+    backend: restic
+spec:
+  schedule: "15 3 * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    metadata:
+      labels:
+        app: backup
+        type: grafana
+        backend: restic
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 3600
+      template:
+        metadata:
+          labels:
+            app: backup
+            type: grafana
+            backend: restic
+        spec:
+          restartPolicy: OnFailure
+          nodeSelector:
+            kubernetes.io/hostname: k3s-worker-2
+          containers:
+            - name: restic
+              image: harbor.basicstack.de/library/restic:0.19.1
+              # Mirrored from docker.io/restic/restic:0.19.1 (DEV-493) —
+              # deterministic ingress via Harbor. Retag procedure in
+              # docs/monitoring/restic-restore.md § "Tag-bump procedure".
+              env:
+                - name: AWS_ACCESS_KEY_ID
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: access-key
+                - name: AWS_SECRET_ACCESS_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: secret-key
+                - name: RESTIC_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: restic-password
+                - name: S3_ENDPOINT
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: endpoint
+                - name: S3_BUCKET
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: bucket
+                - name: RESTIC_REPOSITORY
+                  value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/grafana"
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  echo "=== backup-grafana-restic started at $(date -u +%FT%TZ) ==="
+                  echo "Repository: ${RESTIC_REPOSITORY}"
+
+                  # First-run tolerance: init if the repo isn't there yet.
+                  if restic snapshots >/dev/null 2>&1; then
+                    echo "Repo exists, skipping init."
+                  else
+                    echo "Repo missing, initialising..."
+                    restic init
+                  fi
+
+                  echo "--- restic backup /source ---"
+                  restic backup /source \
+                    --tag grafana \
+                    --host k3s \
+                    --exclude '*.tmp'
+
+                  echo "--- restic forget/prune ---"
+                  restic forget --tag grafana \
+                    --keep-daily 7 \
+                    --keep-weekly 4 \
+                    --keep-monthly 6 \
+                    --prune
+
+                  echo "--- restic check --read-data-subset=5% ---"
+                  CHECK_STATUS=0
+                  restic check --read-data-subset=5% || CHECK_STATUS=$?
+                  echo "restic check exit: ${CHECK_STATUS}"
+
+                  echo "--- restic stats (repo size) ---"
+                  REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
+                    | grep -oE '"total_size":[0-9]+' \
+                    | head -1 \
+                    | cut -d: -f2)
+                  REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
+                  echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
+
+                  # Textfile-collector metrics; identical wiring to the
+                  # loki sibling. See that file for the atomic-write
+                  # rationale (DEV-494).
+                  {
+                    echo "backup_grafana_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
+                    echo "backup_grafana_timestamp_seconds $(date +%s)"
+                    echo "backup_grafana_check_status ${CHECK_STATUS}"
+                    echo "restic_repo_size_bytes{repo=\"grafana\"} ${REPO_SIZE_BYTES}"
+                  } > /metrics/backup_grafana.prom.tmp
+                  mv /metrics/backup_grafana.prom.tmp /metrics/backup_grafana.prom
+
+                  echo "=== backup-grafana-restic finished at $(date -u +%FT%TZ) ==="
+                  exit ${CHECK_STATUS}
+              volumeMounts:
+                - name: grafana-data
+                  mountPath: /source
+                  readOnly: true
+                - name: metrics
+                  mountPath: /metrics
+                - name: cache
+                  mountPath: /root/.cache/restic
+              resources:
+                requests:
+                  cpu: 100m
+                  memory: 128Mi
+                limits:
+                  cpu: 1500m
+                  memory: 1Gi
+          volumes:
+            - name: grafana-data
+              persistentVolumeClaim:
+                claimName: grafana-storage
+            - name: metrics
+              hostPath:
+                # node-exporter's textfile-collector directory
+                # (DEV-494). See sibling loki cronjob for detail.
+                path: /var/lib/node_exporter/textfile_collector
+                type: DirectoryOrCreate
+            - name: cache
+              emptyDir: {}

apps/monitoring/backup-k8s-resources-cronjob.yaml

@@ -1,3 +1,19 @@
+---
+# Kubernetes-resource backup via restic to Hetzner Object Storage
+# (DEV-487, DEV-482 Option 4). Step 4 of the Option 4 rollout.
+#
+# Streams a concatenated YAML dump of cluster-scoped and per-namespace
+# resources through `restic backup --stdin` into
+# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources`. No PVC mount
+# (drops the local-path `backup-storage` dependency), no node pin.
+#
+# Two-container pattern:
+#   1. `kubectl-dump` init container (`alpine/k8s:1.29.4`) writes
+#      /dump/cluster.yaml into an emptyDir. Uses `serviceAccountName:
+#      backup-sa` (unchanged from the legacy job).
+#   2. `restic` main container (`restic/restic:0.19.1`, matches the
+#      loki/grafana siblings) reads that file on stdin and streams it
+#      into the restic repo with `--stdin-filename cluster.yaml`.
 apiVersion: batch/v1
 kind: CronJob
 metadata:
@@ -6,6 +22,7 @@ metadata:
   labels:
     app: backup
     type: k8s-resources
+    backend: restic
 spec:
   schedule: "0 2 * * *"
   concurrencyPolicy: Forbid
@@ -13,72 +30,176 @@ spec:
   failedJobsHistoryLimit: 3
   jobTemplate:
     metadata:
-      annotations:
-        prometheus.io/scrape: "true"
       labels:
         app: backup
         type: k8s-resources
+        backend: restic
     spec:
       backoffLimit: 2
+      activeDeadlineSeconds: 3600
       template:
         metadata:
           labels:
             app: backup
+            type: k8s-resources
+            backend: restic
         spec:
           restartPolicy: OnFailure
           serviceAccountName: backup-sa
-          # backup-storage PVC (local-path) is bound to k3s-worker-2, so pin here.
-          nodeSelector:
-            kubernetes.io/hostname: k3s-worker-2
-          containers:
-          - name: kubectl-backup
+          initContainers:
+            - name: kubectl-dump
               image: alpine/k8s:1.29.4
               command:
                 - /bin/sh
                 - -c
                 - |
-              set -e
-              BACKUP_DIR="/backup/k8s-$(date +%Y%m%d-%H%M%S)"
-              mkdir -p "$BACKUP_DIR"
+                  set -eu
+                  echo "=== kubectl-dump started at $(date -u +%FT%TZ) ==="
+                  DUMP=/dump/cluster.yaml
+                  : > "${DUMP}"
 
-              echo "Starting Kubernetes resources backup to $BACKUP_DIR"
+                  echo "--- namespaces ---"
+                  kubectl get namespaces -o yaml >> "${DUMP}"
+                  echo "---" >> "${DUMP}"
 
-              kubectl get namespaces -o yaml > "$BACKUP_DIR/namespaces.yaml"
+                  echo "--- cluster-scoped resources ---"
+                  kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \
+                    -o yaml >> "${DUMP}"
+                  echo "---" >> "${DUMP}"
 
+                  echo "--- namespaced resources ---"
                   for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
-                mkdir -p "$BACKUP_DIR/$ns"
-                kubectl get configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \
-                  -n "$ns" -o yaml > "$BACKUP_DIR/$ns/resources.yaml" 2>/dev/null || true
+                    echo "  ns=${ns}"
+                    kubectl get \
+                      configmaps,secrets,services,deployments,statefulsets,daemonsets,jobs,cronjobs,ingresses,persistentvolumeclaims \
+                      -n "${ns}" -o yaml >> "${DUMP}" 2>/dev/null || true
+                    echo "---" >> "${DUMP}"
                   done
 
-              kubectl get persistentvolumes,storageclasses,clusterroles,clusterrolebindings \
-                -o yaml > "$BACKUP_DIR/cluster-resources.yaml"
-
-              cd /backup
-              tar -czf "k8s-backup-$(date +%Y%m%d-%H%M%S).tar.gz" "$(basename $BACKUP_DIR)"
-              rm -rf "$BACKUP_DIR"
-
-              find /backup -name "k8s-backup-*.tar.gz" -mtime +7 -delete
-
-              echo "Backup completed successfully"
-
-              echo "backup_k8s_resources_success 1" > /metrics/backup_success.prom
-              echo "backup_k8s_resources_timestamp $(date +%s)" >> /metrics/backup_success.prom
+                  echo "dump size: $(wc -c < ${DUMP}) bytes"
+                  echo "=== kubectl-dump finished at $(date -u +%FT%TZ) ==="
               resources:
                 requests:
                   cpu: 50m
                   memory: 128Mi
                 limits:
                   cpu: 500m
-                memory: 384Mi
+                  memory: 512Mi
               volumeMounts:
-            - mountPath: /backup
-              name: backup-storage
-            - mountPath: /metrics
-              name: metrics
-          volumes:
-          - name: backup-storage
-            persistentVolumeClaim:
-              claimName: backup-storage
+                - name: dump
+                  mountPath: /dump
+          containers:
+            - name: restic
+              image: harbor.basicstack.de/library/restic:0.19.1
+              # Mirrored from docker.io/restic/restic:0.19.1 (DEV-493) —
+              # deterministic ingress via Harbor. Retag procedure in
+              # docs/monitoring/restic-restore.md § "Tag-bump procedure".
+              env:
+                - name: AWS_ACCESS_KEY_ID
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: access-key
+                - name: AWS_SECRET_ACCESS_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: secret-key
+                - name: RESTIC_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: restic-password
+                - name: S3_ENDPOINT
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: endpoint
+                - name: S3_BUCKET
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: bucket
+                - name: RESTIC_REPOSITORY
+                  value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/k8s-resources"
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  echo "=== backup-k8s-resources-restic started at $(date -u +%FT%TZ) ==="
+                  echo "Repository: ${RESTIC_REPOSITORY}"
+
+                  # First-run tolerance: init if the repo isn't there yet.
+                  if restic snapshots >/dev/null 2>&1; then
+                    echo "Repo exists, skipping init."
+                  else
+                    echo "Repo missing, initialising..."
+                    restic init
+                  fi
+
+                  echo "--- restic backup --stdin cluster.yaml ---"
+                  restic backup --stdin \
+                    --stdin-filename cluster.yaml \
+                    --tag k8s-resources \
+                    --host k3s < /dump/cluster.yaml
+
+                  echo "--- restic forget/prune ---"
+                  restic forget --tag k8s-resources \
+                    --keep-daily 7 \
+                    --keep-weekly 4 \
+                    --keep-monthly 6 \
+                    --prune
+
+                  echo "--- restic check --read-data-subset=5% ---"
+                  CHECK_STATUS=0
+                  restic check --read-data-subset=5% || CHECK_STATUS=$?
+                  echo "restic check exit: ${CHECK_STATUS}"
+
+                  echo "--- restic stats (repo size) ---"
+                  REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
+                    | grep -oE '"total_size":[0-9]+' \
+                    | head -1 \
+                    | cut -d: -f2)
+                  REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
+                  echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
+
+                  # Textfile-collector metrics; identical wiring to the
+                  # loki/grafana siblings. See loki cronjob for the
+                  # atomic-write rationale (DEV-494).
+                  {
+                    echo "backup_k8s_resources_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
+                    echo "backup_k8s_resources_timestamp_seconds $(date +%s)"
+                    echo "backup_k8s_resources_check_status ${CHECK_STATUS}"
+                    echo "restic_repo_size_bytes{repo=\"k8s-resources\"} ${REPO_SIZE_BYTES}"
+                  } > /metrics/backup_k8s_resources.prom.tmp
+                  mv /metrics/backup_k8s_resources.prom.tmp /metrics/backup_k8s_resources.prom
+
+                  echo "=== backup-k8s-resources-restic finished at $(date -u +%FT%TZ) ==="
+                  exit ${CHECK_STATUS}
+              volumeMounts:
+                - name: dump
+                  mountPath: /dump
+                  readOnly: true
                 - name: metrics
+                  mountPath: /metrics
+                - name: cache
+                  mountPath: /root/.cache/restic
+              resources:
+                requests:
+                  cpu: 100m
+                  memory: 128Mi
+                limits:
+                  cpu: 1500m
+                  memory: 1Gi
+          volumes:
+            - name: dump
+              emptyDir: {}
+            - name: metrics
+              hostPath:
+                # node-exporter's textfile-collector directory
+                # (DEV-494). See sibling loki cronjob for detail.
+                path: /var/lib/node_exporter/textfile_collector
+                type: DirectoryOrCreate
+            - name: cache
               emptyDir: {}

apps/monitoring/backup-loki-restic-cronjob.yaml

@@ -0,0 +1,187 @@
+---
+# Loki data backup via restic to Hetzner Object Storage (DEV-485,
+# DEV-482 Option 4). Step 2 of the Option 4 rollout.
+#
+# Streams the RWO PVC `loki-storage-encrypted` (mounted read-only)
+# into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki`, a client-side
+# encrypted restic repository.
+#
+# podAffinity co-schedules with the Loki pod (app=loki, topology
+# kubernetes.io/hostname). RWO permits additional read-only mounts
+# on the node that holds the PVC's VolumeAttachment, so this
+# survives Loki being rescheduled to a different worker.
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: backup-loki-restic
+  namespace: monitoring
+  labels:
+    app: backup
+    type: loki
+    backend: restic
+spec:
+  schedule: "0 3 * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    metadata:
+      labels:
+        app: backup
+        type: loki
+        backend: restic
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 3600
+      template:
+        metadata:
+          labels:
+            app: backup
+            type: loki
+            backend: restic
+        spec:
+          restartPolicy: OnFailure
+          affinity:
+            podAffinity:
+              requiredDuringSchedulingIgnoredDuringExecution:
+                - labelSelector:
+                    matchExpressions:
+                      - key: app
+                        operator: In
+                        values:
+                          - loki
+                  topologyKey: kubernetes.io/hostname
+          containers:
+            - name: restic
+              image: harbor.basicstack.de/library/restic:0.19.1
+              # Mirrored from docker.io/restic/restic:0.19.1 (DEV-493) —
+              # deterministic ingress via Harbor. Retag procedure in
+              # docs/monitoring/restic-restore.md § "Tag-bump procedure".
+              env:
+                - name: AWS_ACCESS_KEY_ID
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: access-key
+                - name: AWS_SECRET_ACCESS_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: secret-key
+                - name: RESTIC_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: restic-password
+                - name: S3_ENDPOINT
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: endpoint
+                - name: S3_BUCKET
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: bucket
+                - name: RESTIC_REPOSITORY
+                  value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/loki"
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  echo "=== backup-loki-restic started at $(date -u +%FT%TZ) ==="
+                  echo "Repository: ${RESTIC_REPOSITORY}"
+
+                  # First-run tolerance: init if the repo isn't there yet.
+                  # `restic cat config` is the tightest existence probe; use
+                  # `snapshots` per plan spec — either exits 0 iff the repo
+                  # is initialised.
+                  if restic snapshots >/dev/null 2>&1; then
+                    echo "Repo exists, skipping init."
+                  else
+                    echo "Repo missing, initialising..."
+                    restic init
+                  fi
+
+                  echo "--- restic backup /source ---"
+                  restic backup /source \
+                    --tag loki \
+                    --host k3s \
+                    --exclude '*.tmp'
+
+                  echo "--- restic forget/prune ---"
+                  restic forget --tag loki \
+                    --keep-daily 7 \
+                    --keep-weekly 4 \
+                    --keep-monthly 6 \
+                    --prune
+
+                  echo "--- restic check --read-data-subset=5% ---"
+                  CHECK_STATUS=0
+                  restic check --read-data-subset=5% || CHECK_STATUS=$?
+                  echo "restic check exit: ${CHECK_STATUS}"
+
+                  echo "--- restic stats (repo size) ---"
+                  # `restic stats --json --mode raw-data` prints e.g.
+                  # {"total_size":123,"total_file_count":45,...}. Extract
+                  # total_size without jq (not present in the restic image)
+                  # via grep/cut; fall back to 0 on empty output.
+                  REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
+                    | grep -oE '"total_size":[0-9]+' \
+                    | head -1 \
+                    | cut -d: -f2)
+                  REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
+                  echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
+
+                  # Textfile-collector metrics. Written to the shared
+                  # host directory that node-exporter's textfile collector
+                  # scrapes (DEV-494). Atomic write: build the file with a
+                  # `.tmp` extension (ignored by node-exporter) and rename
+                  # into place, so a mid-write read never surfaces a
+                  # truncated sample.
+                  {
+                    echo "backup_loki_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
+                    echo "backup_loki_timestamp_seconds $(date +%s)"
+                    echo "backup_loki_check_status ${CHECK_STATUS}"
+                    echo "restic_repo_size_bytes{repo=\"loki\"} ${REPO_SIZE_BYTES}"
+                  } > /metrics/backup_loki.prom.tmp
+                  mv /metrics/backup_loki.prom.tmp /metrics/backup_loki.prom
+
+                  echo "=== backup-loki-restic finished at $(date -u +%FT%TZ) ==="
+                  exit ${CHECK_STATUS}
+              volumeMounts:
+                - name: loki-data
+                  mountPath: /source
+                  readOnly: true
+                - name: metrics
+                  mountPath: /metrics
+                - name: cache
+                  mountPath: /root/.cache/restic
+              resources:
+                # Requests deliberately lowered from the plan doc's
+                # 200m/256Mi — worker-2 (Loki node) has ~150m free CPU
+                # and podAffinity forces us onto it. 100m/128Mi mirrors
+                # the sibling backup CronJobs; limits stay generous so
+                # restic can burst during pack/check.
+                requests:
+                  cpu: 100m
+                  memory: 128Mi
+                limits:
+                  cpu: 1500m
+                  memory: 1Gi
+          volumes:
+            - name: loki-data
+              persistentVolumeClaim:
+                claimName: loki-storage-encrypted
+            - name: metrics
+              hostPath:
+                # node-exporter's textfile-collector directory
+                # (DEV-494). `DirectoryOrCreate` lets kubelet create
+                # the dir on the current node if it does not yet
+                # exist — nodes were pre-created out-of-band, this
+                # is a safety net.
+                path: /var/lib/node_exporter/textfile_collector
+                type: DirectoryOrCreate
+            - name: cache
+              emptyDir: {}

apps/monitoring/backup-restic-alerts.test.yaml

@@ -0,0 +1,260 @@
+---
+# promtool test file for backup-restic-alerts.yaml (DEV-490 DoD).
+#
+# Extract the rules from the PrometheusRule wrapper first:
+#   awk '/^spec:/{f=1;next} f{sub(/^  /,"");print}' \
+#     apps/monitoring/backup-restic-alerts.yaml > /tmp/rules.yaml
+# Then run:
+#   promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
+# (this file uses `rule_files: [/tmp/rules.yaml]` — pass the extracted
+# path explicitly).
+rule_files:
+  - /tmp/backup-restic-rules.yaml
+
+# Evaluation cadence for the test scenarios. The alerts' `for:` uses 5m/15m/30m,
+# so we run 60m of series and evaluate at 60m so all alerts have had time to
+# settle into the firing state.
+evaluation_interval: 1m
+
+tests:
+  # ---------------------------------------------------------------------------
+  # 1. Freshness alerts fire when timestamp is stale (>28h old).
+  # ---------------------------------------------------------------------------
+  - interval: 1m
+    input_series:
+      # Timestamp value = "current time - 29h" in Unix seconds. During the
+      # test, promtool's clock starts at the Unix epoch (0). We fix the
+      # value to a constant far in the past so `time() - value > 28*3600`
+      # for the entire test run.
+      #
+      # Concretely: time() returns the sample timestamp in seconds. Over a
+      # 60-minute test starting at t=0, time() runs 0 .. 3600. A constant
+      # value of -104400 (=-29h) keeps `time() - value` >= 29h across the
+      # whole window, safely above the 28h threshold.
+      - series: 'backup_loki_timestamp_seconds'
+        values: '-104400x60'
+      - series: 'backup_grafana_timestamp_seconds'
+        values: '-104400x60'
+      - series: 'backup_k8s_resources_timestamp_seconds'
+        values: '-104400x60'
+      - series: 'backup_prometheus_timestamp_seconds'
+        values: '-104400x60'
+    alert_rule_test:
+      - eval_time: 30m
+        alertname: BackupLokiStale
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: loki
+            exp_annotations:
+              summary: "Loki restic backup is stale (>28h)"
+              description: |
+                `backup-loki-restic` has not written a fresh
+                `backup_loki_timestamp_seconds` sample in more than
+                28 hours. Expected daily at 03:00 UTC.
+                Runbook: docs/monitoring/restic-restore.md
+      - eval_time: 30m
+        alertname: BackupGrafanaStale
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: grafana
+            exp_annotations:
+              summary: "Grafana restic backup is stale (>28h)"
+              description: |
+                `backup-grafana-restic` has not written a fresh
+                `backup_grafana_timestamp_seconds` sample in more
+                than 28 hours. Expected daily at 03:15 UTC.
+                Runbook: docs/monitoring/restic-restore.md
+      - eval_time: 30m
+        alertname: BackupK8sResourcesStale
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: k8s-resources
+            exp_annotations:
+              summary: "K8s-resources restic backup is stale (>28h)"
+              description: |
+                `backup-k8s-resources` has not written a fresh
+                `backup_k8s_resources_timestamp_seconds` sample in
+                more than 28 hours. Expected daily at 02:00 UTC.
+                Runbook: docs/monitoring/restic-restore.md
+      - eval_time: 30m
+        alertname: BackupPrometheusStale
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: prometheus
+            exp_annotations:
+              summary: "Prometheus restic backup is stale (>28h)"
+              description: |
+                `prometheus-backup` has not written a fresh
+                `backup_prometheus_timestamp_seconds` sample in more
+                than 28 hours. Expected daily at 03:30 UTC.
+                Runbook: docs/monitoring/restic-restore.md
+
+  # ---------------------------------------------------------------------------
+  # 2. Freshness alerts stay silent when timestamp is fresh (<28h).
+  # ---------------------------------------------------------------------------
+  - interval: 1m
+    input_series:
+      # Value = -3600 (=-1h) → time() - value stays around 1-2h, well below
+      # the 28h threshold across the run.
+      - series: 'backup_loki_timestamp_seconds'
+        values: '-3600x60'
+      - series: 'backup_grafana_timestamp_seconds'
+        values: '-3600x60'
+      - series: 'backup_k8s_resources_timestamp_seconds'
+        values: '-3600x60'
+      - series: 'backup_prometheus_timestamp_seconds'
+        values: '-3600x60'
+    alert_rule_test:
+      - eval_time: 30m
+        alertname: BackupLokiStale
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: BackupGrafanaStale
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: BackupK8sResourcesStale
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: BackupPrometheusStale
+        exp_alerts: []
+
+  # ---------------------------------------------------------------------------
+  # 3. Check-status alerts fire when the metric is non-zero.
+  # ---------------------------------------------------------------------------
+  - interval: 1m
+    input_series:
+      - series: 'backup_loki_check_status'
+        values: '1x30'
+      - series: 'backup_grafana_check_status'
+        values: '2x30'
+      - series: 'backup_k8s_resources_check_status'
+        values: '1x30'
+      - series: 'backup_prometheus_check_status'
+        values: '1x30'
+    alert_rule_test:
+      - eval_time: 15m
+        alertname: BackupLokiCheckFailed
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: loki
+            exp_annotations:
+              summary: "restic check failed on loki repo"
+              description: |
+                `restic check --read-data-subset=5%` returned
+                exit code 1 on the loki repository.
+                Runbook: docs/monitoring/restic-restore.md
+      - eval_time: 15m
+        alertname: BackupGrafanaCheckFailed
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: grafana
+            exp_annotations:
+              summary: "restic check failed on grafana repo"
+              description: |
+                `restic check --read-data-subset=5%` returned
+                exit code 2 on the grafana repository.
+                Runbook: docs/monitoring/restic-restore.md
+      - eval_time: 15m
+        alertname: BackupK8sResourcesCheckFailed
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: k8s-resources
+            exp_annotations:
+              summary: "restic check failed on k8s-resources repo"
+              description: |
+                `restic check --read-data-subset=5%` returned
+                exit code 1 on the k8s-resources
+                repository.
+                Runbook: docs/monitoring/restic-restore.md
+      - eval_time: 15m
+        alertname: BackupPrometheusCheckFailed
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: prometheus
+            exp_annotations:
+              summary: "restic check failed on prometheus repo"
+              description: |
+                `restic check --read-data-subset=5%` returned
+                exit code 1 on the prometheus
+                repository.
+                Runbook: docs/monitoring/restic-restore.md
+
+  # ---------------------------------------------------------------------------
+  # 4. Check-status alerts stay silent on 0.
+  # ---------------------------------------------------------------------------
+  - interval: 1m
+    input_series:
+      - series: 'backup_loki_check_status'
+        values: '0x30'
+      - series: 'backup_grafana_check_status'
+        values: '0x30'
+      - series: 'backup_k8s_resources_check_status'
+        values: '0x30'
+      - series: 'backup_prometheus_check_status'
+        values: '0x30'
+    alert_rule_test:
+      - eval_time: 15m
+        alertname: BackupLokiCheckFailed
+        exp_alerts: []
+      - eval_time: 15m
+        alertname: BackupGrafanaCheckFailed
+        exp_alerts: []
+      - eval_time: 15m
+        alertname: BackupK8sResourcesCheckFailed
+        exp_alerts: []
+      - eval_time: 15m
+        alertname: BackupPrometheusCheckFailed
+        exp_alerts: []
+
+  # ---------------------------------------------------------------------------
+  # 5. Repo-size alert fires when >20 GiB (21474836480 bytes).
+  # ---------------------------------------------------------------------------
+  - interval: 1m
+    input_series:
+      # 25 GiB = 26843545600 bytes.
+      - series: 'restic_repo_size_bytes{repo="loki"}'
+        values: '26843545600x45'
+      # 15 GiB = 16106127360 bytes (below threshold).
+      - series: 'restic_repo_size_bytes{repo="grafana"}'
+        values: '16106127360x45'
+    alert_rule_test:
+      - eval_time: 40m
+        alertname: ResticRepoOversize
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              service: monitoring
+              component: backup
+              repo: loki
+            exp_annotations:
+              summary: "restic repository loki exceeds 20 GiB"
+              description: |
+                Repository loki is
+                25GiB, above the 20 GiB
+                guard. Baseline is <5 GiB per repo. Investigate
+                retention/prune (see forget flags in the CronJob)
+                and dedup effectiveness.

apps/monitoring/backup-restic-alerts.yaml

@@ -0,0 +1,216 @@
+---
+# Prometheus alerting rules for the restic-based monitoring backups
+# (DEV-490 / DEV-482 Option 4). Freshness alerts fire when a
+# CronJob has not emitted its `backup_<kind>_timestamp_seconds`
+# metric within 28h (schedules are daily; 28h gives one missed
+# run + 4h grace before we page).
+#
+# Check-status alerts fire on the first non-zero exit from
+# `restic check --read-data-subset=5%`.
+#
+# Repo-size alerts fire when a repository exceeds 20 GiB — the
+# baseline is expected < 5 GiB per repo. The alert catches
+# retention/prune bugs and runaway growth. The metric is emitted
+# by the same CronJob step (see backup-*-restic-cronjob.yaml,
+# `restic stats --json --mode raw-data`).
+#
+# The metrics are written to the node's textfile-collector directory
+# (`/var/lib/node_exporter/textfile_collector`) and scraped by the
+# kube-prometheus-stack node-exporter DaemonSet (DEV-494).
+#
+# Because a CronJob may run on a different worker across days (loki
+# backup follows the loki pod; k8s-resources is unpinned), stale
+# `.prom` files can linger on nodes the job has since left. That
+# would leave a per-node series with an old timestamp/check-status
+# indefinitely. The alerts below aggregate across instances so a
+# single fresh sample from the node where the job currently runs is
+# enough to keep the freshness alert quiet, and check/size alerts
+# fire when *any* node reports a bad value (which is the correct
+# behaviour — a recent failure is still a signal).
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: backup-restic-alerts
+  namespace: monitoring
+  labels:
+    app: backup
+    backend: restic
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: backup-restic.freshness
+      interval: 60s
+      rules:
+        - alert: BackupLokiStale
+          # `max()` collapses per-node samples so a stale `.prom` file
+          # left behind on a node the loki backup no longer runs on
+          # does not fire this alert; the freshest sample wins.
+          expr: time() - max(backup_loki_timestamp_seconds) > 28 * 3600
+          for: 15m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: loki
+          annotations:
+            summary: "Loki restic backup is stale (>28h)"
+            description: |
+              `backup-loki-restic` has not written a fresh
+              `backup_loki_timestamp_seconds` sample in more than
+              28 hours. Expected daily at 03:00 UTC.
+              Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupGrafanaStale
+          expr: time() - max(backup_grafana_timestamp_seconds) > 28 * 3600
+          for: 15m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: grafana
+          annotations:
+            summary: "Grafana restic backup is stale (>28h)"
+            description: |
+              `backup-grafana-restic` has not written a fresh
+              `backup_grafana_timestamp_seconds` sample in more
+              than 28 hours. Expected daily at 03:15 UTC.
+              Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupK8sResourcesStale
+          expr: time() - max(backup_k8s_resources_timestamp_seconds) > 28 * 3600
+          for: 15m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: k8s-resources
+          annotations:
+            summary: "K8s-resources restic backup is stale (>28h)"
+            description: |
+              `backup-k8s-resources` has not written a fresh
+              `backup_k8s_resources_timestamp_seconds` sample in
+              more than 28 hours. Expected daily at 02:00 UTC.
+              Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupPrometheusStale
+          expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600
+          for: 15m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: prometheus
+          annotations:
+            summary: "Prometheus restic backup is stale (>28h)"
+            description: |
+              `prometheus-backup` has not written a fresh
+              `backup_prometheus_timestamp_seconds` sample in more
+              than 28 hours. Expected daily at 03:30 UTC.
+              Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupForgejoStale
+          expr: time() - max(backup_forgejo_timestamp_seconds) > 28 * 3600
+          for: 15m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: forgejo
+          annotations:
+            summary: "Forgejo restic backup is stale (>28h)"
+            description: |
+              `forgejo-backup` has not written a fresh
+              `backup_forgejo_timestamp_seconds` sample in more
+              than 28 hours. Expected daily at 03:00 UTC.
+              Runbook: docs/monitoring/restic-restore.md
+    - name: backup-restic.integrity
+      interval: 60s
+      rules:
+        - alert: BackupLokiCheckFailed
+          expr: backup_loki_check_status != 0
+          for: 5m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: loki
+          annotations:
+            summary: "restic check failed on loki repo"
+            description: |
+              `restic check --read-data-subset=5%` returned
+              exit code {{ $value }} on the loki repository.
+              Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupGrafanaCheckFailed
+          expr: backup_grafana_check_status != 0
+          for: 5m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: grafana
+          annotations:
+            summary: "restic check failed on grafana repo"
+            description: |
+              `restic check --read-data-subset=5%` returned
+              exit code {{ $value }} on the grafana repository.
+              Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupK8sResourcesCheckFailed
+          expr: backup_k8s_resources_check_status != 0
+          for: 5m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: k8s-resources
+          annotations:
+            summary: "restic check failed on k8s-resources repo"
+            description: |
+              `restic check --read-data-subset=5%` returned
+              exit code {{ $value }} on the k8s-resources
+              repository.
+              Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupPrometheusCheckFailed
+          expr: backup_prometheus_check_status != 0
+          for: 5m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: prometheus
+          annotations:
+            summary: "restic check failed on prometheus repo"
+            description: |
+              `restic check --read-data-subset=5%` returned
+              exit code {{ $value }} on the prometheus
+              repository.
+              Runbook: docs/monitoring/restic-restore.md
+        - alert: BackupForgejoCheckFailed
+          expr: backup_forgejo_check_status != 0
+          for: 5m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+            repo: forgejo
+          annotations:
+            summary: "restic check failed on forgejo repo"
+            description: |
+              `restic check --read-data-subset=5%` returned
+              exit code {{ $value }} on the forgejo
+              repository.
+              Runbook: docs/monitoring/restic-restore.md
+    - name: backup-restic.size
+      interval: 60s
+      rules:
+        - alert: ResticRepoOversize
+          # 20 GiB = 20 * 1024^3 = 21474836480 bytes.
+          expr: restic_repo_size_bytes > 21474836480
+          for: 30m
+          labels:
+            severity: warning
+            service: monitoring
+            component: backup
+          annotations:
+            summary: "restic repository {{ $labels.repo }} exceeds 20 GiB"
+            description: |
+              Repository {{ $labels.repo }} is
+              {{ $value | humanize1024 }}B, above the 20 GiB
+              guard. Baseline is <5 GiB per repo. Investigate
+              retention/prune (see forget flags in the CronJob)
+              and dedup effectiveness.

apps/monitoring/backup-volumes-cronjob.yaml

@@ -1,102 +0,0 @@
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: backup-volumes
-  namespace: monitoring
-  labels:
-    app: backup
-    type: volumes
-spec:
-  schedule: "0 3 * * *"
-  concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    metadata:
-      annotations:
-        prometheus.io/scrape: "true"
-      labels:
-        app: backup
-        type: volumes
-    spec:
-      backoffLimit: 2
-      template:
-        metadata:
-          labels:
-            app: backup
-        spec:
-          restartPolicy: OnFailure
-          # backup-storage, grafana-storage and loki-storage-encrypted are all
-          # RWO PVCs pinned to k3s-worker-2. Prometheus data lives on
-          # k3s-worker-1 and is intentionally NOT backed up here — see
-          # DEV-464 for the split rationale and the follow-up ticket for
-          # a dedicated Prometheus data backup.
-          nodeSelector:
-            kubernetes.io/hostname: k3s-worker-2
-          containers:
-          - name: volume-backup
-            image: alpine:3.19
-            command:
-            - /bin/sh
-            - -c
-            - |
-              set -e
-              apk add --no-cache rsync
-
-              BACKUP_DATE=$(date +%Y%m%d-%H%M%S)
-              BACKUP_DIR="/backup/volumes-$BACKUP_DATE"
-              mkdir -p "$BACKUP_DIR"
-
-              echo "Starting volume backup to $BACKUP_DIR"
-
-              if [ -d "/source/grafana" ]; then
-                echo "Backing up Grafana data..."
-                rsync -a /source/grafana/ "$BACKUP_DIR/grafana/" || echo "Warning: Grafana backup incomplete"
-              fi
-
-              if [ -d "/source/loki" ]; then
-                echo "Backing up Loki data..."
-                rsync -a /source/loki/ "$BACKUP_DIR/loki/" || echo "Warning: Loki backup incomplete"
-              fi
-
-              cd /backup
-              tar -czf "volumes-backup-$BACKUP_DATE.tar.gz" "$(basename $BACKUP_DIR)"
-              rm -rf "$BACKUP_DIR"
-
-              find /backup -name "volumes-backup-*.tar.gz" -mtime +7 -delete
-
-              BACKUP_SIZE=$(du -sh "/backup/volumes-backup-$BACKUP_DATE.tar.gz" | cut -f1)
-              echo "Volume backup completed successfully: $BACKUP_SIZE"
-
-              echo "backup_volumes_success 1" > /metrics/backup_success.prom
-              echo "backup_volumes_timestamp $(date +%s)" >> /metrics/backup_success.prom
-            resources:
-              requests:
-                cpu: 100m
-                memory: 128Mi
-              limits:
-                cpu: 1000m
-                memory: 512Mi
-            volumeMounts:
-            - mountPath: /backup
-              name: backup-storage
-            - mountPath: /source/grafana
-              name: grafana-data
-              readOnly: true
-            - mountPath: /source/loki
-              name: loki-data
-              readOnly: true
-            - mountPath: /metrics
-              name: metrics
-          volumes:
-          - name: backup-storage
-            persistentVolumeClaim:
-              claimName: backup-storage
-          - name: grafana-data
-            persistentVolumeClaim:
-              claimName: grafana-storage
-          - name: loki-data
-            persistentVolumeClaim:
-              claimName: loki-storage-encrypted
-          - name: metrics
-            emptyDir: {}

apps/monitoring/loki-deployment.yaml

@@ -0,0 +1,52 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: loki
+  namespace: monitoring
+  labels:
+    app: loki
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: loki
+  # loki-storage-encrypted is RWO on hcloud-volumes-encrypted, so a rolling
+  # update deadlocks (new pod cannot attach the PVC while the old pod holds
+  # it). Recreate drops the old pod first so the CSI detaches the volume
+  # before the new pod tries to attach it. Same pattern as harbor's RWO fix.
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: loki
+    spec:
+      containers:
+      - name: loki
+        image: grafana/loki:2.9.2
+        args:
+        - -config.file=/etc/loki/loki.yaml
+        ports:
+        - containerPort: 3100
+          name: http
+        - containerPort: 9096
+          name: grpc
+        resources:
+          requests:
+            cpu: 200m
+            memory: 512Mi
+          limits:
+            cpu: 500m
+            memory: 1Gi
+        volumeMounts:
+        - name: loki-config
+          mountPath: /etc/loki
+        - name: loki-storage
+          mountPath: /loki
+      volumes:
+      - name: loki-config
+        configMap:
+          name: loki-config
+      - name: loki-storage
+        persistentVolumeClaim:
+          claimName: loki-storage-encrypted

apps/monitoring/prometheus-backup-cronjob.yaml

@@ -0,0 +1,228 @@
+---
+# Prometheus data backup via restic to Hetzner Object Storage
+# (DEV-492, DEV-482 Option 4). Replaces the DEV-465 rclone-sync job so
+# every monitoring backup ships client-side-encrypted; Hetzner Object
+# Storage has no SSE-S3/SSE-KMS, so the previous plaintext-at-rest
+# object layout was the only remaining gap.
+#
+# Streams the RWO PVC `prometheus-data-encrypted` (mounted read-only)
+# into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, a client-side
+# encrypted restic repository (tag=`prometheus`, host=`k3s`).
+#
+# Node scheduling matches the previous job: podAffinity co-schedules
+# with the Prometheus pod (app=prometheus, topology
+# kubernetes.io/hostname). Hetzner CSI RWO permits additional read-only
+# mounts on the node that holds the PVC's VolumeAttachment, so this
+# survives Prometheus being rescheduled to a different worker.
+#
+# Prometheus TSDB compaction race
+# --------------------------------
+# Prometheus rewrites the on-disk store roughly every 2 h: it creates a
+# new block dir, then deletes the source dirs. restic walks the source
+# tree once and may catch a file that disappeared mid-walk; restic
+# 0.17.3 exits 3 ("at least one source file could not be read") in
+# that case, and the snapshot excludes only the missing file. The
+# next daily run picks up the successor block, so the race is not a
+# data-loss risk — but we must not treat exit 3 as a hard failure, or
+# the daily job will alert-flap.
+#
+# Mitigation:
+#   - exclude `wal/*` (WAL is replayed from a fresh instance on
+#     restart; we accept losing the last ~15 s of ingested samples
+#     rather than snapshotting a moving segment)
+#   - exclude `chunks_head/*` (in-memory head block; ephemeral, would
+#     be rebuilt from WAL which we do not keep)
+#   - exclude Prometheus lock/scratch files (`lock`, `queries.active`,
+#     `*.tmp`, `lost+found/*`)
+#   - treat restic exit code 3 as a soft warning (log, continue);
+#     any other non-zero exit is still fatal
+#   - restore drill re-runs `promtool tsdb analyze` against every
+#     block so a corrupted snapshot is caught end-to-end
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: prometheus-backup
+  namespace: monitoring
+  labels:
+    app: backup
+    type: prometheus
+    backend: restic
+spec:
+  schedule: "30 3 * * *"  # daily 03:30, offset from loki/grafana/k8s-resources
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    metadata:
+      labels:
+        app: backup
+        type: prometheus
+        backend: restic
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 3600
+      template:
+        metadata:
+          labels:
+            app: backup
+            type: prometheus
+            backend: restic
+        spec:
+          restartPolicy: OnFailure
+          affinity:
+            podAffinity:
+              requiredDuringSchedulingIgnoredDuringExecution:
+                - labelSelector:
+                    matchExpressions:
+                      - key: app
+                        operator: In
+                        values:
+                          - prometheus
+                  topologyKey: kubernetes.io/hostname
+          containers:
+            - name: restic
+              image: harbor.basicstack.de/library/restic:0.19.1
+              env:
+                - name: AWS_ACCESS_KEY_ID
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: access-key
+                - name: AWS_SECRET_ACCESS_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: secret-key
+                - name: RESTIC_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: restic-password
+                - name: S3_ENDPOINT
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: endpoint
+                - name: S3_BUCKET
+                  valueFrom:
+                    secretKeyRef:
+                      name: monitoring-s3-backup
+                      key: bucket
+                - name: RESTIC_REPOSITORY
+                  value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/prometheus"
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  echo "=== backup-prometheus-restic started at $(date -u +%FT%TZ) ==="
+                  echo "Repository: ${RESTIC_REPOSITORY}"
+
+                  # First-run tolerance: init if the repo isn't there yet.
+                  if restic snapshots >/dev/null 2>&1; then
+                    echo "Repo exists, skipping init."
+                  else
+                    echo "Repo missing, initialising..."
+                    restic init
+                  fi
+
+                  # See "Prometheus TSDB compaction race" in the file
+                  # header. WAL + chunks_head are excluded on purpose;
+                  # exit code 3 (source file vanished mid-walk during a
+                  # compaction) is accepted and reported, any other
+                  # non-zero exit is fatal.
+                  echo "--- restic backup /source (exclude wal/chunks_head + lock files) ---"
+                  BACKUP_STATUS=0
+                  restic backup /source \
+                    --tag prometheus \
+                    --host k3s \
+                    --exclude 'wal/*' \
+                    --exclude 'chunks_head/*' \
+                    --exclude 'lock' \
+                    --exclude 'queries.active' \
+                    --exclude 'lost+found/*' \
+                    --exclude '*.tmp' || BACKUP_STATUS=$?
+                  echo "restic backup exit: ${BACKUP_STATUS}"
+                  if [ "${BACKUP_STATUS}" -eq 0 ]; then
+                    echo "backup: all files captured cleanly"
+                  elif [ "${BACKUP_STATUS}" -eq 3 ]; then
+                    echo "backup: exit 3 (source files vanished mid-walk) — expected under Prometheus compaction, continuing"
+                  else
+                    echo "backup: FATAL exit ${BACKUP_STATUS} (not compaction-race)"
+                    exit ${BACKUP_STATUS}
+                  fi
+
+                  echo "--- restic forget/prune ---"
+                  restic forget --tag prometheus \
+                    --keep-daily 7 \
+                    --keep-weekly 4 \
+                    --keep-monthly 6 \
+                    --prune
+
+                  echo "--- restic check --read-data-subset=5% ---"
+                  CHECK_STATUS=0
+                  restic check --read-data-subset=5% || CHECK_STATUS=$?
+                  echo "restic check exit: ${CHECK_STATUS}"
+
+                  echo "--- restic stats (repo size) ---"
+                  REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
+                    | grep -oE '"total_size":[0-9]+' \
+                    | head -1 \
+                    | cut -d: -f2)
+                  REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
+                  echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
+
+                  # Textfile-collector metrics. Same hostPath pattern as
+                  # the loki/grafana/k8s-resources siblings (DEV-494) —
+                  # written atomically via `.tmp` + rename so a mid-write
+                  # read never surfaces a truncated sample.
+                  # backup_prometheus_success rolls in both stages: the
+                  # backup step (accepting exit 3) and restic check.
+                  BACKUP_OK=0
+                  if [ "${BACKUP_STATUS}" -eq 0 ] || [ "${BACKUP_STATUS}" -eq 3 ]; then
+                    BACKUP_OK=1
+                  fi
+                  SUCCESS=0
+                  if [ "${BACKUP_OK}" -eq 1 ] && [ "${CHECK_STATUS}" -eq 0 ]; then
+                    SUCCESS=1
+                  fi
+                  {
+                    echo "backup_prometheus_success ${SUCCESS}"
+                    echo "backup_prometheus_timestamp_seconds $(date +%s)"
+                    echo "backup_prometheus_check_status ${CHECK_STATUS}"
+                    echo "backup_prometheus_backup_status ${BACKUP_STATUS}"
+                    echo "restic_repo_size_bytes{repo=\"prometheus\"} ${REPO_SIZE_BYTES}"
+                  } > /metrics/backup_prometheus.prom.tmp
+                  mv /metrics/backup_prometheus.prom.tmp /metrics/backup_prometheus.prom
+
+                  echo "=== backup-prometheus-restic finished at $(date -u +%FT%TZ) ==="
+                  exit ${CHECK_STATUS}
+              volumeMounts:
+                - name: prometheus-data
+                  mountPath: /source
+                  readOnly: true
+                - name: metrics
+                  mountPath: /metrics
+                - name: cache
+                  mountPath: /root/.cache/restic
+              resources:
+                # Prometheus TSDB is ~8-10 GiB; give restic room to
+                # burst during pack/check but keep steady-state small.
+                requests:
+                  cpu: 100m
+                  memory: 256Mi
+                limits:
+                  cpu: 1500m
+                  memory: 1Gi
+          volumes:
+            - name: prometheus-data
+              persistentVolumeClaim:
+                claimName: prometheus-data-encrypted
+            - name: metrics
+              hostPath:
+                # node-exporter's textfile-collector directory
+                # (DEV-494). See sibling loki cronjob for detail.
+                path: /var/lib/node_exporter/textfile_collector
+                type: DirectoryOrCreate
+            - name: cache
+              emptyDir: {}

apps/monitoring/prometheus-backup-sealed.yaml

@@ -0,0 +1,20 @@
+---
+apiVersion: bitnami.com/v1alpha1
+kind: SealedSecret
+metadata:
+  creationTimestamp: null
+  name: monitoring-s3-backup
+  namespace: monitoring
+spec:
+  encryptedData:
+    access-key: AgBf1XxCF3/Uv1vVlJHyTSEMONqWBqPfSOax8jJJHy3NOpgGga3SsxDfAbPXe2rBOYSgGzRZcaT8TcNg8Ng1ipv5Dhu7LzmRb6v6fW0NimrHSU/dtzzJSIR7jQ1sf+Qn4CPS73ReDRvUJ7g4o57ClZIVFBk9oPUDJYdjtg9kAM5Me0ZyqVp4Cnyj2TRXYcQPM+RN1r/PXqK4MhlUsmhsbQCPYwocM0liqv0T8L5xR1g2845oMaMx2nmLs1c1R8nCRugK8IeBKf/CCKLYg8yTqWKS2arrfxm+dkl1mrqLdOgnXrqeeo/63lT1sbd06ub86PJr83MXWW0yh1Uc4r7DX7xK/JIGJ4w+jtQa0vRutYANDcdeVlzjVl+ghoxxw/ZizYVQ/mUFCc+ApJD9u709mI39xof77U5yYBRhnZEExRseL756APubmCc0tPfUSNA1GmMhiYaMnyWu6F5Okreto/03AgI3t7lQm9SPRRM7hcHETG/QC8PrZ0NZEN/A2Nm/ZzwDG4HPDaNxH+IV90f5X4wuh6Mh2lffidOix88rkaizQAeVY5gt8tvfSfyfRc5RoOOov+fvUVCqJHnWaD1A2lKScVeCgLv8/0CJTyR0zM6cg2XN8z0HvU/sahfZrG23fVs3E/IFM5D7//CDp+GfcwQOCBZKPLvJtTRh5nA5EbXQusGEaMOTBiiGG0IJKPejKcPRWS1g+owu+G5R6QQjZgNDoP0cPg==
+    bucket: AgCYDCrQalif3+XZYhe4Xxyypg2OMGvzdNyBmHzMaP+ou1g2Q0wIjH6S1+YpPu+JgmaATZVISoSZ+OIl0mNOibiFMr0kFjuMhvKRt9Pq05+6axnqnJ17U2GNsZQtY3Ic93qxkkhJKCOQ87mvAJ5AIVBc3pAqfkD5YYTllgnSKwUwyaWK3iX3hxuKLYDvsQ2WTtcc/gsnlCkm1dwPE4E7DFt4wh+jQJRD314Mt7KSpvPBpz7yVVFFX+xRKCfRxjcwshUSUI5OgagfOQvsivkxlL0oSqA9veX2MJpd0SJH0T1Y0na0evkJitxxe+ECwM4FluK/iwW4cdUu+EpQIdd5otY9C5b8e8PJp8ah1k1hYzXdX+j975MeyXWMUn/NgXfyT2CUef6hmmmImialrSYsHT72bz3xpAnZ/oQ/0qcZpNje2oJeokNgkoiC0E5XRnfwEF9jL3XL+aB6gQ3BD9YRptaxLLxOQNj51Oe82vVsOnUz+utzJO1DtqpF+8I0lvHfgcUhFQytUuLscilfBQO/ru8wh47ZI+LRiS3qCZp2Pqi9+r7Y7jJKcbb2awWp6lgrrqgLMvnbVMuvxBi2hUy0C+EkP8tayRJDzz72utx8FY/v0FWoRDl9BAi8kn11+kbbo8a2OXe+20UkmNzunz5H0ijIxhJWzVh/oLuuLFCS0YJtJNjuqUB0VctAFcR4p5Agua8LA9+73HIBsHBYexzmvEZ6Ng==
+    endpoint: AgCV6YtzikWYMRwHZM1tT8epq6XiSVqhbxMhS7htVjVUfdR4V7/KIcNExxSBul+TU+RCq3S7fppL7CxKbI/IsEdirRMj2PvESXEmzLY/xId7+wJFEdxeXxn59cbV2MfCyysKIwzsVXbmzO66yGaeMKU2oyqJVYu+00L9xS0RbtQRbi0UksMxtw0PhZVMee0aHfu9jynWI3lmPzC/EPRuwxQqBIjYM2v2JCDmYjP0fsBSZSJc+SnAaNmmIf7ypcFwy14qIF7C9tIg02lBx+Sg30oMPjn600AbyF6RmZ+mdclp/Qp+zXOUsbMym0Lu9BjbM7G/z0ExpR5sgE/6+05gJhPF7MuHkrYIMo9LHWWdgzNTnjikSTGEeuFzzRlThPMoUfVi/S7KhG9NtQRnNqEoVCGGW6tt5sygXVE/lWSnmmd//4jXvF9ZJHU+5LrjL7UfVIYNH4y7aYtXpoTgmUe85wWqeMdqr1EiSVueoSwCggGITYoXoKYfERfij8ln/xnX1Kt2xxuyLtpDOczBYZK6LLwTpaXVVe+lO/xBgevWaoEz6si2hyj+glluMQWxPsOnPwUfRi5qqlQxgMZK/9aALYhPLquc16iMG+4FdtoOUfQ0XlLvurHsbCt9lUy3YrsmpDhv42FfamVlcK+gMz7mqYhNhd3fVjxCiUuwvzd01QHcx8ikv48YOnDuRiweRszbPNvlaK2KH6fo7b4l9mmaa4W+MaxZCxG3KSMOafwRmyusEUVNWw==
+    secret-key: AgBItdfzRgcgJvK5HALDnkbGH1DveAgcmS+Q4f16VwX1KY8TzGWpjoaOdsLCwzTWOFME5xjrRyOwwAcdtP65ccWlEZvvoS4kpqv49G1ebblD1wmApnpa8vD63Rd028vUYFF3CnwntX2USxF6t4rD68sZqqzMDyuG9Av93HAKVMNdx9V4AJ45nvfF2WR7ohk37NhQifOdKuelgRV6ahl3/fm/XPRGzf+u7OER8jll9n/ywE0mSkoKBXocbkQ2Xrn+utzplh9BoR9KSPTdnmI8ZXUkDTR9p+7PN4r/HheEdTUG0aaefdEDCbdDlv1EW84l5KzhY4v9g0DW8GQQdyHWi3sMm0hSGFg9NV/Vs1HM9FZZCyjS+mzR4CbuHgkY9mvj+Svn7oS+Fy5d24s1ekmPheakBffXKy2Q2CLMRGNymjUIBQsDwoGC8o4IF+yPyDcw/OvXlr0lruxfRM6IN0oxgRmLAQiqwRzsZb3RPgILEZI9VXKxzCBEuCO7fE2pC05K/mujhUutWmJ5tQ0NFzP1mhNK59cAIVMl7y7Llq5/GZ5ax+voP0c3VandXJeaSRr6O40vyzfCMgpgEgRs9t2WfxCmVYEN2NWpFv07JNICWzJX40PLT9n/euRXzSmJld6UfjzRteaTfmo+OxFczJ6Zl8XdC+6x6wO0Nk0hVYo27s2LH4o0MsM84x9Z3StCCGl77/NnyavpxfDkj3qJQ4BgPtCw26tHTt9DIWYfgeAVqvoRZVlg8YxM3N1k
+    restic-password: AgBuoBFK5Y2Vlf1Vj6Q7y5OwWV2iOLMATe+Kjca8uwHgiYnpka+619ab52P1P+tkxbGyGdnwF+3q0KzvJJaNfuoduwlPuRjiYmaLLoCuQahZmTLNbzwEFst1cbm4oG6vCTBGsK9IMngT6VMfmvz029X+jsGaJHaECHlJjWnMteKN2pwxbzt4/3Wx6wP3//qAk1ah4QLRZUEWvn6O7djTFWJtzM4MRfZPI6EmM9emlQRN9iHwWSUmgBZ5zvaAylRHHUHK1lkNgWJJr7+tC4D8DbZ+c4+FO6XefXDheG6jO2Ial09KTISC0+v86xKFq/aSko5TKczpdXEdYNfyf/aPJTRd4LTcTySUdIpflmjIUnJBdn2Yn4Jox8LKc6ecUJ+IxMXhWlToFNtSFJwx744uv9WsrQ7p/lH7VTZOT+R+0wKTe9vn07h6B78xX6JJWKHMepb9aJB0gB1fn9cx1KHzm4/Y30SFwuOpYpVx19lyl1BRfSry4kmPuDKouSO6D415NJ/lMNYqTvafRho5SzlFfle/n7wAK7OIRdNbgsabO2mfHjHpw6w8Bvo9K7+3QJiwpeesY8OKAubcpxy1RBDw3fp2JQCNNqIer41vUtprsRU+tBnzDboImhOTXpP71AgTa+L68k/iGVJB0YqjFDxjcJB1f1hKGvH+yCV0GPFSeUFOZ/gmxV+Q0Fmoq/ezLrqpjoTbjp+ZbAi6qbVrMN/3t3lheKf47dU5HEOQMyZkXqDrkyuc7kXhmzN/V/ISmW0=
+  template:
+    metadata:
+      creationTimestamp: null
+      name: monitoring-s3-backup
+      namespace: monitoring
+    type: Opaque

apps/observability/patches/README.md

@@ -0,0 +1,38 @@
+# observability patches
+
+Strategic-merge patches applied on top of Helm-managed observability
+resources. Each file is idempotent (re-applying is a no-op) and is
+reasserted by hand rather than by a controller, so re-run after any
+`helm upgrade` of the affected release.
+
+## `node-exporter-textfile-collector.yaml` (DEV-494)
+
+Enables the node-exporter textfile collector on the
+`kube-prometheus-stack-prometheus-node-exporter` DaemonSet by:
+
+1. adding `--collector.textfile.directory=/host/textfile_collector`
+   to the container args, and
+2. mounting the host directory `/var/lib/node_exporter/textfile_collector`
+   read-only at `/host/textfile_collector` (`hostPath` type
+   `DirectoryOrCreate`, kubelet creates it on nodes where the
+   directory does not exist yet).
+
+The four monitoring backup CronJobs in `apps/monitoring/` write their
+textfile-collector `.prom` files into that same host directory, so the
+metrics surface in Prometheus via node-exporter's normal scrape.
+
+Apply / re-apply:
+
+```bash
+kubectl -n observability patch daemonset \
+  kube-prometheus-stack-prometheus-node-exporter \
+  --type=strategic \
+  --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
+kubectl -n observability rollout status daemonset \
+  kube-prometheus-stack-prometheus-node-exporter
+```
+
+The kube-prometheus-stack chart is not currently tracked in ArgoCD;
+if it moves under GitOps, fold these values into the chart values as
+`prometheus-node-exporter.extraArgs` + `.extraHostVolumeMounts`
+instead of maintaining this patch.

apps/observability/patches/node-exporter-textfile-collector.yaml

@@ -0,0 +1,70 @@
+---
+# Strategic-merge patch enabling node-exporter's textfile collector on
+# the kube-prometheus-stack node-exporter DaemonSet (DEV-494).
+#
+# The chart is Helm-managed (release `kube-prometheus-stack` in
+# namespace `observability`, chart kube-prometheus-stack-86.2.2 /
+# prometheus-node-exporter-4.55.0) and is NOT currently tracked in
+# ArgoCD, so a direct DaemonSet patch is the pragmatic wiring path.
+# The change:
+#
+#   1. adds the `--collector.textfile.directory=/host/textfile_collector`
+#      arg to the node-exporter container, and
+#   2. mounts the host directory `/var/lib/node_exporter/textfile_collector`
+#      read-only at `/host/textfile_collector` (type DirectoryOrCreate so
+#      kubelet creates the dir on nodes where it does not yet exist).
+#
+# Apply with:
+#   kubectl -n observability patch daemonset \
+#     kube-prometheus-stack-prometheus-node-exporter \
+#     --type=strategic \
+#     --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
+#
+# If the Helm release is ever `helm upgrade`d without folding these
+# values into the chart values, this patch will be reverted — re-apply
+# it after the upgrade (or move it into a repo-owned values file).
+spec:
+  template:
+    spec:
+      containers:
+        - name: node-exporter
+          args:
+            - --path.procfs=/host/proc
+            - --path.sysfs=/host/sys
+            - --path.rootfs=/host/root
+            - --path.udev.data=/host/root/run/udev/data
+            - --web.listen-address=[$(HOST_IP)]:9100
+            - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
+            - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$
+            - --collector.textfile.directory=/host/textfile_collector
+          volumeMounts:
+            - mountPath: /host/proc
+              name: proc
+              readOnly: true
+            - mountPath: /host/sys
+              name: sys
+              readOnly: true
+            - mountPath: /host/root
+              mountPropagation: HostToContainer
+              name: root
+              readOnly: true
+            - mountPath: /host/textfile_collector
+              name: textfile-collector
+              readOnly: true
+      volumes:
+        - hostPath:
+            path: /proc
+            type: ""
+          name: proc
+        - hostPath:
+            path: /sys
+            type: ""
+          name: sys
+        - hostPath:
+            path: /
+            type: ""
+          name: root
+        - hostPath:
+            path: /var/lib/node_exporter/textfile_collector
+            type: DirectoryOrCreate
+          name: textfile-collector

apps/opencloud/opencloud-deployment.yaml

@@ -42,6 +42,8 @@ metadata:
   namespace: opencloud
 spec:
   replicas: 1
+  strategy:
+    type: Recreate
   selector:
     matchLabels:
       app: opencloud
@@ -56,7 +58,7 @@ spec:
         runAsNonRoot: true
       initContainers:
       - name: init-dirs
-        image: opencloudeu/opencloud-rolling:7.2.0
+        image: opencloudeu/opencloud-rolling:7.4.0
         command:
           - sh
           - -c
@@ -71,7 +73,7 @@ spec:
           runAsNonRoot: true
       containers:
       - name: opencloud
-        image: opencloudeu/opencloud-rolling:7.2.0
+        image: opencloudeu/opencloud-rolling:7.4.0
         command:
           - /bin/sh
           - -c

apps/pangolin/pangolin-deployment.yaml

@@ -35,7 +35,7 @@ spec:
     spec:
       initContainers:
       - name: wait-for-postgres
-        image: postgres:17.5
+        image: postgres:17.11
         command:
         - /bin/sh
         - -c
@@ -58,7 +58,7 @@ spec:
               name: pangolin-postgres-secrets
               key: postgres-db
       - name: render-config
-        image: busybox:1.37
+        image: busybox:1.38.0
         command:
         - /bin/sh
         - -c

apps/pangolin/postgres.yaml

@@ -57,7 +57,7 @@ spec:
         fsGroup: 999
       containers:
         - name: postgres
-          image: postgres:17.5
+          image: postgres:17.11
           imagePullPolicy: IfNotPresent
           ports:
             - containerPort: 5432

apps/paperclip/deployment.yaml

@@ -20,7 +20,7 @@ spec:
         fsGroupChangePolicy: Always
       initContainers:
       - name: fix-permissions
-        image: busybox:1.36
+        image: busybox:1.38.0
         command:
           - sh
           - -c

apps/pocket-id/deployment.yaml

@@ -17,7 +17,7 @@ spec:
     spec:
       containers:
       - name: pocket-id
-        image: ghcr.io/pocket-id/pocket-id:v2.11.0
+        image: ghcr.io/pocket-id/pocket-id:v2.14.0
         imagePullPolicy: IfNotPresent
         ports:
         - containerPort: 1411

apps/stalwart/stalwart-allow-cluster-ips-job.yaml

@@ -1,123 +0,0 @@
----
-# One-time Job to configure Stalwart HTTP listener to allow internal cluster IPs
-#
-# Problem: Stalwart blocks the HTTP port from Traefik's pod IP (10.244.2.227),
-#          causing 502/503 errors when accessing mail.basicstack.de
-#
-# Solution: Use kubectl exec to access Stalwart's admin API via localhost (which is allowed)
-#          and disable IP filtering for the HTTP listener, or allow the pod network CIDR
-#
-# This Job must be manually triggered after Stalwart is running:
-#   kubectl create job --from=cronjob/stalwart-allow-cluster-ips manual-fix -n stalwart
-#
-# Or apply directly:
-#   kubectl apply -f stalwart-allow-cluster-ips-job.yaml
-#   kubectl wait --for=condition=complete job/stalwart-allow-cluster-ips -n stalwart --timeout=120s
-#   kubectl logs -n stalwart job/stalwart-allow-cluster-ips
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: stalwart-config-access
-  namespace: stalwart
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: stalwart-config-access
-  namespace: stalwart
-rules:
-- apiGroups: [""]
-  resources: ["pods"]
-  verbs: ["get", "list"]
-- apiGroups: [""]
-  resources: ["pods/exec"]
-  verbs: ["create"]
-- apiGroups: [""]
-  resources: ["secrets"]
-  verbs: ["get"]
-  resourceNames: ["stalwart-admin-credentials"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: stalwart-config-access
-  namespace: stalwart
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: stalwart-config-access
-subjects:
-- kind: ServiceAccount
-  name: stalwart-config-access
-  namespace: stalwart
----
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: stalwart-allow-cluster-ips
-  namespace: stalwart
-spec:
-  ttlSecondsAfterFinished: 3600  # Keep logs for 1 hour
-  backoffLimit: 3
-  template:
-    metadata:
-      labels:
-        app: stalwart-security-fix
-    spec:
-      serviceAccountName: stalwart-config-access
-      restartPolicy: OnFailure
-      containers:
-      - name: fix-security
-        image: bitnami/kubectl:latest
-        command:
-        - /bin/bash
-        - -c
-        - |
-          set -e
-          echo "=== Stalwart HTTP Listener Security Fix ==="
-          echo "Configuring Stalwart to allow internal cluster IPs for the HTTP listener"
-          echo ""
-
-          # Wait for Stalwart pod to be ready
-          echo "Waiting for stalwart-0 pod to be ready..."
-          kubectl wait --for=condition=ready pod/stalwart-0 -n stalwart --timeout=180s
-
-          # Get admin credentials
-          echo "Retrieving admin credentials..."
-          ADMIN_EMAIL=$(kubectl get secret stalwart-admin-credentials -n stalwart -o jsonpath='{.data.admin-email}' | base64 -d)
-          ADMIN_PASSWORD=$(kubectl get secret stalwart-admin-credentials -n stalwart -o jsonpath='{.data.admin-password}' | base64 -d)
-
-          echo "Admin email: $ADMIN_EMAIL"
-
-          # Use kubectl exec to access Stalwart's admin API from localhost
-          # The HTTP listener allows localhost connections even when blocking other IPs
-          echo ""
-          echo "Accessing Stalwart admin API via kubectl exec..."
-
-          # Test API access first
-          echo "Testing API connectivity..."
-          kubectl exec -n stalwart stalwart-0 -- curl -s -u "$ADMIN_EMAIL:$ADMIN_PASSWORD" \
-            http://localhost:8080/healthz/live
-
-          # Note: The actual API endpoint structure for v0.16.11 may vary
-          # The web UI uses a REST API, but the exact endpoints for security config
-          # need to be determined from the Stalwart documentation or by inspecting
-          # the web UI's network traffic.
-
-          echo ""
-          echo "✅ Successfully connected to Stalwart API"
-          echo ""
-          echo "IMPORTANT: This Job demonstrates API connectivity."
-          echo "The actual security configuration change requires:"
-          echo "1. Identifying the correct API endpoint for security settings"
-          echo "2. Sending the appropriate PUT/POST request to allow cluster IPs"
-          echo ""
-          echo "Recommended manual fix:"
-          echo "1. Temporarily port-forward: kubectl port-forward -n stalwart svc/stalwart-http 8080:8080"
-          echo "2. Access https://mail.basicstack.de from your browser"
-          echo "3. Login with admin credentials"
-          echo "4. Navigate to Settings > Security"
-          echo "5. Disable IP filtering for the HTTP listener or add 10.244.0.0/16 to allowed IPs"
-
-          exit 0

apps/stalwart/stalwart-bootstrap-config.yaml

@@ -5,4 +5,7 @@ metadata:
   name: stalwart-bootstrap-config
   namespace: stalwart
 data:
-  config.json: '{"@type":"RocksDb","path":"/var/lib/stalwart"}'
+  # DEV-476: Bootstrap now points at the PostgreSQL config/data store.
+  # PGPASSWORD is injected into the stalwart container from
+  # secret/stalwart-postgres-credentials (see stalwart-fresh-deployment.yaml).
+  config.json: '{"@type":"PostgreSql","host":"stalwart-postgres","port":5432,"database":"stalwart","authUsername":"stalwart","authSecret":{"@type":"EnvironmentVariable","variableName":"PGPASSWORD"},"useTls":false}'

apps/stalwart/stalwart-config.yaml

@@ -10,12 +10,22 @@ data:
     # Stalwart Mail Server Configuration
     #
 
+    # DEV-476: primary store is PostgreSQL (see stalwart-bootstrap-config.yaml).
+    # Note: the container starts with `--config /etc/stalwart/config.json`, so
+    # this stalwart.toml is not read at runtime for a live pod — the bootstrap
+    # JSON is authoritative. Kept in sync here as documentation and for any
+    # one-shot tooling that references the toml.
     [store]
-    data = "rocksdb"
+    data = "postgres"
 
-    [store.rocksdb]
-    type = "rocksdb"
-    path = "/var/lib/stalwart"
+    [store.postgres]
+    type = "postgresql"
+    host = "stalwart-postgres"
+    port = 5432
+    database = "stalwart"
+    user = "stalwart"
+    password = "%{env:PGPASSWORD}%"
+    tls.enable = false
 
     #
     # Server Configuration

apps/stalwart/stalwart-fresh-deployment.yaml

@@ -122,7 +122,7 @@ spec:
           mountPath: /var/lib/stalwart
       containers:
       - name: stalwart
-        image: stalwartlabs/stalwart:v0.16.11
+        image: stalwartlabs/stalwart:v0.16.18
         ports:
         - containerPort: 25
           name: smtp
@@ -145,6 +145,12 @@ spec:
           value: "/etc/stalwart/certs/tls.crt"
         - name: TLS_PRIVATE_KEY
           value: "/etc/stalwart/certs/tls.key"
+        # DEV-476: bootstrap config.json resolves authSecret via this env var.
+        - name: PGPASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: stalwart-postgres-credentials
+              key: POSTGRES_PASSWORD
         volumeMounts:
         - name: data
           mountPath: /var/lib/stalwart
@@ -289,7 +295,7 @@ spec:
   jobTemplate:
     spec:
       backoffLimit: 2
-      activeDeadlineSeconds: 600  # 10 minute timeout
+      activeDeadlineSeconds: 1800  # 30 minute timeout (was 600s; retries + affinity settling can push past 10 min)
       template:
         metadata:
           labels:
@@ -297,9 +303,20 @@ spec:
         spec:
           serviceAccountName: stalwart-backup
           restartPolicy: OnFailure
+          # Co-locate with stalwart-0 so both pods share the same hcloud block volume
+          # attachment (RWO). Without this the backup pod can be scheduled on a
+          # different node and hits FailedAttachVolume/Multi-Attach (DEV-468).
+          affinity:
+            podAffinity:
+              requiredDuringSchedulingIgnoredDuringExecution:
+              - labelSelector:
+                  matchLabels:
+                    app: stalwart
+                    statefulset.kubernetes.io/pod-name: stalwart-0
+                topologyKey: kubernetes.io/hostname
           containers:
           - name: backup
-            image: alpine:3.19
+            image: alpine:3.24
             command:
             - /bin/sh
             - -c

docs/monitoring/restic-restore.md

@@ -0,0 +1,464 @@
+# Restic Restore Runbook — Hetzner Object Storage
+
+Author: CTO agent (`4b5f09a2-22d8-4e3d-9ac4-46d008ad1385`)
+Ticket: [DEV-488](/DEV/issues/DEV-488)
+Parent: [DEV-482](/DEV/issues/DEV-482) (Option 4 — restic → Hetzner OS)
+
+The monitoring stack's data-durability layer is a client-side-encrypted
+restic repo set stored in a single Hetzner Object Storage bucket. This
+document is the operator playbook for **restoring** those snapshots
+into a scratch namespace when we need to (a) recover from a real loss
+event, or (b) run the quarterly restore drill that keeps the
+[`backup-volumes` retirement plan](/DEV/issues/DEV-489) honest.
+
+## Repo layout
+
+Single bucket, four repo prefixes:
+
+| repo prefix                                    | source                                        | writer CronJob         | tag             |
+| ---------------------------------------------- | --------------------------------------------- | ---------------------- | --------------- |
+| `restic/loki`                                  | PVC `loki-storage-encrypted`                  | `backup-loki-restic`   | `loki`          |
+| `restic/grafana`                               | PVC `grafana-storage` (worker-2)              | `backup-grafana-restic`| `grafana`       |
+| `restic/k8s-resources`                         | `kubectl get -o yaml` (stdin)                 | `backup-k8s-resources` | `k8s-resources` |
+| `restic/prometheus`                            | PVC `prometheus-data-encrypted` (excl. WAL)   | `prometheus-backup`    | `prometheus`    |
+
+Password + endpoint + bucket + AWS creds live in `SealedSecret`
+`monitoring-s3-backup` (namespace `monitoring`). Keys:
+
+- `restic-password` — 32-byte random, generated in DEV-484, sealed
+  locally, mirrored to Passbolt entry `restic / monitoring backups`.
+- `access-key`, `secret-key` — Hetzner S3 credentials.
+- `endpoint` — e.g. `https://hel1.your-objectstorage.com`.
+- `bucket` — Hetzner bucket name (all four restic repos share it).
+
+Encryption is done **client-side by restic**. Hetzner Object Storage
+has no SSE-S3 / SSE-KMS ([FAQ](https://docs.hetzner.com/storage/object-storage/faq/general/));
+the CTO's [Hetzner Object Storage encryption memory](../../memory/hetzner-object-storage-encryption.md)
+covers the constraints.
+
+## Credentials projection
+
+Any restore pod must project **only** `monitoring-s3-backup` — never
+mount cluster admin creds into the drill namespace. The pattern below
+copies the secret cross-namespace once and lets restic project it
+read-only.
+
+```bash
+kubectl create namespace restore-drill
+kubectl -n monitoring get secret monitoring-s3-backup -o json |
+  jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,
+          .metadata.creationTimestamp,.metadata.ownerReferences,
+          .metadata.annotations,.status) |
+      .metadata.namespace="restore-drill"' |
+  kubectl apply -n restore-drill -f -
+```
+
+## Restore pod
+
+Throwaway pod pinned to the API-server node (no data mount, just the
+S3 secret). The `emptyDir` holds the restored tree; deleting the pod
+wipes it.
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: restore-drill
+  namespace: restore-drill
+  labels: { app: restore-drill }
+spec:
+  restartPolicy: Never
+  containers:
+    - name: restic
+      image: harbor.basicstack.de/library/restic:0.19.1  # matches CronJob image (DEV-493, bumped in DEV-541)
+      command: ["sleep", "3600"]
+      env:
+        - { name: AWS_ACCESS_KEY_ID,     valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: access-key } } }
+        - { name: AWS_SECRET_ACCESS_KEY, valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: secret-key } } }
+        - { name: RESTIC_PASSWORD,       valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: restic-password } } }
+        - { name: S3_ENDPOINT,           valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: endpoint } } }
+        - { name: S3_BUCKET,             valueFrom: { secretKeyRef: { name: monitoring-s3-backup, key: bucket } } }
+      volumeMounts:
+        - { name: work, mountPath: /work }
+  volumes:
+    - name: work
+      emptyDir: {}
+```
+
+Apply it, then `kubectl -n restore-drill exec restore-drill -- <cmd>`
+for everything below.
+
+## Per-repo restore commands
+
+`RESTIC_REPOSITORY` is set inline per repo to keep repo scope explicit.
+All commands run inside the scratch pod.
+
+### Loki
+
+```sh
+export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki"
+restic snapshots
+restic restore latest --target /work/loki --tag loki
+find /work/loki/source -type f
+```
+
+Restored tree lands under `/work/loki/source/…` because the backup
+source was `/source` inside the CronJob container.
+
+### K8s resources
+
+```sh
+export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/k8s-resources"
+restic snapshots
+restic restore latest --target /work/k8s --tag k8s-resources
+ls -lh /work/k8s/cluster.yaml
+```
+
+`cluster.yaml` is the concatenated dump written by the
+`backup-k8s-resources` CronJob's `kubectl-dump` init container.
+
+### Grafana
+
+```sh
+export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana"
+restic snapshots
+restic restore latest --target /work/grafana --tag grafana
+```
+
+Follows the loki pattern (restored tree under `/work/grafana/source/`).
+
+### Prometheus
+
+```sh
+export RESTIC_REPOSITORY="s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus"
+restic snapshots
+restic restore latest --target /work/prometheus --tag prometheus
+find /work/prometheus/source -maxdepth 1 -type d | head
+```
+
+Restored tree lands under `/work/prometheus/source/…`. **WAL and
+`chunks_head/` are excluded on purpose** — see the "compaction race"
+notes at the top of `apps/monitoring/prometheus-backup-cronjob.yaml`.
+Expect the last ~15 s of ingested samples to be lost on recovery; the
+compacted 2h/24h blocks are complete.
+
+**Integrity check — `promtool tsdb analyze` (required for the DEV-492
+drill).** The snapshot must load cleanly through Prometheus's own
+verifier before we call the drill green.
+
+```sh
+kubectl -n restore-drill run promtool-check --rm -it --restart=Never \
+  --image=prom/prometheus:v2.53.1 \
+  --overrides='{"spec":{"containers":[{"name":"promtool-check","image":"prom/prometheus:v2.53.1","command":["sh","-c","for b in /work/prometheus/source/*/; do echo \"--- $b\"; promtool tsdb analyze /work/prometheus/source \"$(basename $b)\" || exit 1; done"],"volumeMounts":[{"name":"work","mountPath":"/work"}]}],"volumes":[{"name":"work","emptyDir":{}}]}}'
+```
+
+Simpler drill path (the restore pod already has restic; `promtool` is
+not shipped in `restic/restic`, so run it from a `prom/prometheus`
+sidecar or an interactive pod that mounts the same `emptyDir` — see
+DEV-492 drill entry for the concrete two-pod recipe used on the
+first run).
+
+Exit 0 on every block = pass. A non-zero exit on any block means the
+compaction-race mitigation slipped and the snapshot is corrupt.
+Re-run the CronJob and re-drill.
+
+## Integrity checks
+
+Two layers.
+
+**Repository self-check** — cryptographic integrity of packs and
+snapshots, no source needed:
+
+```sh
+restic check                       # metadata only
+restic check --read-data-subset=5% # sample-read + decrypt 5% of packs
+```
+
+The CronJobs run `--read-data-subset=5%` on every write. A restore
+drill should re-run at minimum `restic check` end-to-end.
+
+**Source parity** — sha256 the restored tree against a fresh readout
+of the live PVC. For loki, exec into the live pod:
+
+```sh
+LOKI=$(kubectl -n monitoring get pods -l app=loki -o jsonpath='{.items[0].metadata.name}')
+kubectl -n monitoring exec "$LOKI" -- sh -c '
+  cd /loki && find . -type f ! -path "./lost+found/*" -exec sha256sum {} +' \
+  > /tmp/live.sha256
+kubectl -n restore-drill exec restore-drill -- sh -c '
+  cd /work/loki/source && find . -type f -exec sha256sum {} +' \
+  > /tmp/restored.sha256
+diff <(sort /tmp/live.sha256) <(sort /tmp/restored.sha256)
+```
+
+Empty diff = parity. Any drift is expected only on files that Loki is
+actively writing (WAL segments) — capture the snapshot when Loki is
+quiet, or accept a WAL delta and re-hash the boltdb-shipper /
+chunks trees only.
+
+For k8s-resources, parity is meaningless (a live dump varies by the
+second); rely on `restic check` and on being able to `grep` the
+restored `cluster.yaml` for expected namespaces / secrets.
+
+## Cleanup
+
+```sh
+kubectl -n restore-drill delete pod restore-drill --wait
+kubectl delete namespace restore-drill
+```
+
+Deleting the namespace tears down the projected secret. The `emptyDir`
+lives in the pod's ephemeral scratch on the node; the sleep container
+never persists anything outside `/work`.
+
+## Rotation and disaster recovery
+
+If the `restic-password` is lost:
+
+1. Recover the plaintext from Passbolt (`restic / monitoring backups`).
+2. If Passbolt is also lost, the data is unrecoverable by design —
+   client-side encryption with a lost key cannot be reversed.
+
+To rotate:
+
+```sh
+# on any workstation with kubectl + kubeseal + the restic password
+NEW=$(openssl rand -base64 32)
+restic key add                       # opens editor for new password;
+                                     # paste NEW and confirm
+restic key list
+restic key remove <old-id>
+# then re-seal the SealedSecret with NEW, redeploy, Argo syncs.
+```
+
+Rotate annually or immediately on suspected compromise.
+
+## Restic image (Harbor mirror)
+
+All three restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
+`backup-k8s-resources`) reference a Harbor-hosted copy of upstream to
+keep the backup pipeline off the Docker Hub pull path and immune to
+upstream retagging:
+
+- Manifest reference: `harbor.basicstack.de/library/restic:0.19.1`
+- Upstream: `docker.io/restic/restic:0.19.1`
+- Ticket: [DEV-493](/DEV/issues/DEV-493) (bumped to 0.19.1 in [DEV-541](/DEV/issues/DEV-541))
+
+The `library` project is public, so no `imagePullSecret` is required
+on the CronJob pods.
+
+### Tag-bump procedure
+
+Run this when we want to move restic to a new pinned tag (e.g. 0.17.3
+→ 0.17.4). Do the mirror push **before** editing manifests so Argo
+cannot roll pods onto an unmirrored tag.
+
+1. **Pick and verify the upstream tag.** Confirm the tag exists on
+   Docker Hub and (ideally) read the upstream restic release notes for
+   breaking changes:
+
+   ```sh
+   curl -s "https://hub.docker.com/v2/repositories/restic/restic/tags/0.17.4"
+   ```
+
+2. **Mirror the image to Harbor** with a one-shot in-cluster crane
+   Job. Requires an admin (or `library`-scoped robot) Harbor
+   credential; the Job auth secret is throwaway.
+
+   ```sh
+   # from the CTO workstation with kubectl + Harbor admin access
+   NEW_TAG=0.17.4
+   ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
+     -o jsonpath='{.data.harborAdminPassword}' | base64 -d)
+   AUTH=$(printf 'admin:%s' "$ADMIN_PW" | base64 -w0)
+   cat >/tmp/dc.json <<JSON
+   {"auths":{"harbor.basicstack.de":{"auth":"$AUTH"}}}
+   JSON
+   kubectl -n harbor create secret generic crane-mirror-cfg \
+     --from-file=config.json=/tmp/dc.json \
+     --dry-run=client -o yaml | kubectl apply -f -
+   rm /tmp/dc.json
+
+   cat <<YAML | kubectl apply -f -
+   apiVersion: batch/v1
+   kind: Job
+   metadata:
+     name: crane-mirror-restic-${NEW_TAG//./-}
+     namespace: harbor
+   spec:
+     backoffLimit: 1
+     ttlSecondsAfterFinished: 600
+     template:
+       spec:
+         restartPolicy: Never
+         containers:
+           - name: crane
+             image: gcr.io/go-containerregistry/crane:v0.21.9
+             env: [{name: DOCKER_CONFIG, value: /docker}]
+             args:
+               - copy
+               - docker.io/restic/restic:${NEW_TAG}
+               - harbor.basicstack.de/library/restic:${NEW_TAG}
+             volumeMounts:
+               - {name: docker-config, mountPath: /docker, readOnly: true}
+         volumes:
+           - name: docker-config
+             secret: {secretName: crane-mirror-cfg}
+   YAML
+
+   kubectl -n harbor wait --for=condition=complete \
+     job/crane-mirror-restic-${NEW_TAG//./-} --timeout=5m
+   kubectl -n harbor logs job/crane-mirror-restic-${NEW_TAG//./-} | tail -20
+   ```
+
+3. **Sanity-check the artifact** — the Harbor digest must match the
+   digest crane just pushed, and an anonymous pull must resolve:
+
+   ```sh
+   ADMIN_PW=$(kubectl -n harbor get secret harbor-secrets \
+     -o jsonpath='{.data.harborAdminPassword}' | base64 -d)
+   curl -sk -u "admin:$ADMIN_PW" \
+     "https://harbor.basicstack.de/api/v2.0/projects/library/repositories/restic/artifacts" \
+     | jq -r '.[] | "digest=\(.digest) tags=\((.tags//[])|map(.name)|join(","))"'
+   ```
+
+4. **Clean up the mirror secret and Job** (the Job also TTLs itself
+   in 10 min):
+
+   ```sh
+   kubectl -n harbor delete secret crane-mirror-cfg
+   kubectl -n harbor delete job crane-mirror-restic-${NEW_TAG//./-} \
+     --ignore-not-found
+   ```
+
+5. **Bump the manifest.** Update the three files under `apps/monitoring/`
+   in the `basicstack-repo`:
+
+   - `backup-loki-restic-cronjob.yaml`
+   - `backup-grafana-restic-cronjob.yaml`
+   - `backup-k8s-resources-cronjob.yaml`
+
+   Each references `harbor.basicstack.de/library/restic:<tag>` under
+   the `restic` container. Bump `<tag>` in all three, commit, push.
+
+6. **Let Argo sync** — the `monitoring` Application picks up the new
+   manifests. Watch for successful reconciliation:
+
+   ```sh
+   kubectl -n argocd get application monitoring \
+     -o jsonpath='{.status.sync.status}{"  "}{.status.health.status}{"\n"}'
+   ```
+
+7. **Trigger one CronJob run to prove the Harbor pull is green.**
+   Create a manual Job from any of the CronJobs (loki is fine) and
+   inspect its logs:
+
+   ```sh
+   kubectl -n monitoring create job --from=cronjob/backup-loki-restic \
+     smoke-restic-$(date +%Y%m%d-%H%M%S)
+   ```
+
+   The Pod should pull from `harbor.basicstack.de/library/restic:<tag>`
+   (`kubectl describe pod ...` → Events → `Pulling image …`) and
+   finish with `restic backup /source` output ending in a snapshot id.
+
+8. **Update this doc's "restore pod" template** if the new tag is
+   incompatible with the existing restore pod command — the drill pod
+   image must match the CronJob image, otherwise the restore-drill
+   won't round-trip.
+
+**Rollback:** if the new tag misbehaves, edit the three manifests back
+to the last-known-good tag, commit, and let Argo re-sync. The old
+tag's manifest remains in Harbor until it's explicitly deleted, so
+rollback is a manifest change only.
+
+## Drill log
+
+Every drill appends to this section. Include:
+
+- date (UTC), operator, snapshot IDs restored, parity result,
+  restic check result, cleanup confirmation.
+
+### 2026-08-16 — First drill (DEV-488)
+
+- Operator: CTO agent, run [DEV-488](/DEV/issues/DEV-488).
+- Cluster: k3s at `178.105.17.239` (Hetzner).
+- Snapshots:
+  - `restic/loki`: `97092888` (2026-08-16 15:44:21Z), parent `98c6fee6`
+    (2026-08-16 15:26:10Z). Both tag=`loki`, host=`k3s`, paths=`/source`.
+  - `restic/k8s-resources`: `9b0155cf` (2026-08-16 15:40:28Z),
+    tag=`k8s-resources`, paths=`/cluster.yaml`.
+- Restore output:
+  - loki: `Restored 14 files/dirs (292 B) in 0:00` — matches live PVC
+    which is nearly empty (fresh Loki, no ingested chunks yet).
+  - k8s-resources: `Restored 1 files/dirs (11.167 MiB) in 0:00` —
+    `cluster.yaml` = 11 709 295 bytes, sha256
+    `5aa93f7326b5cbd2408cb763395bba259de6cc0f99b0c8e1bca4421edc7931c0`.
+- Parity — loki restored tree vs. live loki-storage-encrypted PVC:
+
+  | file                                     | restored sha256                                                    | live sha256 | match |
+  | ---------------------------------------- | ------------------------------------------------------------------ | ----------- | ----- |
+  | `boltdb-shipper-active/uploader/name`    | `51215284bd61bf79d48ccd1aec59445bfa28dec7b2cdce4d9476bc0b61342813` | identical   | yes   |
+  | `chunks/loki_cluster_seed.json`          | `8d214a355d0057586e1c142d5f5d2658ccec39bd66e193b37dc12c21d6d7edd5` | identical   | yes   |
+  | `wal/checkpoint.019215/00000000`         | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | identical   | yes   |
+  | `wal/00019216`                           | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | identical   | yes   |
+
+- k8s-resources sanity: `cluster.yaml` opens with a valid v1
+  Namespace list including `argocd`, `basicstack-web`, `monitoring`
+  etc. No parse errors.
+- `restic check --read-data-subset=5%` was run by the writing
+  CronJob on both repos; drill relied on that. Re-running end-to-end
+  `restic check` is a follow-up for the quarterly drill.
+- Cleanup: pod + `restore-drill` namespace deleted; scratch tree
+  discarded with the emptyDir.
+
+**Outcome: PASS** for `restic/loki` and `restic/k8s-resources`.
+
+Next drill target: 2026-11-16 (quarterly). Include `restic/grafana`
+once its first CronJob run has produced ≥ 1 snapshot.
+
+### 2026-08-16 — Prometheus migration drill (DEV-492)
+
+Migrated `prometheus-backup` from `rclone sync` (plaintext at rest) to
+restic (client-side encrypted). Same-day full-cycle drill against the
+freshly-populated `restic/prometheus` repo.
+
+- Operator: CTO agent, run [DEV-492](/DEV/issues/DEV-492).
+- Repo: `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, initialised
+  by first CronJob run (repo id `f0eda506fd`).
+- Backup runs (manual, from the deployed CronJob):
+  - `71420465` (2026-08-16 16:16:35Z) — first run, cold repo,
+    `restic backup` walk 5:10, 85 files 42 dirs, 8.017 GiB added
+    (2.441 GiB stored after compression + dedup), exit 0. `restic
+    check --read-data-subset=5%`: no errors.
+  - `825f983a` (2026-08-16 16:23:56Z) — second run, warm repo, 17 s
+    total (no new blocks — dedup 100 %), `restic check` clean.
+- Drill: `harbor.basicstack.de/library/restic:0.17.3` (restore) +
+  `prom/prometheus:v2.53.1` (promtool) sidecars sharing an `emptyDir`
+  in namespace `restore-drill-dev492`.
+  - `restic restore latest --tag prometheus --target /work`:
+    `Restored 127 files/dirs (8.069 GiB) in 0:34`.
+  - Excluded paths verified absent on restored tree: `wal/`,
+    `chunks_head/`, `lost+found/` restored as empty dirs; `lock` and
+    `queries.active` not present at all.
+  - `promtool tsdb list /work/source` — 19 blocks parsed, all with
+    valid ULID + duration + samples/chunks/series counts.
+  - `promtool tsdb analyze /work/source <block>` — ran against every
+    block dir; 0 failed. Summary lines validated non-zero `Total
+    Series` and reasonable `Duration` (2h / 18h / 54h buckets, matching
+    Prometheus's compaction schedule).
+  - End-to-end `restic check`: `no errors were found` (metadata pass;
+    on the freshly-initialised repo `--read-data-subset=5%` was
+    already exercised by the writing CronJob).
+- Cleanup: `restore-drill-prom` pod + `restore-drill-dev492` namespace
+  deleted; scratch tree discarded with the `emptyDir`. Manual
+  `prom-restic-manual-{01,02}` jobs deleted from `monitoring`.
+
+**Outcome: PASS** for `restic/prometheus`. All Definition-of-Done
+items on [DEV-492](/DEV/issues/DEV-492) satisfied — backup encrypted
+at rest, `promtool tsdb analyze` clean, docs updated, restic password
+handling documented (shared `monitoring-s3-backup` SealedSecret,
+rotation procedure covers all four repos).
+
+Next drill target for `restic/prometheus`: 2026-11-16 alongside the
+other three repos.

infrastructure/CP1_UPDATE_PROCEDURE.md

@@ -0,0 +1,14 @@
+# k3s-cp-1 OS Update Procedure — SUPERSEDED
+
+**Superseded by:** [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) (DEV-515, 2026-08-23).
+
+**Why:** this document was written when cp-1 was the **sole** control-plane node and the datastore was embedded SQLite via kine. After [DEV-510](/DEV/issues/DEV-510) the cluster runs HA embedded etcd across `k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`, and all three CPs are `NoSchedule`-tainted (they host no StatefulSets or single-replica Deployments anymore). The single-CP kine cascade risk, the batched stateful eviction dance, and the external-`/livez` polling pattern no longer describe the current cluster.
+
+The generalized procedure covers cp-1 and cp-2 and cp-3. Use it.
+
+**Scripts:**
+
+- `infrastructure/scripts/os-update/update-cp-node.sh <node>` — the HA-aware entry point.
+- `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compatibility with any existing runbook that names it explicitly.
+
+**History note:** the original phase design (add swap → preflight → batched stateful eviction → drain → apt → reboot → finalize) is preserved in git history at commit `19af691` and `989033d` for future reference. The new procedure retains Phases A/B/D/E/F essentially unchanged and drops Phase C (batched stateful eviction) as obsolete.

infrastructure/CP_UPDATE_PROCEDURE.md

@@ -0,0 +1,298 @@
+# HA k3s Control-Plane OS Update Procedure
+
+**Purpose:** apply Ubuntu OS updates (kernel, security, apt) to any control-plane node in the HA k3s cluster (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) without losing etcd quorum and without a longer api-server outage than a normal reboot.
+
+**Audience:** the CTO agent, or an operator with root SSH to the cluster. Execution is board-approval-gated — see `## Approval gates` below.
+
+**Supersedes:** the earlier single-CP `CP1_UPDATE_PROCEDURE.md` (kept as a redirect stub). The cp-1-only procedure was written when cp-1 was the sole api-server and the datastore was embedded SQLite via kine. After [DEV-510](/DEV/issues/DEV-510) (2026-08-22), the cluster runs HA embedded etcd across cp-1/cp-2/cp-3 and all three CPs are `NoSchedule`-tainted, so:
+
+- **The kine cascade is no longer the driving risk.** The datastore is etcd 3.6.12 with 3-node quorum; a single CP drain no longer leaves a lone SQLite writer. The kubectl-latency guardrail stays as a sanity check but is now a much softer signal.
+- **CPs host no StatefulSet or single-replica Deployment workloads.** The pre-drain stateful eviction dance from the old cp-1 doc no longer applies — CPs only host DaemonSets + `metrics-server` (currently on cp-2) + occasional CronJob completions.
+- **The api-server is not lost on reboot.** During a CP reboot the other two apiservers keep serving. `/livez` is polled from the *other* apiservers, not from an external machine forced to poll the one being rebooted.
+
+---
+
+## Scope
+
+**In scope**
+
+- Add a durable ≥ 2 GiB swapfile on any CP that has zero swap (idempotent one-off — `--add-swap`).
+- k3s etcd snapshot as the restore point.
+- `kubectl cordon` + `kubectl drain` on the target CP.
+- `apt-get update/upgrade/dist-upgrade/autoremove` on the target CP.
+- Controlled reboot with etcd-quorum-aware liveness monitoring.
+- Uncordon + cluster health verify.
+
+**Out of scope — do NOT do here**
+
+- Any change to `/etc/rancher/k3s/*`, `/etc/systemd/system/k3s*.service*`, or the k3s binary version. k3s upgrades go through system-upgrade-controller (see `K3S_OPERATIONS.md`).
+- Any change to manifests under `apps/`, `infrastructure/`, or applied via ArgoCD.
+- Deleting PVs / PVCs. Any pod that gets rescheduled off the target CP stays on its new node.
+- Fixing application-level problems.
+- Rebuilding the node — if a CP does not return after reboot, escalate; the rebuild path is `ADD_WORKER_NODE.md` plus board approval, not this document.
+
+---
+
+## Approval gates
+
+This procedure has **two** independent gates. Neither happens without explicit board approval on the corresponding Paperclip issue:
+
+1. **Add swap (Phase A).** Non-invasive, non-state-mutating, kubelet already runs with `failSwapOn=false`. Requires board approval per stateful-service safety rules because it modifies a CP node.
+2. **Full OS update (Phases B–E).** Requires board approval because it drains + reboots one of the etcd members. Do NOT execute without an explicit `request_board_approval` acceptance on the execution ticket.
+
+Both gates are independent — swap can (and should) be added first, in a quiet window, before the full update is scheduled.
+
+---
+
+## CP ordering rule (multiple CPs in one cycle)
+
+- **One at a time. Never two CPs cordoned or draining at once.** Two of three CPs down = etcd quorum loss = api-server unavailability for the whole cluster.
+- **Leader last.** Query the current etcd leader before starting; update the two followers first (in any order), then the leader. Rationale: draining a follower is a no-op for the raft leader; draining the leader forces a re-election. Doing followers first minimises leader flapping.
+- **Health gate between CPs.** After each CP finishes Phase F (finalize) and cluster health is green, wait at least `POST_UNCORDON_WAIT_SECONDS` (default 180 s) and re-check etcd endpoint status before touching the next CP. This gives etcd time to fully re-sync the just-rebooted member.
+- **Halt on any yellow.** Any of the following aborts the cycle at the current CP (do NOT proceed to the next CP): etcd reports a member as not `started`, `kubectl get nodes` > 5 s, any node not `Ready`, any workload deployment/sts under desired replicas.
+
+Query the leader:
+
+```bash
+ssh root@$CP1_HOST bash -c '
+ETCDCTL_API=3 etcdctl \
+  --endpoints=https://127.0.0.1:2379 \
+  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
+  --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
+  --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
+  endpoint status --cluster -w table'
+```
+
+---
+
+## Current CP workload (snapshot 2026-08-23)
+
+Re-derive live before any execution — this table is a design reference only.
+
+| Node | Kernel | Swap | StatefulSets | Single-replica Deployments | DaemonSets |
+|------|--------|------|--------------|-----------------------------|------------|
+| k3s-cp-1 | 6.8.0-137 | **0 B** | 0 | 0 | node-exporter, promtail, hcloud-csi-node, svclb-* |
+| k3s-cp-2 | 6.8.0-137 | **0 B** | 0 | metrics-server (evictable) | node-exporter, promtail, hcloud-csi-node, svclb-* |
+| k3s-cp-3 | 6.8.0-137 | **0 B** | 0 | 0 | node-exporter, promtail, hcloud-csi-node, svclb-* |
+
+etcd datastore live snapshot: 38 MiB per member, all three `started`, cp-3 is current leader.
+
+Because CPs are `NoSchedule`-tainted, no workload will end up on them via normal scheduling — the pre-drain stateful redistribution loop from the old cp-1 procedure is obsolete.
+
+---
+
+## Phase A — Add swap (one-off, idempotent)
+
+**Goal:** eliminate the "3.7 GiB, no swap" underlying constraint on any CP before ever draining it. All three CPs currently have zero swap; add swap once per CP.
+
+**Preconditions:**
+
+- kubelet in this k3s already runs with `failSwapOn=false` (confirmed on cp-1 via `/api/v1/nodes/k3s-cp-1/proxy/configz`) — enabling swap does NOT break the kubelet.
+- Target CP's `/` has ≥ `2 * SWAP_SIZE_MB` free.
+- Board approval on the swap-add ticket.
+
+**Sizing:** default **4 GiB** swap. Rationale — cp-1 baseline (k3s + hosted daemons) sits at ~2 GiB used; 4 GiB swap gives us headroom for the drain-eviction transient without inflating disk usage past ~10 % of `/`. Minimum acceptable per guardrail: 2 GiB.
+
+**Location:** `/swapfile` (root filesystem). Not a separate partition — reversible, no LVM changes.
+
+**Steps (encoded in `update-cp-node.sh <node> --add-swap`):** identical to the old cp-1 procedure — `fallocate` → `mkswap` → `swapon` → `/etc/fstab` → `vm.swappiness=10` in `/etc/sysctl.d/99-k3s-swap.conf`. Idempotent: skips if a swapfile of the target size is already active. Full script source in `scripts/os-update/update-cp-node.sh`.
+
+**Phase A also installs `etcd-client`** on the target CP if `etcdctl` is missing. This is a hard requirement for the etcd-quorum probes used in Phase B/E when the CP is later itself the update target and one of the *other* CPs must query etcd cluster status. cp-1 already has `etcdctl` (installed pre-HA); cp-2 and cp-3 pick it up here.
+
+**Rollback for Phase A:** `swapoff /swapfile && rm /swapfile` and remove the fstab line. Safe at any time — swap is a soft resource.
+
+**Verification after Phase A:**
+
+- `free -h` shows `Swap: 4.0Gi` used ≈ 0.
+- `swapon --show` shows `/swapfile 4G`.
+- `sysctl vm.swappiness` returns `10`.
+- kubelet still Ready (`kubectl get node <node>`).
+- No new `MemoryPressure` condition.
+
+---
+
+## Phase B — Preflight
+
+Runs from the operator machine (or from any healthy CP). Every command's stdout+stderr goes to `/tmp/os-update-<node>-<UTC-timestamp>.log` on the operator machine. Attach that log to the Paperclip execution ticket at the end.
+
+Run `update-cp-node.sh <node> --preflight`. Checks:
+
+1. **Cluster is currently healthy:** `cluster-health.sh` returns 0.
+2. **Target node is a CP:** `kubectl get node <node>` carries the `node-role.kubernetes.io/control-plane` label.
+3. **All 3 CPs are healthy etcd members:** `etcdctl endpoint status --cluster` shows every member `started`, no errors.
+4. **Only one CP is being updated this cycle:** no other CP is currently cordoned.
+5. **Target CP has swap on:** abort if `free -h` shows `Swap: 0B`. Run `--add-swap` first.
+6. **kubectl latency probe:** `time kubectl get nodes` returns in ≤ 5 s.
+7. **etcd snapshot** (restore point): `k3s etcd-snapshot save --name pre-cp-os-update-<node>-<UTC-timestamp>` executed on the target CP.
+8. **Records the current etcd leader** — if the target *is* the leader, prints a warning ("prefer updating a follower first"). Does NOT auto-swap the target; the operator/agent makes the call per the CP ordering rule.
+
+If any preflight check fails: STOP. Do not proceed.
+
+---
+
+## Phase C — Cordon + drain the target CP
+
+CPs are `NoSchedule`-tainted, so cordon is mostly a belt-and-suspenders measure. Drain evicts the small tail of not-DaemonSet workloads (`metrics-server` currently lives on cp-2).
+
+```bash
+kubectl cordon <node>
+kubectl drain <node> \
+  --ignore-daemonsets \
+  --delete-emptydir-data \
+  --timeout="${DRAIN_TIMEOUT_SECONDS:-600}s"
+```
+
+If drain reports a PDB block: do NOT `--force`. Uncordon the node, mark the run `blocked` on the PDB, and escalate. This is a workload PDB bug — fix separately.
+
+**Kine-cascade guardrail (softened for HA etcd):** during drain, `time kubectl get nodes` should still return in ≤ 5 s. Because we now have 3-member etcd (not lone SQLite kine), a single-CP drain does not create the write-amplification hazard from [DEV-495](/DEV/issues/DEV-495). But `> 5 s` still indicates something is wrong (etcd slow disk, leader flapping) — halt and investigate.
+
+---
+
+## Phase D — apt on the target CP
+
+Identical to the worker-node apt block (`update-node.sh` step 3). `update-cp-node.sh <node> --apt` runs this via `ssh root@<node> bash -s`:
+
+```bash
+export DEBIAN_FRONTEND=noninteractive
+APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
+uname -r > /root/pre-apt-kernel
+dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv
+
+if dpkg --audit | grep -qE .; then dpkg --configure -a || true; fi
+apt-get update
+apt-get $APT_OPTS upgrade || { apt-get $APT_OPTS -f install; apt-get $APT_OPTS upgrade; }
+apt-get $APT_OPTS dist-upgrade
+apt-get $APT_OPTS autoremove --purge
+apt-get clean
+[ -f /var/run/reboot-required ] && echo REBOOT_REQUIRED=yes || echo REBOOT_REQUIRED=no
+```
+
+`pre-apt-kernel` and `pre-apt-packages.tsv` are the local rollback references — see Rollback below.
+
+---
+
+## Phase E — Reboot handling (etcd-quorum-aware)
+
+**Duration expectation:** 90–180 s of *this member's* api-server unavailability. The other two apiservers keep serving; the operator machine's kubectl continues to work via one of them.
+
+**Escalation trigger:** rebooted apiserver not back on `/livez` after **10 minutes** → escalate. First check the Hetzner console via `hcloud server describe <node>` for boot state; if kernel-panic / initramfs, use grub previous-kernel path (see Rollback). Do NOT rebuild the node.
+
+**Steps (executed by `update-cp-node.sh <node> --reboot`):**
+
+```bash
+# 1. Reboot the target CP (ssh will hang up mid-command — expected).
+ssh $SSH_OPTS root@$TARGET_HOST 'systemctl reboot' || true
+sleep 15
+
+# 2. Poll the TARGET's /livez from the operator machine. It is served on port 6443.
+#    Insecure (`-k`) because the server cert is self-signed by k3s.
+deadline=$(( $(date +%s) + ${REBOOT_MAX_WAIT_SECONDS:-600} ))
+while [ $(date +%s) -lt $deadline ]; do
+  code=$(curl -sk -o /dev/null -w '%{http_code}' https://$TARGET_HOST:6443/livez 2>/dev/null || echo 000)
+  [ "$code" = "200" ] && break
+  sleep 5
+done
+
+# 3. Verify etcd membership from a peer CP — the target should be back as `started`.
+ssh $SSH_OPTS root@$PEER_HOST '
+  ETCDCTL_API=3 etcdctl \
+    --endpoints=https://127.0.0.1:2379 \
+    --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
+    --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
+    --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
+    endpoint status --cluster -w table'
+
+# 4. Wait for kubelet Ready on the target from the api-server view.
+deadline=$(( $(date +%s) + 300 ))
+while [ $(date +%s) -lt $deadline ]; do
+  ready=$(kubectl get node "$TARGET_NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
+  [ "$ready" = "True" ] && break
+  sleep 5
+done
+```
+
+**What to do if the api-server takes 3–10 minutes:** it may be normal for a large etcd on a slow disk. Wait it out. Do NOT restart k3s to "help" — an etcd member catching up on raft log is stateful and interrupting it can slow the whole cluster.
+
+**What to do if it takes > 10 minutes:** escalate; run the Hetzner-console diagnosis; if the console shows a bootable Ubuntu but the k3s service is failing, that's the boundary — this procedure stops here. Follow `K3S_OPERATIONS.md` for the k3s recovery path.
+
+---
+
+## Phase F — Uncordon + verify + finalize
+
+```bash
+kubectl uncordon <node>
+sleep "${POST_UNCORDON_WAIT_SECONDS:-180}"
+RETRY_ON_TRANSIENT=1 infrastructure/scripts/os-update/cluster-health.sh
+ssh root@<node> 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
+```
+
+Between CPs (when multiple CPs will be updated in the same cycle), also verify:
+
+```bash
+ssh root@<peer-cp> '
+  ETCDCTL_API=3 etcdctl \
+    --endpoints=https://127.0.0.1:2379 \
+    --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
+    --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
+    --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
+    endpoint status --cluster -w table'
+```
+
+All three members must be `started`, no errors, DB sizes within 10 % of each other.
+
+Attach the `/tmp/os-update-<node>-<ts>.log` to the execution ticket. Mark the ticket `done` on green health, or `blocked` naming the specific residual issue.
+
+---
+
+## Rollback / recovery
+
+### If apt broke a package
+- On the target CP, `dpkg --audit` to find half-configured packages.
+- `dpkg --configure -a`, then `apt-get -f install`.
+- If a specific package broke and you know the previous version from `/root/pre-apt-packages.tsv`, `apt-get install <pkg>=<old-version>`.
+
+### If the new kernel does not boot
+- Hetzner cloud console: `hcloud server request-console <node>` → get VNC URL, watch boot.
+- If grub is up, select the previous-kernel entry. Ubuntu keeps ≥ 1 old kernel installed by default.
+- Once booted on the old kernel, `apt-get remove` the broken kernel and pin the working one:
+  ```bash
+  apt-mark hold linux-image-<broken-version> linux-headers-<broken-version>
+  ```
+- Escalate on the ticket regardless — a kernel rollback is a follow-up investigation, not a "done" outcome.
+
+### If the node does not return at all
+- Do NOT `hcloud server delete`. Do NOT re-provision.
+- Escalate to the board with the Hetzner console output and `/tmp/os-update-<node>-*.log`.
+- The `--cluster-reset --cluster-reset-restore-path=<snapshot>` recovery path exists (see `K3S_OPERATIONS.md`) but requires board approval per stateful-service safety rules. The pre-flight snapshot from Phase B is the restore point.
+- **HA-etcd advantage:** if one CP is permanently lost but the other two are healthy, you have quorum. Do not rush a `cluster-reset`. Follow the "member remove + re-add" path in the k3s docs first.
+
+### If etcd quorum degrades mid-cycle
+- Two of three CPs down = write outage until quorum returns. Do NOT attempt to update the third.
+- Uncordon all CPs, restore the just-updated one first, and confirm all three members `started` before doing anything else.
+
+---
+
+## Automation entry points
+
+- `infrastructure/scripts/os-update/update-cp-node.sh <node> [phase-flag]` — the generalized HA-aware flow.
+  - `--add-swap` — Phase A only (idempotent, safe standalone).
+  - `--dry-run` — walk all phases printing what would be done, no state change.
+  - `--preflight` — Phase B only.
+  - `--drain` — Phase C only (cordon + drain).
+  - `--apt` — Phase D apt commands only.
+  - `--reboot` — Phase E only.
+  - `--finalize` — Phase F only.
+  - `--run` — all phases with a confirmation between each unless `ASSUME_YES=1`.
+
+- `infrastructure/scripts/os-update/update-cp-1.sh` — retained as a thin wrapper that calls `update-cp-node.sh k3s-cp-1 "$@"`. Historical callers keep working.
+
+The script follows the same log-dir contract as `update-node.sh` (`/tmp/os-update-<node>-<UTC-timestamp>.log`).
+
+---
+
+## Change history
+
+| Date | Change | By |
+|------|--------|-----|
+| 2026-08-16 | Initial cp-1-only procedure (DEV-496) | CTO agent |
+| 2026-08-23 | Generalized to HA (cp-1/cp-2/cp-3); dropped Phase C batched stateful eviction (CPs are NoSchedule-tainted now); softened kine cascade guardrail (embedded etcd 3-member quorum); added CP ordering rule (leader last, one at a time); introduced `update-cp-node.sh` (DEV-515) | CTO agent |

infrastructure/OS_UPDATE_PROCEDURE.md

@@ -22,7 +22,9 @@
 
 | Role | Node | Private IP | Public IP | Datacenter | Notes |
 |------|------|-----------|-----------|------------|-------|
-| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | update LAST |
+| control-plane | k3s-cp-1 | 10.42.1.1 | 178.105.17.239 | fsn1 | EXCLUDED from `os-update.sh` — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
+| control-plane | k3s-cp-2 | — | 188.245.85.199 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
+| control-plane | k3s-cp-3 | — | 49.13.92.162 | fsn1 | EXCLUDED — see [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) |
 | worker | k3s-worker-1 | 10.42.1.2 | (via CP) | fsn1 | |
 | worker | k3s-worker-2 | 10.42.1.3 | (via CP) | fsn1 | |
 | worker | k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | fsn1 | |
@@ -36,16 +38,31 @@ Always re-derive the live list before running — nodes may have been added/remo
 ssh root@178.105.17.239 'kubectl get nodes -o wide'
 ```
 
-**Order rule:** update ALL workers first, control plane LAST. Within workers, update in this order to protect stateful workloads:
+**Order rule:** update workers only. Within workers, update in this order to protect stateful workloads:
 1. runner + workers that host no PVs (safest — lowest disruption)
 2. remaining workers
 3. **Stalwart-hosting fsn1 workers last among workers** — Stalwart has hard fsn1 affinity, so draining a fsn1 worker while another fsn1 worker is also unavailable can leave Stalwart Pending. Never have two fsn1 workers cordoned/down at the same time.
-4. **k3s-cp-1 last** — single control plane; the API server goes away during its reboot.
+4. **ALL control-plane nodes excluded** — `os-update.sh` selects nodes without the `node-role.kubernetes.io/control-plane` label, so `k3s-cp-1`, `k3s-cp-2`, and `k3s-cp-3` are all skipped automatically. CPs have their own procedure ([`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) — HA-aware, one CP at a time, leader last).
 
 **Concurrency:** exactly one node at a time. Never in parallel.
 
 ---
 
+## Kine thundering-herd guardrails (added after 2026-08-16 incident, see DEV-495)
+
+**Historical context (single-CP kine era, 2026-08-16).** When the cluster ran a single CP with embedded SQLite via kine, draining a worker with many StatefulSets could trigger a self-amplifying eviction-and-slow-SQL cascade: kine backs up on writes → apiserver hangs → node-lease renewals fail → taint-eviction kicks in on more nodes → more writes → kine falls further behind → OOM risk on cp-1.
+
+**Current state (HA-etcd era, since DEV-510 on 2026-08-22).** The datastore is now embedded etcd 3.6.12 with 3-node quorum across cp-1/cp-2/cp-3. A single-worker drain no longer creates the lone-SQLite writer risk, and etcd tolerates one member being slow or briefly unreachable. Guardrails (1) and (3) below are retained as sanity checks but the failure mode they were named for is much less likely to trigger. Guardrail (2) is largely done (add swap on cp-1/cp-2/cp-3 tracked in the CP update procedure).
+
+All four rules below MUST be observed on every DEV-478 fire. The reference implementation (`os-update.sh`) does not yet enforce (1)/(3) automatically; the operator must actively watch.
+
+1. **Pre-plan drain order for StatefulSets.** Before draining any worker, `kubectl get pods -n <ns> -o wide` against every namespace with StatefulSets and count how many will be evicted from the target node. If a single drain would evict **more than 3 StatefulSets at once**, redistribute first: cordon+delete individual StatefulSet pods one namespace at a time and wait for each reschedule to settle before draining the whole node.
+2. **All CPs must have swap before their update cycle.** cp-1/cp-2/cp-3 currently all have 0 swap. Add at least 2 GiB (default 4 GiB) of swap on each CP before its OS-update run — see Phase A in [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md). This is a one-off setup task per CP; once done it is a durable capability.
+3. **Halt on kine slowness.** During any drain, keep a `time kubectl get nodes` running from cp-1. If it exceeds **5 s** in real time, halt the cycle immediately (uncordon the current node, do not proceed), verify cluster health, and escalate. The 5 s threshold is the leading indicator that kine has fallen behind and the taint-eviction cascade is about to start.
+4. **CP updates are a separate design task.** With HA etcd across three CPs the risk is now etcd quorum loss (two CPs down simultaneously), not the sole-apiserver reboot. CP updates MUST be planned and board-approved per node; they are NOT covered by the standard `os-update.sh` cycle. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) and `scripts/os-update/update-cp-node.sh` (DEV-515). The old cp-1-only doc `CP1_UPDATE_PROCEDURE.md` is a redirect stub.
+
+---
+
 ## Access
 
 Prereqs are the same as `CLUSTER_ACCESS.md`:
@@ -192,6 +209,16 @@ kubectl get node "$NODE"
 
 If Ready never returns True: escalate. Do NOT change k3s config.
 
+### 5b. Reconcile Docker on labeled nodes (before uncordon)
+
+For nodes carrying the `basicstack.de/docker=true` label (currently the workers that run the Forgejo runner), ensure `docker.io` is present + running before scheduling resumes. Without this the runner pod comes back `ContainerCreating` because `hostPath` requires the docker socket to exist (see [DEV-498](/DEV/issues/DEV-498) and [DEV-499](/DEV/issues/DEV-499)).
+
+```bash
+scripts/os-update/ensure-node-docker.sh "$NODE"
+```
+
+This step is a no-op on nodes without the label. `update-node.sh` runs it automatically between step 5 and step 6; the script is also safe to run ad-hoc after any manual OS operation. As a defensive extra measure, the apt phase (step 3) now runs `apt-mark manual docker.io` on any node where it is installed, so `apt-get autoremove --purge` cannot silently strip it.
+
 ### 6. Uncordon
 
 ```bash
@@ -224,10 +251,13 @@ Only proceed to the next node when ALL of the above are green. If not:
 
 ### 8. Control plane special handling
 
-`k3s-cp-1` is the single control plane node. During its reboot:
-- The kube-apiserver is unavailable — kubectl commands from the operator machine will error out.
-- The `kubectl` waits in step 5/7 must run from a machine that is NOT the control plane, or must be scheduled after the control plane's SSH is back and `curl -k https://localhost:6443/healthz` returns `ok`.
-- Skip the drain for DaemonSet pods on the control plane (`--ignore-daemonsets` covers that), but hosted apps that scheduled onto CP (rare — verify with `kubectl get pods -A --field-selector spec.nodeName=k3s-cp-1`) will be evicted.
+All three CPs (`k3s-cp-1`, `k3s-cp-2`, `k3s-cp-3`) are **NOT** updated by the weekly `os-update.sh` cycle — they have their own dedicated procedure and script: [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md) and `scripts/os-update/update-cp-node.sh`. Reasons:
+
+- CPs are `NoSchedule`-tainted and host no StatefulSet workloads today, but they are the etcd quorum. Two of three CPs cordoned/rebooting at once = write outage. One CP at a time, leader last.
+- All three CPs currently have zero swap. The CP procedure adds a durable 4 GiB swapfile per CP before draining.
+- The CP procedure polls the target's `/livez` from the operator machine and cross-checks `etcdctl endpoint status --cluster` from a peer CP to confirm the just-rebooted member rejoined.
+
+`os-update.sh` excludes any node carrying the `node-role.kubernetes.io/control-plane` label (see `--dry-run` output) and prints a pointer to the CP procedure.
 
 ---
 
@@ -288,8 +318,10 @@ If a node was skipped or errored → mark `blocked` with the unblock action, or
 
 ## Automation entry points
 
-- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything.
-- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry.
+- `infrastructure/scripts/os-update/os-update.sh` — full cycle runner (preflight → per-node loop → finalization). Idempotent, resumable via `--start-from <node>`. Use `--dry-run` to print the plan without touching anything. Explicitly skips `k3s-cp-1`.
+- `infrastructure/scripts/os-update/update-node.sh <node>` — single-node update (all 7 per-node steps). Callable standalone for retry. Not for `k3s-cp-1`.
+- `infrastructure/scripts/os-update/update-cp-node.sh <node>` — HA-aware CP update flow (add swap, preflight, drain, apt, reboot, finalize). Accepts `k3s-cp-1`, `k3s-cp-2`, or `k3s-cp-3`. See [`CP_UPDATE_PROCEDURE.md`](CP_UPDATE_PROCEDURE.md).
+- `infrastructure/scripts/os-update/update-cp-1.sh` — thin wrapper (calls `update-cp-node.sh k3s-cp-1 "$@"`). Kept for backwards compat.
 - `infrastructure/scripts/os-update/cluster-health.sh` — the preflight/post-node health check as a standalone command; exits non-zero on any failure.
 
 Read the script sources for the exact behavior before running them. They mirror this procedure step for step.
@@ -307,3 +339,5 @@ A Paperclip routine (see `infrastructure/OS_UPDATE_ROUTINE.md`) fires this proce
 | Date | Change | By |
 |------|--------|-----|
 | 2026-08-09 | Initial procedure + automation scripts | CTO agent (DEV-462) |
+| 2026-08-16 | Added kine thundering-herd guardrails after DEV-495 incident (worker-3 drain caused CP kine cascade + near-OOM on cp-1). Four new rules: pre-plan StatefulSet drain order, swap on cp-1, halt on kubectl >5 s, cp-1 update as separate design task. | CTO agent (DEV-478/DEV-495) |
+| 2026-08-23 | Downgraded the kine cascade section for the HA-etcd era (DEV-510 migrated cp-1 from kine SQLite to embedded etcd, cp-2/cp-3 joined as full members). Pointed §8 and the "Automation entry points" list at the new HA-aware `CP_UPDATE_PROCEDURE.md` / `update-cp-node.sh`. Old `CP1_UPDATE_PROCEDURE.md` and `update-cp-1.sh` are now redirect stubs. | CTO agent (DEV-515) |

infrastructure/scripts/os-update/README.md

@@ -9,7 +9,8 @@ Scripts that implement the weekly rolling Ubuntu OS-update procedure.
 | Script | Purpose |
 |--------|---------|
 | `cluster-health.sh` | Non-zero if any node is not Ready, any pod is not Running/Ready, or any Deployment/StatefulSet is below its desired replica count. Used at preflight and after each node. |
-| `update-node.sh <node>` | Drain, apt-update, reboot-if-required, wait for Ready, uncordon, post-node health check. Retriable per-node. |
+| `update-node.sh <node>` | Drain, apt-update, reboot-if-required, wait for Ready, ensure Docker on labeled nodes, uncordon, post-node health check. Retriable per-node. |
+| `ensure-node-docker.sh <node>` | Idempotent: on nodes labeled `basicstack.de/docker=true`, install docker.io if missing, `apt-mark manual`, enable+start the docker service, wait for `/var/run/docker.sock`, re-apply the label. No-op on nodes without the label. Called from `update-node.sh` between "kubelet Ready" and "uncordon"; also runnable ad-hoc for recovery. |
 | `os-update.sh` | Full cycle runner: preflight → etcd snapshot → ordered per-node loop → finalization + apt history digest. |
 
 ## Order of operations (encoded in `os-update.sh`)
@@ -42,6 +43,9 @@ Scripts that implement the weekly rolling Ubuntu OS-update procedure.
 
 # Restart a partial cycle from a specific node onward.
 ./os-update.sh --start-from k3s-worker-4
+
+# Ad-hoc: reconcile Docker on a single node (e.g. after emergency ops).
+./ensure-node-docker.sh k3s-worker-3
 ```
 
 Logs land in `/tmp/os-update-<UTC-timestamp>/` on the machine that ran the cycle.

infrastructure/scripts/os-update/ensure-node-docker.sh

@@ -0,0 +1,112 @@
+#!/bin/bash
+# ensure-node-docker.sh — ensure docker.io is installed, enabled, and running on
+# a node that carries (or should carry) the `basicstack.de/docker=true` label.
+#
+# Runs from the operator machine (or the control plane); needs kubectl and
+# ssh access to root@<node-ssh-target>.
+#
+# Behavior:
+#   - If the node has label `basicstack.de/docker=true`, ensure docker.io is
+#     installed, marked `apt-mark manual`, systemd `docker` is enabled+active,
+#     and /var/run/docker.sock exists.
+#   - Re-apply the label (idempotent) so that returning nodes always end the
+#     step in a known state.
+#   - If the node does NOT carry the label, this is a no-op — do NOT install
+#     docker on nodes that were not designated to run the Forgejo runner.
+#
+# Usage:
+#   ensure-node-docker.sh <node-name>
+#
+# Exit codes:
+#   0 — node is either not designated (no label) or Docker is confirmed healthy
+#   1 — hard failure: label present but docker could not be brought up
+#
+# Motivated by DEV-499: the weekly rolling OS update was purging docker.io from
+# workers, which broke the Forgejo runner (DEV-498). Making Docker part of the
+# post-reboot reconciliation removes the manual "apt-get install docker.io &&
+# systemctl enable --now docker && kubectl label node" step.
+set -euo pipefail
+
+NODE="${1:-}"
+if [ -z "$NODE" ]; then
+  echo "usage: $0 <node-name>" >&2
+  exit 2
+fi
+
+CONTROL_PLANE_HOST="${CONTROL_PLANE_HOST:-178.105.17.239}"
+SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}"
+DOCKER_LABEL_KEY="basicstack.de/docker"
+
+log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] ensure-docker/$NODE: $*"; }
+die() { log "FATAL: $*"; exit 1; }
+
+# node -> ssh target. Keep in sync with update-node.sh::node_ssh_target.
+node_ssh_target() {
+  case "$1" in
+    k3s-cp-1)            echo "root@178.105.17.239" ;;
+    k3s-worker-1)        echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.2" ;;
+    k3s-worker-2)        echo "-J root@$CONTROL_PLANE_HOST root@10.42.1.3" ;;
+    k3s-worker-3)        echo "root@167.233.121.121" ;;
+    k3s-worker-4)        echo "root@128.140.3.80" ;;
+    k3s-worker-5)        echo "root@167.233.192.86" ;;
+    k3s-update-runner)   echo "root@167.233.79.65" ;;
+    *)                   die "unknown node $1 — update node_ssh_target() in $0" ;;
+  esac
+}
+
+# --- 1. is this node designated to run Docker? --------------------------------
+kubectl get node "$NODE" >/dev/null || die "node $NODE not found in cluster"
+LABEL_VAL=$(kubectl get node "$NODE" \
+  -o jsonpath="{.metadata.labels.${DOCKER_LABEL_KEY//./\\.}}" 2>/dev/null || echo "")
+
+if [ "$LABEL_VAL" != "true" ]; then
+  log "no ${DOCKER_LABEL_KEY}=true label — skipping Docker reconciliation"
+  exit 0
+fi
+
+SSH_TARGET=$(node_ssh_target "$NODE")
+log "label ${DOCKER_LABEL_KEY}=true present — reconciling docker.io via $SSH_TARGET"
+
+# --- 2. ensure docker.io on the node -----------------------------------------
+REMOTE=$(cat <<'REMOTE'
+set -euo pipefail
+export DEBIAN_FRONTEND=noninteractive
+APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
+
+# Recover from any half-finished dpkg state before touching apt.
+dpkg --configure -a >/dev/null 2>&1 || true
+
+if ! dpkg -s docker.io >/dev/null 2>&1; then
+  echo "docker.io not installed — installing"
+  apt-get update
+  apt-get $APT_OPTS install docker.io
+else
+  echo "docker.io already installed"
+fi
+
+# Keep docker.io out of the reach of apt-get autoremove --purge, which is what
+# the weekly OS-update cycle runs. Idempotent.
+apt-mark manual docker.io >/dev/null
+
+systemctl enable --now docker
+
+# Wait for the docker socket to appear so the forgejo-runner pod can bind it.
+for _ in $(seq 1 30); do
+  if [ -S /var/run/docker.sock ]; then break; fi
+  sleep 1
+done
+if [ ! -S /var/run/docker.sock ]; then
+  echo "docker.sock missing after enable" >&2
+  systemctl status docker --no-pager | tail -20 >&2
+  exit 1
+fi
+
+echo "docker OK: $(docker version --format '{{.Server.Version}}')"
+REMOTE
+)
+
+ssh $SSH_OPTS $SSH_TARGET "bash -s" <<< "$REMOTE" | sed 's/^/    /'
+
+# --- 3. re-apply the label (idempotent, in case someone stripped it) ---------
+kubectl label node "$NODE" "${DOCKER_LABEL_KEY}=true" --overwrite >/dev/null
+log "docker + label reconciled OK"

infrastructure/scripts/os-update/os-update.sh

@@ -68,19 +68,33 @@ fi
 
 # --- 3. node ordering ---------------------------------------------------------
 # Ordering rule:
-#   - workers before control plane
+#   - workers only
 #   - within workers: nodes NOT hosting Stalwart first, Stalwart-hosting fsn1 nodes last
-#   - k3s-cp-1 always last
+#   - ALL control-plane nodes are EXCLUDED and never updated by this script. Rationale:
+#     * kine/etcd write-path is sensitive to concurrent drains (see kine thundering-herd
+#       guardrails in OS_UPDATE_PROCEDURE.md, added after DEV-495).
+#     * rebooting a CP removes one apiserver — needs external liveness monitoring.
+#     * CP nodes host their own StatefulSet workloads that need batched eviction.
+#   Since DEV-510 (2026-08-22) the cluster runs HA control plane (cp-1/cp-2/cp-3). The
+#   exclusion here is role-based so ALL current and future CPs are covered automatically.
+#   To update a CP, use `scripts/os-update/update-cp-1.sh` (currently cp-1-only; will be
+#   generalized to any CP as part of the HA-aware CP OS-update procedure follow-up).
 STALWART_NODE=$(kubectl -n stalwart get pod -l app=stalwart -o jsonpath='{.items[*].spec.nodeName}' 2>/dev/null | tr ' ' '\n' | sort -u || true)
 # If the pod's not currently up (e.g. Pending) we still want to protect fsn1 workers.
-CP_NAME="k3s-cp-1"
 
+CP_NODES=$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[*].metadata.name}' 2>/dev/null | tr ' ' '\n' || true)
 ALL_NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n')
 
+is_cp() {
+  local n="$1"
+  for c in $CP_NODES; do [ "$c" = "$n" ] && return 0; done
+  return 1
+}
+
 workers=()
 stalwart_workers=()
 for n in $ALL_NODES; do
-  [ "$n" = "$CP_NAME" ] && continue
+  is_cp "$n" && continue
   if [ -n "$STALWART_NODE" ] && [ "$n" = "$STALWART_NODE" ]; then
     stalwart_workers+=("$n")
     continue
@@ -94,7 +108,11 @@ for n in $ALL_NODES; do
   fi
 done
 
-ORDER=("${workers[@]}" "${stalwart_workers[@]}" "$CP_NAME")
+ORDER=("${workers[@]}" "${stalwart_workers[@]}")
+# NOTE: CP nodes intentionally excluded. To update a CP, see CP1_UPDATE_PROCEDURE.md.
+if [ -n "$CP_NODES" ]; then
+  log "[plan] EXCLUDING control-plane nodes: $(echo $CP_NODES | tr '\n' ' ')— use scripts/os-update/update-cp-1.sh (see CP1_UPDATE_PROCEDURE.md)"
+fi
 
 if [ -n "$ONLY" ]; then
   ORDER=("$ONLY")

infrastructure/scripts/os-update/update-cp-1.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# update-cp-1.sh -- SUPERSEDED thin wrapper.
+#
+# Since DEV-515 (2026-08-23) the HA-aware entry point is update-cp-node.sh.
+# This wrapper forwards to `update-cp-node.sh k3s-cp-1 "$@"` so historical
+# callers keep working (routines, runbook references, board-approval docs).
+#
+# See ../CP_UPDATE_PROCEDURE.md for the current procedure.
+# The old cp-1-only design lives in git history at commit 19af691.
+
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+exec "$SCRIPT_DIR/update-cp-node.sh" k3s-cp-1 "$@"

infrastructure/scripts/os-update/update-cp-node.sh

@@ -0,0 +1,595 @@
+#!/bin/bash
+# update-cp-node.sh — controlled OS update for any HA k3s control-plane node.
+#
+# See ../CP_UPDATE_PROCEDURE.md for the full design rationale.
+#
+# Usage:
+#   update-cp-node.sh <node> --dry-run          # print what would be done, touch nothing
+#   update-cp-node.sh <node> --add-swap         # Phase A only (idempotent, safe standalone)
+#   update-cp-node.sh <node> --preflight        # Phase B only
+#   update-cp-node.sh <node> --drain            # Phase C only (cordon + drain)
+#   update-cp-node.sh <node> --apt              # Phase D only (requires <node> already fully drained)
+#   update-cp-node.sh <node> --reboot           # Phase E only (requires --apt reported REBOOT_REQUIRED=yes)
+#   update-cp-node.sh <node> --finalize         # Phase F only (uncordon + verify)
+#   update-cp-node.sh <node> --run              # all phases with confirmation between each (or ASSUME_YES=1)
+#
+# <node> must be one of: k3s-cp-1, k3s-cp-2, k3s-cp-3.
+#
+# Environment overrides:
+#   SWAP_SIZE_MB                 default 4096       (>=2048 required)
+#   SWAP_PATH                    default /swapfile
+#   DRAIN_TIMEOUT_SECONDS        default 600
+#   REBOOT_MAX_WAIT_SECONDS      default 600
+#   POST_UNCORDON_WAIT_SECONDS   default 180
+#   MIN_TARGET_MEM_MIB           default 200        (target-CP MemAvailable floor mid-drain)
+#   MAX_KUBECTL_SECONDS          default 5          (kine-latency guardrail; softened for HA etcd)
+#   SSH_OPTS                     default "-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new"
+#   ASSUME_YES=1                 skip interactive confirmations in --run
+#   DRY_RUN=1                    do not execute state-mutating commands; log-only
+#
+# Log directory contract (matches update-node.sh):
+#   Every command's stdout+stderr is tee'd to /tmp/os-update-<node>-<UTC-ts>.log on
+#   the operator machine. Attach it to the execution ticket at the end.
+#
+# NEVER touches k3s config, k3s services, containerd, or any manifest. Only
+# fixes it will attempt: apt/dpkg recovery on the target CP (Phase D). Any other
+# problem -> escalate and STOP.
+
+set -euo pipefail
+
+# --------------------------------------------------------------------------- #
+# CP topology — public/routable IPs used to reach each CP via ssh + /livez.
+# Kept out-of-cluster on purpose: during a full-cluster incident the operator
+# machine must be able to reach each CP without going through k3s.
+# --------------------------------------------------------------------------- #
+declare -A CP_HOST=(
+  [k3s-cp-1]="178.105.17.239"
+  [k3s-cp-2]="188.245.85.199"
+  [k3s-cp-3]="49.13.92.162"
+)
+
+# --------------------------------------------------------------------------- #
+# Arg parse — expect <node> as $1
+# --------------------------------------------------------------------------- #
+usage() { grep -E '^# ' "$0" | sed 's/^# \{0,1\}//'; exit 2; }
+[ $# -ge 2 ] || usage
+
+NODE="$1"; shift
+TARGET_HOST="${CP_HOST[$NODE]:-}"
+if [ -z "$TARGET_HOST" ]; then
+  echo "unknown CP node: $NODE (allowed: ${!CP_HOST[*]})" >&2
+  exit 2
+fi
+
+# --------------------------------------------------------------------------- #
+# Config
+# --------------------------------------------------------------------------- #
+SWAP_SIZE_MB="${SWAP_SIZE_MB:-4096}"
+SWAP_PATH="${SWAP_PATH:-/swapfile}"
+DRAIN_TIMEOUT_SECONDS="${DRAIN_TIMEOUT_SECONDS:-600}"
+REBOOT_MAX_WAIT_SECONDS="${REBOOT_MAX_WAIT_SECONDS:-600}"
+POST_UNCORDON_WAIT_SECONDS="${POST_UNCORDON_WAIT_SECONDS:-180}"
+MIN_TARGET_MEM_MIB="${MIN_TARGET_MEM_MIB:-200}"
+MAX_KUBECTL_SECONDS="${MAX_KUBECTL_SECONDS:-5}"
+SSH_OPTS="${SSH_OPTS:--o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new}"
+DRY_RUN="${DRY_RUN:-0}"
+ASSUME_YES="${ASSUME_YES:-0}"
+
+TS="$(date -u +%Y%m%dT%H%M%SZ)"
+LOG_LOCAL="/tmp/os-update-${NODE}-${TS}.log"
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+HEALTH_SCRIPT="$SCRIPT_DIR/cluster-health.sh"
+
+# Pick a peer CP for etcd status probes. Requirements:
+#   - not the target
+#   - etcdctl available (via non-login ssh PATH)
+# We probe each candidate at parse time. Order preference: cp-1 first (it has
+# etcdctl installed since the pre-HA era), then cp-2, then cp-3. The Phase A
+# `--add-swap` step also installs `etcd-client` via apt so cp-2/cp-3 pick up
+# etcdctl on their first pass.
+peer_host_for() {
+  local target="$1"
+  local order=(k3s-cp-1 k3s-cp-2 k3s-cp-3)
+  for n in "${order[@]}"; do
+    [ "$n" = "$target" ] && continue
+    local h="${CP_HOST[$n]}"
+    if ssh $SSH_OPTS -o BatchMode=yes "root@$h" 'command -v etcdctl >/dev/null 2>&1' 2>/dev/null; then
+      echo "$h"; return 0
+    fi
+  done
+  # In DRY_RUN we don't need a real etcdctl-capable peer.
+  if [ "${DRY_RUN:-0}" = "1" ]; then
+    for n in "${order[@]}"; do
+      [ "$n" != "$target" ] && { echo "${CP_HOST[$n]}"; return 0; }
+    done
+  fi
+  # No non-target CP has etcdctl. Return empty so Phase B/E can fail
+  # explicitly with a targeted "install etcd-client on cp-X first" message.
+  echo ""
+  return 0
+}
+PEER_HOST="$(peer_host_for "$NODE")"
+
+# --------------------------------------------------------------------------- #
+# Logging + safe-run helpers
+# --------------------------------------------------------------------------- #
+log()  { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOG_LOCAL"; }
+die()  { log "FATAL: $*"; exit 1; }
+warn() { log "WARN:  $*"; }
+
+run() {
+  if [ "$DRY_RUN" = "1" ]; then
+    log "DRY-RUN would exec: $*"
+    return 0
+  fi
+  log "exec: $*"
+  "$@" 2>&1 | tee -a "$LOG_LOCAL"
+}
+
+ssh_run() {
+  local target="${1:-root@$TARGET_HOST}"; shift || true
+  if [ "$DRY_RUN" = "1" ]; then
+    log "DRY-RUN would ssh $target: $*"
+    return 0
+  fi
+  log "ssh $target: $*"
+  ssh $SSH_OPTS "$target" "$@" 2>&1 | tee -a "$LOG_LOCAL"
+}
+
+# ssh_run_stdin: pipe a heredoc through bash -s on the target; used for multi-line remote blocks.
+ssh_run_stdin() {
+  local target="root@${TARGET_HOST}"
+  if [ "$DRY_RUN" = "1" ]; then
+    log "DRY-RUN would ssh $target with stdin script:"
+    sed 's/^/    | /' | tee -a "$LOG_LOCAL"
+    return 0
+  fi
+  log "ssh $target (heredoc)"
+  ssh $SSH_OPTS "$target" "bash -s" 2>&1 | tee -a "$LOG_LOCAL"
+}
+
+confirm() {
+  local prompt="$1"
+  if [ "$ASSUME_YES" = "1" ]; then
+    log "confirm SKIPPED (ASSUME_YES=1): $prompt"
+    return 0
+  fi
+  echo -n "  >>> $prompt  Continue? [y/N] "
+  read -r a
+  case "$a" in y|Y|yes|YES) return 0 ;; *) die "aborted by operator" ;; esac
+}
+
+# --------------------------------------------------------------------------- #
+# etcd helpers — run etcdctl on the target OR on a peer CP.
+# --------------------------------------------------------------------------- #
+ETCDCTL_ENV='ETCDCTL_API=3 etcdctl \
+  --endpoints=https://127.0.0.1:2379 \
+  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
+  --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
+  --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key'
+
+etcd_status_via() {
+  # $1 = host (IP)
+  local host="$1"
+  ssh $SSH_OPTS "root@$host" "$ETCDCTL_ENV endpoint status --cluster -w table" 2>&1
+}
+
+etcd_leader_name() {
+  # returns the etcd member NAME (e.g. k3s-cp-3-9d305472) whose row shows IS LEADER=true.
+  # Uses simple-format output (no -w table) for reliable parsing.
+  local host="$1"
+  ssh $SSH_OPTS "root@$host" "$ETCDCTL_ENV endpoint status --cluster -w simple" 2>/dev/null \
+    | awk -F, '$5 ~ /true/ {print $1}'
+}
+
+target_is_leader() {
+  # returns 0 if the target IP is the current etcd leader, 1 otherwise.
+  local host="$1"     # peer to query from
+  local target="$2"   # target IP to compare
+  local leader_ep
+  leader_ep=$(ssh $SSH_OPTS "root@$host" "$ETCDCTL_ENV endpoint status --cluster -w simple" 2>/dev/null \
+    | awk -F, '$5 ~ /true/ {print $1}')
+  # leader_ep looks like https://49.13.92.162:2379
+  echo "$leader_ep" | grep -q "://${target}:" && return 0 || return 1
+}
+
+# --------------------------------------------------------------------------- #
+# Phase A — Add swap on the target CP (idempotent)
+# --------------------------------------------------------------------------- #
+phase_add_swap() {
+  log "=== Phase A: add swap on $NODE ($SWAP_SIZE_MB MiB at $SWAP_PATH) ==="
+  [ "$SWAP_SIZE_MB" -ge 2048 ] || die "SWAP_SIZE_MB=$SWAP_SIZE_MB below 2048 MiB guardrail"
+
+  cat <<REMOTE | ssh_run_stdin
+set -euo pipefail
+SWAP_PATH="$SWAP_PATH"
+SIZE_MB="$SWAP_SIZE_MB"
+
+# Skip if a swapfile at this path is already active.
+if swapon --show=NAME 2>/dev/null | grep -qx "\$SWAP_PATH"; then
+  echo "swap already on at \$SWAP_PATH -- skipping"
+  free -h
+  exit 0
+fi
+
+# Root filesystem free space check -- abort if less than 2*swap free.
+avail_mb=\$(df -m --output=avail / | tail -1 | tr -d ' ')
+need_mb=\$(( SIZE_MB * 2 ))
+if [ "\$avail_mb" -lt "\$need_mb" ]; then
+  echo "ERROR: only \${avail_mb} MiB free on /, need \${need_mb} MiB (2x swap for safety)"
+  exit 1
+fi
+
+# Create swapfile. fallocate is fast; dd is the fallback.
+if ! fallocate -l "\${SIZE_MB}M" "\$SWAP_PATH" 2>/dev/null; then
+  dd if=/dev/zero of="\$SWAP_PATH" bs=1M count="\$SIZE_MB" status=progress
+fi
+chmod 600 "\$SWAP_PATH"
+mkswap "\$SWAP_PATH"
+swapon "\$SWAP_PATH"
+
+# Persist via fstab (dedup).
+if ! grep -q "^\$SWAP_PATH " /etc/fstab; then
+  echo "\$SWAP_PATH none swap sw 0 0" >> /etc/fstab
+fi
+
+# Moderate swappiness -- swap as safety net, not aggressive paging.
+sysctl -w vm.swappiness=10
+if [ ! -f /etc/sysctl.d/99-k3s-swap.conf ] || ! grep -q '^vm.swappiness' /etc/sysctl.d/99-k3s-swap.conf; then
+  echo 'vm.swappiness=10' > /etc/sysctl.d/99-k3s-swap.conf
+fi
+
+echo "--- swap after ---"
+free -h
+swapon --show
+sysctl vm.swappiness
+
+# Ensure etcdctl is available for etcd-quorum probes (idempotent apt install).
+# Needed because when this CP is the target of a later update, another CP
+# must probe etcd cluster status; if cp-1 is the target, one of cp-2/cp-3
+# is the probing peer and must have etcdctl.
+if ! command -v etcdctl >/dev/null 2>&1; then
+  echo "--- installing etcd-client (provides etcdctl) ---"
+  export DEBIAN_FRONTEND=noninteractive
+  apt-get update -y >/dev/null
+  apt-get install -y etcd-client
+  command -v etcdctl && etcdctl version
+fi
+REMOTE
+
+  if [ "$DRY_RUN" != "1" ]; then
+    log "verifying kubelet still Ready after swap add"
+    local ready
+    ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
+    [ "$ready" = "True" ] || die "kubelet on $NODE not Ready after swap add -- halt"
+    log "kubelet Ready=True -- Phase A complete"
+  fi
+}
+
+# --------------------------------------------------------------------------- #
+# Guardrail probes
+# --------------------------------------------------------------------------- #
+kubectl_latency_s() {
+  local start end
+  start=$(date +%s)
+  kubectl get nodes >/dev/null 2>&1 || echo "kubectl-error" >&2
+  end=$(date +%s)
+  echo $(( end - start ))
+}
+
+target_mem_available_mib() {
+  ssh $SSH_OPTS "root@$TARGET_HOST" "awk '/^MemAvailable:/{printf \"%d\n\", \$2/1024}' /proc/meminfo" 2>/dev/null || echo 0
+}
+
+guard_kine_healthy() {
+  local s
+  s=$(kubectl_latency_s)
+  if [ "$s" -gt "$MAX_KUBECTL_SECONDS" ]; then
+    die "kubectl get nodes took ${s}s (>${MAX_KUBECTL_SECONDS}s) -- etcd slow, HALT"
+  fi
+  log "  kubectl-latency ok: ${s}s"
+}
+
+guard_target_memory() {
+  local m
+  m=$(target_mem_available_mib)
+  if [ "$m" -lt "$MIN_TARGET_MEM_MIB" ]; then
+    die "$NODE MemAvailable=${m} MiB below ${MIN_TARGET_MEM_MIB} MiB floor -- HALT"
+  fi
+  log "  $NODE mem ok: MemAvailable=${m} MiB"
+}
+
+# --------------------------------------------------------------------------- #
+# Phase B — Preflight
+# --------------------------------------------------------------------------- #
+phase_preflight() {
+  log "=== Phase B: preflight for $NODE ==="
+
+  log "-- cluster health"
+  if [ "$DRY_RUN" != "1" ]; then
+    if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then
+      die "cluster is not healthy -- refuse to start $NODE update"
+    fi
+  else
+    log "DRY-RUN would run: $HEALTH_SCRIPT"
+  fi
+
+  log "-- confirm $NODE is a control-plane node"
+  if [ "$DRY_RUN" != "1" ]; then
+    local is_cp
+    is_cp=$(kubectl get node "$NODE" -o jsonpath='{.metadata.labels.node-role\.kubernetes\.io/control-plane}' 2>/dev/null || echo "")
+    [ "$is_cp" = "true" ] || die "$NODE is not labelled control-plane -- refuse (use update-node.sh for workers)"
+  fi
+
+  log "-- verify no OTHER CP is currently cordoned"
+  if [ "$DRY_RUN" != "1" ]; then
+    local other_cordoned
+    other_cordoned=$(kubectl get nodes -l node-role.kubernetes.io/control-plane=true \
+      -o json | jq -r --arg n "$NODE" '.items[] | select(.metadata.name != $n) | select(.spec.unschedulable == true) | .metadata.name' \
+      | tr '\n' ' ')
+    if [ -n "${other_cordoned// /}" ]; then
+      die "another CP is already cordoned: $other_cordoned -- refuse (one CP at a time)"
+    fi
+    log "  no other CP cordoned -- proceeding"
+  fi
+
+  log "-- etcd cluster status (all members must be started)"
+  if [ "$DRY_RUN" != "1" ]; then
+    if [ -z "$PEER_HOST" ]; then
+      die "no non-target CP has etcdctl installed -- run \`update-cp-node.sh <other-cp> --add-swap\` first on one of the OTHER CPs (that step installs etcd-client), then retry"
+    fi
+    etcd_status_via "$PEER_HOST" | tee -a "$LOG_LOCAL"
+  fi
+
+  log "-- swap on $NODE"
+  if [ "$DRY_RUN" != "1" ]; then
+    local swap_total
+    swap_total=$(ssh $SSH_OPTS "root@$TARGET_HOST" "awk '/^SwapTotal:/{print \$2}' /proc/meminfo")
+    [ "${swap_total:-0}" -ge $((2 * 1024 * 1024)) ] \
+      || die "$NODE SwapTotal=${swap_total} KiB below 2 GiB -- run --add-swap first"
+    log "  $NODE SwapTotal=$(( swap_total / 1024 )) MiB"
+  fi
+
+  log "-- kubectl-latency probe"
+  if [ "$DRY_RUN" != "1" ]; then guard_kine_healthy; fi
+
+  log "-- record current etcd leader"
+  if [ "$DRY_RUN" != "1" ] && [ -n "$PEER_HOST" ]; then
+    if target_is_leader "$PEER_HOST" "$TARGET_HOST"; then
+      warn "$NODE IS the current etcd leader. Per CP ordering rule, prefer updating a follower first."
+      warn "  Not aborting -- operator/agent must confirm this is intentional."
+    else
+      log "  $NODE is a FOLLOWER -- safe to proceed."
+    fi
+  fi
+
+  log "-- k3s etcd snapshot"
+  ssh_run "root@$TARGET_HOST" "k3s etcd-snapshot save --name pre-cp-os-update-${NODE}-${TS}"
+  ssh_run "root@$TARGET_HOST" "ls -la /var/lib/rancher/k3s/server/db/snapshots/ | tail -10"
+
+  log "=== Phase B: preflight OK ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Phase C — Cordon + drain
+# --------------------------------------------------------------------------- #
+phase_drain() {
+  log "=== Phase C: cordon + drain $NODE ==="
+
+  log "-- cordon $NODE"
+  run kubectl cordon "$NODE"
+
+  log "-- drain $NODE (timeout ${DRAIN_TIMEOUT_SECONDS}s)"
+  set +e
+  if [ "$DRY_RUN" = "1" ]; then
+    log "DRY-RUN would run: kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --timeout=${DRAIN_TIMEOUT_SECONDS}s"
+    local rc=0
+  else
+    kubectl drain "$NODE" \
+      --ignore-daemonsets \
+      --delete-emptydir-data \
+      --timeout="${DRAIN_TIMEOUT_SECONDS}s" 2>&1 | tee -a "$LOG_LOCAL"
+    local rc=${PIPESTATUS[0]}
+  fi
+  set -e
+
+  if [ "$rc" -ne 0 ]; then
+    log "drain FAILED (rc=$rc). Never force. Uncordoning."
+    run kubectl uncordon "$NODE"
+    die "drain failed on $NODE -- investigate PDB / orphan pods; do NOT proceed"
+  fi
+
+  log "-- post-drain guardrails"
+  if [ "$DRY_RUN" != "1" ]; then
+    guard_kine_healthy
+    guard_target_memory
+  fi
+
+  log "=== Phase C: $NODE drained ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Phase D — apt on the target
+# --------------------------------------------------------------------------- #
+phase_apt() {
+  log "=== Phase D: apt on $NODE ==="
+
+  cat <<'REMOTE' | ssh_run_stdin
+set -euo pipefail
+export DEBIAN_FRONTEND=noninteractive
+APT_OPTS='-y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold'
+
+uname -r > /root/pre-apt-kernel
+dpkg-query -W -f='${Package}\t${Version}\n' > /root/pre-apt-packages.tsv
+echo "pre-apt kernel: $(cat /root/pre-apt-kernel)"
+
+if dpkg --audit | grep -qE .; then
+  echo "dpkg audit reported issues, running dpkg --configure -a"
+  dpkg --configure -a || true
+fi
+
+apt-get update
+
+if ! apt-get $APT_OPTS upgrade; then
+  echo "upgrade failed, attempting apt-get -f install"
+  apt-get $APT_OPTS -f install
+  apt-get $APT_OPTS upgrade
+fi
+
+apt-get $APT_OPTS dist-upgrade
+apt-get $APT_OPTS autoremove --purge
+apt-get clean
+
+if [ -f /var/run/reboot-required ]; then
+  echo "REBOOT_REQUIRED=yes"
+  echo "REBOOT_REASON<<EOF"
+  cat /var/run/reboot-required.pkgs 2>/dev/null || echo "(no package list)"
+  echo "EOF"
+else
+  echo "REBOOT_REQUIRED=no"
+fi
+REMOTE
+
+  log "=== Phase D: apt complete (check REBOOT_REQUIRED in the log) ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Phase E — Reboot and wait for target /livez + kubelet Ready + etcd rejoin
+# --------------------------------------------------------------------------- #
+phase_reboot() {
+  log "=== Phase E: reboot $NODE ==="
+
+  if [ "$DRY_RUN" != "1" ]; then
+    log "issuing 'systemctl reboot' on $NODE (ssh will drop; expected)"
+    ssh $SSH_OPTS "root@$TARGET_HOST" 'systemctl reboot' 2>&1 | tee -a "$LOG_LOCAL" || true
+    log "waiting 15s for ssh to fully drop before polling"
+    sleep 15
+  else
+    log "DRY-RUN would ssh root@$TARGET_HOST 'systemctl reboot'"
+  fi
+
+  log "-- poll $NODE api-server /livez (timeout ${REBOOT_MAX_WAIT_SECONDS}s)"
+  if [ "$DRY_RUN" != "1" ]; then
+    local deadline=$(( $(date +%s) + REBOOT_MAX_WAIT_SECONDS ))
+    local code=000
+    while [ $(date +%s) -lt $deadline ]; do
+      code=$(curl -sk -o /dev/null -w '%{http_code}' "https://$TARGET_HOST:6443/livez" 2>/dev/null || echo 000)
+      if [ "$code" = "200" ]; then
+        log "  $NODE api-server /livez=200"
+        break
+      fi
+      sleep 5
+    done
+    [ "$code" = "200" ] || die "$NODE api-server did not return within ${REBOOT_MAX_WAIT_SECONDS}s -- escalate; check 'hcloud server describe $NODE' and Hetzner console"
+  fi
+
+  log "-- verify etcd cluster status from peer ($PEER_HOST) -- $NODE should be 'started'"
+  if [ "$DRY_RUN" != "1" ] && [ -n "$PEER_HOST" ]; then
+    etcd_status_via "$PEER_HOST" | tee -a "$LOG_LOCAL"
+  fi
+
+  log "-- wait for kubelet Ready on $NODE (max 300s)"
+  if [ "$DRY_RUN" != "1" ]; then
+    local deadline=$(( $(date +%s) + 300 ))
+    local ready=Unknown
+    while [ $(date +%s) -lt $deadline ]; do
+      ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo Unknown)
+      [ "$ready" = "True" ] && break
+      sleep 5
+    done
+    [ "$ready" = "True" ] || die "kubelet on $NODE never returned Ready -- escalate (do NOT change k3s config)"
+    log "  kubelet Ready=True"
+  fi
+
+  log "=== Phase E: $NODE is back ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Phase F — Uncordon + verify + finalize
+# --------------------------------------------------------------------------- #
+phase_finalize() {
+  log "=== Phase F: uncordon + verify $NODE ==="
+
+  run kubectl uncordon "$NODE"
+
+  log "-- settle wait ${POST_UNCORDON_WAIT_SECONDS}s"
+  [ "$DRY_RUN" = "1" ] || sleep "$POST_UNCORDON_WAIT_SECONDS"
+
+  log "-- cluster health"
+  if [ "$DRY_RUN" != "1" ]; then
+    if ! RETRY_ON_TRANSIENT=1 "$HEALTH_SCRIPT" 2>&1 | tee -a "$LOG_LOCAL"; then
+      die "cluster health failed after $NODE update -- escalate, do NOT touch k3s"
+    fi
+  fi
+
+  log "-- etcd cluster status (all three should be started)"
+  if [ "$DRY_RUN" != "1" ] && [ -n "$PEER_HOST" ]; then
+    etcd_status_via "$PEER_HOST" | tee -a "$LOG_LOCAL"
+  fi
+
+  log "-- apt history summary (audit)"
+  ssh_run "root@$TARGET_HOST" 'zgrep -h "Commandline\|Install\|Upgrade\|Remove" /var/log/apt/history.log* 2>/dev/null | tail -60'
+
+  log "-- old snapshots (>30d) -- listing only, review manually"
+  ssh_run "root@$TARGET_HOST" 'find /var/lib/rancher/k3s/server/db/snapshots/ -type f -mtime +30 -name "pre-*" -print 2>/dev/null || true'
+
+  log "=== $NODE OS update complete -- attach $LOG_LOCAL to the execution ticket ==="
+}
+
+# --------------------------------------------------------------------------- #
+# --run — orchestrate all phases with confirmations
+# --------------------------------------------------------------------------- #
+phase_run_all() {
+  log "=== full $NODE update run (log: $LOG_LOCAL) ==="
+
+  confirm "Phase A (add swap) -- proceed?"
+  phase_add_swap
+
+  confirm "Phase B (preflight) -- proceed?"
+  phase_preflight
+
+  confirm "Phase C (cordon + drain) -- proceed?"
+  phase_drain
+
+  confirm "Phase D (apt) -- proceed?"
+  phase_apt
+
+  confirm "Phase E (reboot $NODE; api-server on THIS node unavailable ~90-180s, other 2 CPs keep serving) -- proceed?"
+  phase_reboot
+
+  confirm "Phase F (uncordon + verify) -- proceed?"
+  phase_finalize
+
+  log "=== FULL RUN COMPLETE for $NODE ==="
+}
+
+# --------------------------------------------------------------------------- #
+# Phase dispatch
+# --------------------------------------------------------------------------- #
+: > "$LOG_LOCAL"
+log "update-cp-node.sh started (NODE=$NODE, TARGET_HOST=$TARGET_HOST, PEER_HOST=$PEER_HOST, DRY_RUN=$DRY_RUN)"
+log "log file: $LOG_LOCAL"
+
+case "$1" in
+  --dry-run)
+    DRY_RUN=1
+    export DRY_RUN
+    log "DRY_RUN=1 -- walking Phases A..F without touching state"
+    phase_add_swap
+    phase_preflight
+    phase_drain
+    phase_apt
+    phase_reboot
+    phase_finalize
+    ;;
+  --add-swap)       phase_add_swap ;;
+  --preflight)      phase_preflight ;;
+  --drain)          phase_drain ;;
+  --apt)            phase_apt ;;
+  --reboot)         phase_reboot ;;
+  --finalize)       phase_finalize ;;
+  --run)            phase_run_all ;;
+  -h|--help)        usage ;;
+  *)                echo "unknown arg: $1" >&2; usage ;;
+esac

infrastructure/scripts/os-update/update-node.sh

@@ -101,6 +101,13 @@ fi
 
 apt-get update
 
+# Before any upgrade/autoremove pass, pin docker.io as manually-installed on
+# nodes where it is present. Without this, apt autoremove --purge has, in the
+# past, taken docker.io out from under the Forgejo runner (DEV-498/DEV-499).
+if dpkg -s docker.io >/dev/null 2>&1; then
+  apt-mark manual docker.io >/dev/null
+fi
+
 # Try upgrade; on broken deps, one attempt at apt-get -f install then retry.
 if ! apt-get $APT_OPTS upgrade; then
   echo "upgrade failed, attempting apt-get -f install"
@@ -167,6 +174,14 @@ done
 [ "$READY" = "True" ] || die "kubelet on $NODE never returned Ready — escalate (do NOT change k3s config)"
 log "  Ready=True"
 
+# --- 5b. ensure Docker on designated nodes ------------------------------------
+# Do this BEFORE uncordoning so the runner pod's first scheduling attempt
+# succeeds instead of racing through ContainerCreating. Idempotent no-op on
+# nodes that are not labeled basicstack.de/docker=true. (DEV-499)
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+log "[5b/7] ensure docker.io on nodes labeled basicstack.de/docker=true"
+"$SCRIPT_DIR/ensure-node-docker.sh" "$NODE"
+
 # --- 6. uncordon --------------------------------------------------------------
 log "[6/7] uncordon $NODE"
 kubectl uncordon "$NODE"
@@ -175,7 +190,6 @@ kubectl uncordon "$NODE"
 log "[7/7] post-node settle (${POST_UNCORDON_WAIT_SECONDS}s) + health check"
 sleep "$POST_UNCORDON_WAIT_SECONDS"
 
-SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 if RETRY_ON_TRANSIENT=1 "$SCRIPT_DIR/cluster-health.sh"; then
   log "=== $NODE update: OK ==="
 else