Argo CD repo-server needs the forgejo.forgejo.svc.cluster.local host key
in its known_hosts to clone the git repos over SSH. Adding it to
configs.ssh.extraHosts so the helm-managed ssh-known-hosts-cm renders
the entry from git and does not drift when the argocd Application syncs.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Replace the vendored ~33k-line apps/argocd/argocd-install.yaml with the
argoproj/argo-helm chart argo-cd 10.4.0 (app v3.5.1). Values live in
apps/argocd/values.yaml; the local kustomize wrapper now only carries the
Traefik ingress and the sealed secrets. The root apps/app-argocd.yaml
Application becomes multi-source (chart + this repo as $values), enables
ServerSideApply + ApplyOutOfSyncOnly, and pins the resources-finalizer
explicitly.
Behavior-equivalent to the previous install: same URL, OIDC (Pocket ID),
argo_admins RBAC mapping, resource.exclusions list, and per-component
memory limits (DEV-281). Ingress is disabled in the chart; ours stays in
kustomize with cert-manager letsencrypt-prod annotations.
README.md updated with the Helm bump procedure. argocd-install.yaml
removed.
Verified locally:
helm template argocd argo/argo-cd --version 10.4.0 \
-f apps/argocd/values.yaml -n argocd
# renders 34k lines, image: quay.io/argoproj/argocd:v3.5.1
kustomize build apps/argocd/
# renders 1 Ingress + 3 SealedSecrets, no errors
Refs: DEV-521, plan DEV-519 §3, §8.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an
HA-aware procedure that covers all three CPs. Rationale: after DEV-510
the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs
are NoSchedule-tainted, so:
- Kine cascade is no longer the driving risk; embedded 3-node etcd
tolerates one member down. The kubectl-latency guardrail stays as a
soft signal.
- CPs host no StatefulSets or single-replica Deployments. The old
Phase C batched stateful eviction dance is obsolete.
- CP reboots no longer lose the api-server; two peers keep serving.
/livez is polled from the operator machine and cross-checked with
etcdctl endpoint status from a peer CP.
New CP ordering rule: one CP at a time, leader last, health gate
between CPs. Refuses to start if another CP is already cordoned.
Automation:
- New scripts/os-update/update-cp-node.sh <node> is the entry point.
Supports --add-swap / --preflight / --drain / --apt / --reboot /
--finalize / --run / --dry-run. Phase A also installs etcd-client
on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the
target. Peer picker prefers a CP that has etcdctl; refuses cp-1
target with a targeted error if no other CP has etcdctl yet.
- scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards
to update-cp-node.sh k3s-cp-1 "$@" for backwards compat.
- CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc.
- OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and
automation entry point list all point at the new HA-aware artifacts.
Verified: bash -n clean on both scripts; --dry-run walks all six
phases for each of cp-1/cp-2/cp-3; --preflight against live cluster
correctly aborts cp-1 with "install etcdctl on another CP first" and
cp-2/cp-3 with "run --add-swap first" (their current state).
Execution is board-approval-gated; DEV-497 (cp-1-only executor) is
superseded by a successor ticket.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Since DEV-510 landed HA control plane (2026-08-22), the cluster runs cp-1
plus cp-2/cp-3. The previous exclusion in os-update.sh matched `k3s-cp-1`
by hard-coded name only, which would have caused cp-2 and cp-3 to be
treated as regular fsn1 workers and drained/rebooted without the
CP-specific procedure.
Fix: select the CP list from `kubectl get nodes -l
node-role.kubernetes.io/control-plane` and skip any of those nodes. This
covers all present and future CPs automatically.
Also updated OS_UPDATE_PROCEDURE.md topology table and order rule to
document that all three CPs exist and are excluded from the weekly
cycle. The HA-aware CP OS-update procedure is a separate follow-up.
Verified on the current cluster:
[plan] EXCLUDING control-plane nodes: k3s-cp-1 k3s-cp-2 k3s-cp-3
[plan] ordered nodes (6): k3s-worker-4 k3s-update-runner k3s-worker-1
k3s-worker-2 k3s-worker-3 k3s-worker-5
Co-Authored-By: Paperclip <noreply@paperclip.ing>
New freshness (BackupForgejoStale, 28h) and integrity
(BackupForgejoCheckFailed) rules matching the sibling
loki/grafana/k8s-resources/prometheus alerts.
Metrics come from backup_forgejo_{timestamp_seconds,check_status,success}
emitted by apps/forgejo/forgejo-backup-cronjob.yaml.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The weekly rolling OS-update cycle has been silently stripping docker.io from
worker nodes (DEV-498), breaking the Forgejo runner whose hostPath mount for
/var/run/docker.sock requires the socket to exist. Fix in two layers:
- Defensive pin: update-node.sh now runs `apt-mark manual docker.io` in the
apt phase (whenever it is installed) so `apt-get autoremove --purge` cannot
silently drop it during subsequent upgrades.
- Post-reboot reconciliation: new `ensure-node-docker.sh` installs docker.io
if missing, enables + starts the systemd unit, waits for /var/run/docker.sock,
and re-applies the `basicstack.de/docker=true` label. Wired into
update-node.sh between kubelet-Ready and uncordon. No-op on nodes without
the label (safe for cp-1 and the update runner).
Verified idempotent against all 5 labeled workers; `apt-mark manual docker.io`
now set on every worker (survived across reboots by design).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Rolling OS updates removed docker.io from all workers, breaking the
runner's hostPath mount of /var/run/docker.sock (type=Socket check).
- Install docker.io on all 5 worker nodes and enable the systemd unit.
- Label the nodes basicstack.de/docker=true so the dependency is explicit.
- Add nodeSelector to the Deployment so scheduling fails fast on
non-labeled nodes instead of hanging in ContainerCreating.
- Document the node prerequisite in the README.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Follow-up to the cp-1 design commit: update the two remaining places in
OS_UPDATE_PROCEDURE.md that still said "cp-1 last" / "planned as a distinct
issue" so they now name CP1_UPDATE_PROCEDURE.md + update-cp-1.sh.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
cp-1 is excluded from os-update.sh because a one-shot drain would trigger
the kine cascade documented in DEV-495. This adds:
- CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful
eviction (Phase C), drain+apt (Phase D), reboot with external livez
monitor (Phase E), uncordon+verify (Phase F), and rollback paths.
- scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the
same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports
--dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot,
--finalize, --run.
- os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1
script; kine thundering-herd guardrails preserved.
- OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure.
Execution requires separate board approval; this change is design +
dry-run artifact only.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add four hard rules to the OS_UPDATE_PROCEDURE:
1. Pre-plan drain order — no drain that evicts >3 StatefulSets at once
2. cp-1 must have swap before the cycle finishes (currently 0 swap, 3.7 GiB RAM)
3. Halt cycle if `kubectl get nodes` from cp-1 exceeds 5 s (kine slowness leading indicator)
4. cp-1 OS update is a separate design task, not part of standard os-update.sh cycle
Root cause reference: DEV-495 (worker-3 drain 2026-08-16 caused kine SQLite cascade
+ taint-eviction storm + near-OOM on cp-1; cluster self-recovered without operator
action after ~80 min).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Migrates the last plaintext-at-rest monitoring backup off rclone-sync
onto the same restic pipeline as loki/grafana/k8s-resources. Closes
the Hetzner Object Storage gap flagged in DEV-482 — no SSE-S3/SSE-KMS
means every object we push must be encrypted client-side.
- apps/monitoring/prometheus-backup-cronjob.yaml: pin the image to the
Harbor mirror `harbor.basicstack.de/library/restic:0.17.3` (DEV-493)
so the pull path matches the sibling CronJobs.
- docs/monitoring/restic-restore.md: append the DEV-492 drill entry —
cold restic init, 8.017 GiB → 2.441 GiB stored, `restic check`
clean, restore of `latest` into a scratch namespace, then
`promtool tsdb list` + `promtool tsdb analyze` against all 19
blocks (0 failed). Confirms the compaction-race mitigation
(`--exclude wal/*`/`chunks_head/*` + accept exit 3) does not leave
a corrupt snapshot. Also documents the DEV-492 Prometheus repo in
the layout table + per-repo restore commands.
- apps/monitoring/README.md: point the restore section at all four
repos and cite DEV-488 (loki/k8s) + DEV-492 (prometheus) drill
entries.
Definition of done (DEV-492):
- backup ships client-side-encrypted to Hetzner: ✅ (restic init
succeeded on the manual run; snapshot 71420465 written)
- restore drill promtool tsdb analyze clean: ✅ (19/19 blocks OK)
- OBSERVABILITY_BASELINE.md + apps/monitoring/README.md updated: ✅
- restic-password ownership documented: ✅ (README shared-SealedSecret
table already covered all four repos after this change)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Point the three monitoring restic CronJobs at the Harbor-hosted
mirror at harbor.basicstack.de/library/restic:0.17.3 so the backup
pipeline is not gated by Docker Hub throttling or upstream retag.
- Mirrored docker.io/restic/restic:0.17.3 into the public
`library` project via a one-shot crane copy Job. Digest in
Harbor matches the upstream index (sha256:8f5a62b4…).
- Updated the three CronJobs
(backup-loki-restic, backup-grafana-restic,
backup-k8s-resources) to pull from Harbor.
- Documented the tag-bump procedure in
docs/monitoring/restic-restore.md (§ Tag-bump procedure) and
pointed the restore-drill pod at the same Harbor image so the
restore round-trip stays image-consistent with the writers.
Refs: [DEV-493](/DEV/issues/DEV-493), parent [DEV-482](/DEV/issues/DEV-482).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- New PrometheusRule apps/monitoring/backup-restic-alerts.yaml with
seven warning-level rules:
* BackupLokiStale / BackupGrafanaStale / BackupK8sResourcesStale
-> time() - backup_<kind>_timestamp_seconds > 28h
* BackupLokiCheckFailed / BackupGrafanaCheckFailed /
BackupK8sResourcesCheckFailed -> backup_<kind>_check_status != 0
* ResticRepoOversize -> restic_repo_size_bytes > 20 GiB (per-repo)
- Emit restic_repo_size_bytes{repo="<kind>"} from all three restic
CronJobs (loki, grafana, k8s-resources) via `restic stats --json
--mode raw-data` so ResticRepoOversize has data to match once the
textfile-collector scrape path is wired.
- Companion promtool unit test backup-restic-alerts.test.yaml with
five scenarios (fresh/stale, check pass/fail, oversize) -- verified
locally with promtool 2.53.1: SUCCESS.
- README.md: document the three restic CronJobs, the shared
SealedSecret keys (access-key/secret-key/endpoint/bucket/
restic-password), the emitted textfile metrics, and the alert list.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The restic pipeline (loki/grafana/k8s-resources) is proven end-to-end
after the DEV-488 restore drill, so the legacy rsync/tar pipeline into
the 100 Gi local-path `backup-storage` PVC is removed.
- Delete `apps/monitoring/backup-volumes-cronjob.yaml`.
- Remove the DEV-483 bridge `nodeSelector: k3s-worker-2` from
`apps/monitoring/loki-deployment.yaml`. Loki's data protection now
runs via `backup-loki-restic`, which follows the pod via podAffinity
regardless of which node the RWO CSI volume attaches on. The
`Recreate` rollout strategy stays — it is unrelated (avoids the
attach-deadlock during a rollout). Resolves the RWO/nodeSelector
attach race that was blocking DEV-478 weekly OS updates.
- Update `apps/monitoring/README.md` to drop the `backup-volumes`
section, link the restic restore runbook, and record the pin
removal.
- Clean stale coexistence comments in the restic/prometheus CronJob
manifests now that the legacy job is gone.
Cluster-side (already applied out-of-band, since these manifests are
`kubectl apply`-based, not Argo-managed):
- `kubectl -n monitoring delete cronjob backup-volumes` -> NotFound.
- `kubectl -n monitoring delete pvc backup-storage` -> gone; local-path
PV `pvc-d0db0ba9-8f89-4f66-9e65-d573ebe1085a` reclaimed automatically
(Delete policy). Two stale pre-DEV-487 `backup-k8s-resources` job
pods that still referenced the PVC were deleted to release the
`pvc-protection` finalizer.
- `kubectl -n monitoring apply -f loki-deployment.yaml` -> Recreate
rollout, new pod Ready in ~60s, no nodeSelector on the new spec.
- No `VolumeAttachment` for the retired PV.
- Restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
`backup-k8s-resources`, `prometheus-backup`) intact.
Pre-delete snapshots retained on k3s-cp-1 under
`/root/dev489-snapshots-20260816T155411Z/` for post-mortem.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Cover credentials projection via scratch ns + monitoring-s3-backup
copy, per-repo restore commands (loki / k8s-resources / grafana),
repo self-check + source-parity flow, cleanup, and rotation.
- Record first drill (2026-08-16): loki snapshot 97092888 and
k8s-resources snapshot 9b0155cf restored; sha256 parity vs live
loki-storage-encrypted PVC confirmed for all 4 files; cluster.yaml
sanity-checked. Outcome: PASS.
- Next drill target: 2026-11-16 (quarterly); include restic/grafana
once its first CronJob run has produced >= 1 snapshot.
Unblocks retirement of the legacy backup-volumes CronJob (DEV-489).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Rewrites the backup-k8s-resources CronJob to drop the local-path tarball
path and stream a YAML dump through `restic backup --stdin` into
`hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern:
init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3`
reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops
the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling
loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and
textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` /
`_check_status`).
Deployed in parallel with the legacy `backup-volumes` CronJob — the
`backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6
(restore drill).
Server-side dry-run validated on the k3s control plane.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add a daily 03:15 UTC restic backup of the grafana-storage PVC to
s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana. Pinned to
k3s-worker-2 via nodeSelector because grafana-storage is a
local-path PV anchored there — no podAffinity needed. Same restic
retention as the loki sibling (7d/4w/6m + prune + 5% check) with
metrics backup_grafana_{success,timestamp_seconds,check_status}
written to the emptyDir textfile path. Deployed in parallel with
the legacy backup-volumes CronJob (Option 4 rollout, DEV-482).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Step 2 of the DEV-482 Option 4 rollout. Adds a restic-based Loki
backup that streams the loki-storage-encrypted PVC (mounted RO) to
s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki, client-side encrypted by
restic. Co-schedules with the Loki pod via podAffinity so it lands on
whichever worker holds the RWO VolumeAttachment.
Verified with a manual run in-cluster:
- snapshot 98c6fee6 written, listable via a fresh restic pod
- restic check --read-data-subset=5% clean
- resource requests dropped from the plan's 200m/256Mi to 100m/128Mi
because worker-2 has ~150m free CPU (Loki + Grafana + backup-* live
there); limits stay generous for pack/check bursts.
Runs in parallel with the legacy backup-volumes CronJob — DEV-482
step 6 will retire that job only after the restore drill passes.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add key restic-password to SealedSecret monitoring-s3-backup so the
upcoming restic CronJobs (DEV-485/486/487) can mount it as
RESTIC_PASSWORD without any further secret rotation. Value is a fresh
32-byte openssl-random base64 password, sealed strict-scope against
the running sealed-secrets-controller. Server-side dry-run of the
new SealedSecret reports "configured".
Bucket write probe against s3://basicstack-backup/ succeeded from an
in-cluster amazon/aws-cli pod using the existing access-key/secret-key
in the same SealedSecret (ls, put, stat, delete).
Recovery-key copy for Passbolt is held root-only on the control plane
at /root/dev-484/restic-password.txt pending the human-side Passbolt
paste; that step tracks as follow-up.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
RollingUpdate deadlocks the Loki rollout because loki-storage-encrypted
is a Hetzner CSI RWO volume — the new pod on worker-2 cannot attach the
PVC while the old pod on worker-1 still holds it. Recreate drains the
old pod first so the CSI detaches the volume cleanly.
Same pattern as the Harbor fix (commit 56e598f).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Bridge fix so DEV-478 weekly OS-update preflight can proceed while
DEV-482 (Option 4: restic -> Hetzner Object Storage) is designed and
rolled out. Captures the previously out-of-band Loki Deployment as a
manifest and adds nodeSelector kubernetes.io/hostname: k3s-worker-2 so
the loki-storage-encrypted RWO CSI volume co-locates with the
backup-volumes CronJob on worker-2, resolving the RWO/nodeSelector race.
This nodeSelector is temporary. It comes back out as part of DEV-482
step 6 when the legacy backup-volumes CronJob is retired.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Following the DEV-473 rocksdb-to-postgres migration, point Stalwart's bootstrap
config at the PostgreSQL store and expose PGPASSWORD to the container so the
`authSecret: EnvironmentVariable` lookup resolves.
- stalwart-bootstrap-config.yaml: config.json switches @type RocksDb -> PostgreSql
(host=stalwart-postgres, db=stalwart, authUsername=stalwart,
authSecret=EnvironmentVariable/PGPASSWORD, useTls=false).
- stalwart-fresh-deployment.yaml: adds PGPASSWORD env from
secret/stalwart-postgres-credentials.POSTGRES_PASSWORD on the stalwart container.
- stalwart-config.yaml: mirrors the switch in stalwart.toml as documentation
(the running pod uses `--config /etc/stalwart/config.json`, not the toml).
Board (DEV-476) explicitly asked for the manifest-level switch after the web UI
save failed with a `Permission denied` on the read-only bootstrap emptyDir.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The stalwart-backup CronJob has failed nightly since 2026-08-13, all with
FailureTarget=DeadlineExceeded. Root cause: the backup pod had no
scheduling constraint and got placed on a node different from stalwart-0.
The hcloud CSI block volume is RWO and can only be attached to one node,
so the backup pod stayed in ContainerCreating with FailedAttachVolume /
Multi-Attach until the 600s active deadline killed it.
- Add podAffinity requiredDuringScheduling on
app=stalwart,statefulset.kubernetes.io/pod-name=stalwart-0 with topology
key kubernetes.io/hostname so the backup pod always lands on the same
node. Same-node co-location lets both pods share the already-attached
block volume; the in-container script then scales stalwart-0 down,
backs up, and scales it back up as before.
- Raise activeDeadlineSeconds from 600s to 1800s as safety headroom
(successful runs are ~86s; the extra budget covers prune growth).
Verified: manual run of the patched CronJob completed in 86s and wrote
restic snapshot f76f534e (2026-08-15 12:07:30) to
s3://basicstack-backup/stalwart. stalwart-0 is back to Ready.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Phase 2 of the Stalwart RocksDB -> PostgreSQL migration (DEV-467).
Purely additive - does not touch the running stalwart-0 pod, its
ConfigMap, PVC, or bootstrap-config.
- stalwart-postgres.yaml: single-replica StatefulSet with a 10Gi
hcloud-volumes-encrypted PVC (subPath pgdata), ClusterIP Service
on 5432, nodeAffinity csi.hetzner.cloud/location=fsn1 so the DB
co-locates with stalwart-0 (which is fsn1-pinned by its PVC).
Resources match forgejo-postgres (250m/512Mi req, 500m/1Gi lim).
- stalwart-postgres-credentials-sealed.yaml: sealed secret with
POSTGRES_USER=stalwart, POSTGRES_DB=stalwart, POSTGRES_PASSWORD,
plus a copy of the password under stalwart-db-password for
Stalwart's [store.postgres] config in the Phase 4 cutover.
- stalwart-postgres-backup.yaml: daily CronJob at 02:30 UTC that
streams pg_dump | gzip into the existing stalwart-s3-backup restic
repo tagged stalwart-postgres, with independent retention keys
(14d/8w/6m) so it doesn't collide with the RocksDB snapshots.
activeDeadlineSeconds=1800 mirrors the DEV-464 sibling fix.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The original `rclone delete --min-age 7d` step ate the freshly uploaded
backup: rclone preserves each source file's mtime, and Prometheus TSDB
chunk/block files retain very old mtimes (weeks-old immutable blocks),
so filtering by file age deleted ~72% of the objects immediately after
sync (verified: 8.764 GiB destination reduced to 2.310 GiB / 37 objects
before we noticed).
Replaced the mtime prune with a prefix-name-based prune: every top-level
prefix is `prometheus-YYYYMMDD-HHMMSS`, so we parse the encoded date and
`rclone purge` whole prefixes older than 7 days. This keeps the latest
7 daily snapshots intact regardless of the Prometheus block ages.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
After the DEV-464 split, `monitoring/backup-volumes` only backs up
grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving
Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner
Cloud volume attached to whichever node currently runs the Prometheus
pod (typically k3s-worker-1), so it cannot join the shared backup-volumes
job without provoking Multi-Attach errors.
This introduces a dedicated `monitoring/prometheus-backup` CronJob that:
- Streams `prometheus-data-encrypted` to Hetzner S3 via rclone
(`basicstack-backup/prometheus/prometheus-<DATE>/`).
- Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO
PVC always attaches on the same node.
- Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline.
- Retains 7 days of dated backups (rclone delete --min-age 7d).
- Tolerates the expected TSDB compaction race (Prometheus deletes old
block dirs mid-copy): rclone's non-zero exit from those transient
errors is captured, then success is validated by comparing dest
bytes to source bytes (>= 80% and > 100 MiB floor).
S3 credentials are the same Hetzner Object Storage account used by
`opencloud-backup` and `stalwart-backup`, resealed for the `monitoring`
namespace as `SealedSecret monitoring-s3-backup`.
Verified with a manual job on k3s-worker-1 (see DEV-465 for logs).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Create the missing forgejo/platform-backup-data PVC that forgejo-backup
references (20Gi hcloud-volumes-encrypted).
- Record monitoring/backup-k8s-resources with a k3s-worker-2 nodeSelector
(backup-storage PVC is local-path pinned there), lower memory request
(128Mi) so it fits worker-2 pressure, and switch to alpine/k8s image
(bitnami/kubectl is no longer resolvable).
- Rewrite monitoring/backup-volumes to only back up grafana + loki
co-located with backup-storage on k3s-worker-2. Prometheus data
lives on k3s-worker-1 and is intentionally excluded here; a
dedicated Prometheus data backup follows in a separate ticket.
The three CronJobs previously left Pending/ContainerCreating pods that
blocked the OS-update health guard in DEV-463.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Investigation confirmed all SMTP/IMAP ports working correctly as of 21:00 UTC.
SMTP issue at 20:31 UTC coincided with LoadBalancer IP updates (10.42.1.1 → 178.105.17.239).
External port tests verify correct protocol greetings on all ports.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update
procedure (drain -> apt -> reboot -> verify -> uncordon -> health ->
next). Explicit MUST NOT list around k3s config, PVs, and manifests.
- infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip
routine (Sun 03:00 Europe/Berlin) that fires this procedure.
- infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh,
os-update.sh, README. Enforces the same guardrails in code:
workers-first-then-CP, one node at a time, no --force drains, halts on
reboot/kubelet/health failure, never touches k3s config.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Documents the Pangolin + pangolin-kube-controller deployment: architecture,
per-file component map, the two SSO flows (dashboard login vs downstream
resource protection), the ingress-protection request flow, a step-by-step
runbook for adding a new protected ingress, troubleshooting rooted in the
DEV-457 findings, and hard "do not delete" notes on the two PVCs, the
sealed secrets, and the Traefik HelmChartConfig that Pangolin depends on.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Matches what was manually set on the drifted live Application so that
Argo CD reconciles from Forgejo automatically. Non-destructive
(prune: false).
Refs DEV-458.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Harbor's jobservice and registry Deployments have RWO PVCs (Hetzner
Cloud Volumes). The default RollingUpdate strategy deadlocks: the new
pod cannot attach the volume while the old pod still holds it on a
different node. This has left the harbor Argo CD Application in a
Degraded state since 2026-08-07.
The goharbor chart exposes a top-level updateStrategy specifically for
this case ("Set it as Recreate when RWM for volumes isn't supported").
Refs DEV-458.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Pangolin's IngressRoute 1-paperclip-router (pangolin ns) now owns the
paperclip.basicstack.de route (SSO gate via badger middleware). The legacy
Ingress at paperclip/paperclip bypassed Pangolin entirely and produced
"secret paperclip/paperclip-tls does not exist" noise in Traefik.
Cluster deletion follows in this heartbeat; Argo sync policy has no
automated prune, so removal from source is safe first.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds infrastructure/networking/traefik-helmchartconfig.yaml so the k3s
Traefik overrides (badger plugin, allowCrossNamespace, letsencrypt
resolver + persistent acme.json, non-root fsGroup) are tracked in git.
kube-system is not managed by ArgoCD in this cluster; kubectl apply of
this file is the manual reproducibility path.
Refs: DEV-455, DEV-457
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Traefik runs in kube-system, so the short DNS name "pangolin" that Pangolin's
default `internal_hostname` uses does not resolve. Set both
`server.internal_hostname` and `server.badger_override` to
`pangolin.pangolin.svc.cluster.local` so the badger middleware and internal
callbacks resolve from any namespace.
Fixes the HTTP 500 on `https://paperclip.basicstack.de/` after cross-namespace
IngressRoute wiring was fixed in DEV-457.
Refs: DEV-457
Co-Authored-By: Paperclip <noreply@paperclip.ing>
DEV-452. After the SQLite -> Postgres migration in DEV-451 the pangolin
initial setup was redone, so the controller's stale bearer token was
invalid. That was only half the problem: /api/v1/traefik-config is
served exclusively by Pangolin's internal API (port 3001), not the
external dashboard API (port 3000). Pointing a Bearer request at port
3000 always returned 401 because that path lives on the session-auth
router. This has been the underlying cause of the controller's
CrashLoopBackOff, not just the stale key.
Changes:
- pangolin-controller-api-key-sealed.yaml: reseal new bearer token
(kubeseal against sealed-secrets-controller in kube-system, includes
the Bearer prefix expected by the controller).
- pangolin-controller-config.yaml: CONFIG_ENDPOINT now targets
http://pangolin.pangolin.svc.cluster.local:3001/api/v1/traefik-config.
- pangolin-deployment.yaml: Service now exposes port 3001 as the
"internal" port so in-cluster clients (kube-controller) can reach it.
Ingress still only routes / and /api to ports 3002/3000; port 3001
is not published to the internet.
- pangolin-controller-deployment.yaml: replicas back to 1, dropped the
temporary "scaled to 0" comment block.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The DB switch (DEV-451, SQLite -> PostgreSQL) starts Pangolin from a
fresh schema with no admin user or API key. The controller's bearer
token in pangolin-controller-api-key SealedSecret was issued against
the old SQLite state and is now invalid (the controller was already
in CrashLoopBackOff for the same reason).
Scale to 0 replicas so the ArgoCD Application reports Healthy after
the DB migration. Follow-up work will run Pangolin initial setup,
issue a new controller API key, and re-seal the secret before scaling
back to 1.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Pangolin's PostgreSQL build requires postgres.connection_string in
config.yml — DATABASE_URL alone is not honored as an override, so the
container was crashing with "Postgres configuration is missing in the
configuration file".
Render the final config.yml at pod startup via a busybox init container
that substitutes the DATABASE_URL secret into a __DATABASE_URL__
placeholder in the ConfigMap template, then mount the rendered file
into the pangolin container.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Deploy a dedicated PostgreSQL 17 instance in the pangolin namespace
and switch Pangolin (fosrl/pangolin) to the postgresql-1.21.1 image
variant. The database URL is provided via a sealed secret and pulled
into the Pangolin container as DATABASE_URL, which the PostgreSQL
Pangolin build reads at startup.
- New: postgres.yaml (Deployment + PVC on hcloud-volumes-encrypted + Service)
- New: pangolin-postgres-secrets-sealed.yaml (postgres creds + connection string)
- pangolin-deployment.yaml: image -> fosrl/pangolin:postgresql-1.21.1,
DATABASE_URL from secret, /app/config PVC mount, init container waits
for postgres, removed obsolete DATABASE_PATH env
- pangolin-config.yaml: dropped SQLite database.path stanza
Fixes DEV-451.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The previous sealed secret was missing the "Bearer " prefix in the
auth-header value. The controller uses CONFIG_AUTH_HEADER directly
as the Authorization header, so it needs the complete value.
Sealed value now contains: Bearer abw3ud1ipko7ock.2al7fqzhbmfsqp3bm4byvynyx6jnqe733icjdprd
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The previous sealed secret was created with an incorrect API key.
Re-sealed with the correct API key: abw3ud1ipko7ock.2al7fqzhbmfsqp3bm4byvynyx6jnqe733icjdprd
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Sealed the correct auth header value from the controller config into the
sealed secret. This ensures the controller can authenticate properly with
the Pangolin API.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Update sealed API key (abw3ud1ipko7ock...) in controller secret
- Remove insecure plaintext API key from ConfigMap
- Controller now uses only the sealed secret for authentication
No changes to main Pangolin deployment or database configuration.
Safe to deploy - no risk of database reinitialization.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Update CONFIG_ENDPOINT to use port 3000 (API) instead of port 3002 (Web UI)
to resolve empty Traefik configuration issue.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
