The weekly rolling OS-update cycle has been silently stripping docker.io from
worker nodes (DEV-498), breaking the Forgejo runner whose hostPath mount for
/var/run/docker.sock requires the socket to exist. Fix in two layers:
- Defensive pin: update-node.sh now runs `apt-mark manual docker.io` in the
apt phase (whenever it is installed) so `apt-get autoremove --purge` cannot
silently drop it during subsequent upgrades.
- Post-reboot reconciliation: new `ensure-node-docker.sh` installs docker.io
if missing, enables + starts the systemd unit, waits for /var/run/docker.sock,
and re-applies the `basicstack.de/docker=true` label. Wired into
update-node.sh between kubelet-Ready and uncordon. No-op on nodes without
the label (safe for cp-1 and the update runner).
Verified idempotent against all 5 labeled workers; `apt-mark manual docker.io`
now set on every worker (survived across reboots by design).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Rolling OS updates removed docker.io from all workers, breaking the
runner's hostPath mount of /var/run/docker.sock (type=Socket check).
- Install docker.io on all 5 worker nodes and enable the systemd unit.
- Label the nodes basicstack.de/docker=true so the dependency is explicit.
- Add nodeSelector to the Deployment so scheduling fails fast on
non-labeled nodes instead of hanging in ContainerCreating.
- Document the node prerequisite in the README.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Follow-up to the cp-1 design commit: update the two remaining places in
OS_UPDATE_PROCEDURE.md that still said "cp-1 last" / "planned as a distinct
issue" so they now name CP1_UPDATE_PROCEDURE.md + update-cp-1.sh.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
cp-1 is excluded from os-update.sh because a one-shot drain would trigger
the kine cascade documented in DEV-495. This adds:
- CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful
eviction (Phase C), drain+apt (Phase D), reboot with external livez
monitor (Phase E), uncordon+verify (Phase F), and rollback paths.
- scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the
same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports
--dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot,
--finalize, --run.
- os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1
script; kine thundering-herd guardrails preserved.
- OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure.
Execution requires separate board approval; this change is design +
dry-run artifact only.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add four hard rules to the OS_UPDATE_PROCEDURE:
1. Pre-plan drain order — no drain that evicts >3 StatefulSets at once
2. cp-1 must have swap before the cycle finishes (currently 0 swap, 3.7 GiB RAM)
3. Halt cycle if `kubectl get nodes` from cp-1 exceeds 5 s (kine slowness leading indicator)
4. cp-1 OS update is a separate design task, not part of standard os-update.sh cycle
Root cause reference: DEV-495 (worker-3 drain 2026-08-16 caused kine SQLite cascade
+ taint-eviction storm + near-OOM on cp-1; cluster self-recovered without operator
action after ~80 min).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Migrates the last plaintext-at-rest monitoring backup off rclone-sync
onto the same restic pipeline as loki/grafana/k8s-resources. Closes
the Hetzner Object Storage gap flagged in DEV-482 — no SSE-S3/SSE-KMS
means every object we push must be encrypted client-side.
- apps/monitoring/prometheus-backup-cronjob.yaml: pin the image to the
Harbor mirror `harbor.basicstack.de/library/restic:0.17.3` (DEV-493)
so the pull path matches the sibling CronJobs.
- docs/monitoring/restic-restore.md: append the DEV-492 drill entry —
cold restic init, 8.017 GiB → 2.441 GiB stored, `restic check`
clean, restore of `latest` into a scratch namespace, then
`promtool tsdb list` + `promtool tsdb analyze` against all 19
blocks (0 failed). Confirms the compaction-race mitigation
(`--exclude wal/*`/`chunks_head/*` + accept exit 3) does not leave
a corrupt snapshot. Also documents the DEV-492 Prometheus repo in
the layout table + per-repo restore commands.
- apps/monitoring/README.md: point the restore section at all four
repos and cite DEV-488 (loki/k8s) + DEV-492 (prometheus) drill
entries.
Definition of done (DEV-492):
- backup ships client-side-encrypted to Hetzner: ✅ (restic init
succeeded on the manual run; snapshot 71420465 written)
- restore drill promtool tsdb analyze clean: ✅ (19/19 blocks OK)
- OBSERVABILITY_BASELINE.md + apps/monitoring/README.md updated: ✅
- restic-password ownership documented: ✅ (README shared-SealedSecret
table already covered all four repos after this change)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Point the three monitoring restic CronJobs at the Harbor-hosted
mirror at harbor.basicstack.de/library/restic:0.17.3 so the backup
pipeline is not gated by Docker Hub throttling or upstream retag.
- Mirrored docker.io/restic/restic:0.17.3 into the public
`library` project via a one-shot crane copy Job. Digest in
Harbor matches the upstream index (sha256:8f5a62b4…).
- Updated the three CronJobs
(backup-loki-restic, backup-grafana-restic,
backup-k8s-resources) to pull from Harbor.
- Documented the tag-bump procedure in
docs/monitoring/restic-restore.md (§ Tag-bump procedure) and
pointed the restore-drill pod at the same Harbor image so the
restore round-trip stays image-consistent with the writers.
Refs: [DEV-493](/DEV/issues/DEV-493), parent [DEV-482](/DEV/issues/DEV-482).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- New PrometheusRule apps/monitoring/backup-restic-alerts.yaml with
seven warning-level rules:
* BackupLokiStale / BackupGrafanaStale / BackupK8sResourcesStale
-> time() - backup_<kind>_timestamp_seconds > 28h
* BackupLokiCheckFailed / BackupGrafanaCheckFailed /
BackupK8sResourcesCheckFailed -> backup_<kind>_check_status != 0
* ResticRepoOversize -> restic_repo_size_bytes > 20 GiB (per-repo)
- Emit restic_repo_size_bytes{repo="<kind>"} from all three restic
CronJobs (loki, grafana, k8s-resources) via `restic stats --json
--mode raw-data` so ResticRepoOversize has data to match once the
textfile-collector scrape path is wired.
- Companion promtool unit test backup-restic-alerts.test.yaml with
five scenarios (fresh/stale, check pass/fail, oversize) -- verified
locally with promtool 2.53.1: SUCCESS.
- README.md: document the three restic CronJobs, the shared
SealedSecret keys (access-key/secret-key/endpoint/bucket/
restic-password), the emitted textfile metrics, and the alert list.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The restic pipeline (loki/grafana/k8s-resources) is proven end-to-end
after the DEV-488 restore drill, so the legacy rsync/tar pipeline into
the 100 Gi local-path `backup-storage` PVC is removed.
- Delete `apps/monitoring/backup-volumes-cronjob.yaml`.
- Remove the DEV-483 bridge `nodeSelector: k3s-worker-2` from
`apps/monitoring/loki-deployment.yaml`. Loki's data protection now
runs via `backup-loki-restic`, which follows the pod via podAffinity
regardless of which node the RWO CSI volume attaches on. The
`Recreate` rollout strategy stays — it is unrelated (avoids the
attach-deadlock during a rollout). Resolves the RWO/nodeSelector
attach race that was blocking DEV-478 weekly OS updates.
- Update `apps/monitoring/README.md` to drop the `backup-volumes`
section, link the restic restore runbook, and record the pin
removal.
- Clean stale coexistence comments in the restic/prometheus CronJob
manifests now that the legacy job is gone.
Cluster-side (already applied out-of-band, since these manifests are
`kubectl apply`-based, not Argo-managed):
- `kubectl -n monitoring delete cronjob backup-volumes` -> NotFound.
- `kubectl -n monitoring delete pvc backup-storage` -> gone; local-path
PV `pvc-d0db0ba9-8f89-4f66-9e65-d573ebe1085a` reclaimed automatically
(Delete policy). Two stale pre-DEV-487 `backup-k8s-resources` job
pods that still referenced the PVC were deleted to release the
`pvc-protection` finalizer.
- `kubectl -n monitoring apply -f loki-deployment.yaml` -> Recreate
rollout, new pod Ready in ~60s, no nodeSelector on the new spec.
- No `VolumeAttachment` for the retired PV.
- Restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
`backup-k8s-resources`, `prometheus-backup`) intact.
Pre-delete snapshots retained on k3s-cp-1 under
`/root/dev489-snapshots-20260816T155411Z/` for post-mortem.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Cover credentials projection via scratch ns + monitoring-s3-backup
copy, per-repo restore commands (loki / k8s-resources / grafana),
repo self-check + source-parity flow, cleanup, and rotation.
- Record first drill (2026-08-16): loki snapshot 97092888 and
k8s-resources snapshot 9b0155cf restored; sha256 parity vs live
loki-storage-encrypted PVC confirmed for all 4 files; cluster.yaml
sanity-checked. Outcome: PASS.
- Next drill target: 2026-11-16 (quarterly); include restic/grafana
once its first CronJob run has produced >= 1 snapshot.
Unblocks retirement of the legacy backup-volumes CronJob (DEV-489).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Rewrites the backup-k8s-resources CronJob to drop the local-path tarball
path and stream a YAML dump through `restic backup --stdin` into
`hetzner-s3:${BUCKET}/restic/k8s-resources`. Uses a two-container pattern:
init `alpine/k8s:1.29.4` dumps into an emptyDir, main `restic/restic:0.17.3`
reads that file on stdin. Retains `serviceAccountName: backup-sa`, drops
the `k3s-worker-2` nodeSelector (no more local-path pin), matches sibling
loki/grafana jobs on retention, `restic check --read-data-subset=5%`, and
textfile metrics (`backup_k8s_resources_success` / `_timestamp_seconds` /
`_check_status`).
Deployed in parallel with the legacy `backup-volumes` CronJob — the
`backup-storage` PVC keeps serving `backup-volumes` until DEV-482 step 6
(restore drill).
Server-side dry-run validated on the k3s control plane.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add a daily 03:15 UTC restic backup of the grafana-storage PVC to
s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana. Pinned to
k3s-worker-2 via nodeSelector because grafana-storage is a
local-path PV anchored there — no podAffinity needed. Same restic
retention as the loki sibling (7d/4w/6m + prune + 5% check) with
metrics backup_grafana_{success,timestamp_seconds,check_status}
written to the emptyDir textfile path. Deployed in parallel with
the legacy backup-volumes CronJob (Option 4 rollout, DEV-482).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Step 2 of the DEV-482 Option 4 rollout. Adds a restic-based Loki
backup that streams the loki-storage-encrypted PVC (mounted RO) to
s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki, client-side encrypted by
restic. Co-schedules with the Loki pod via podAffinity so it lands on
whichever worker holds the RWO VolumeAttachment.
Verified with a manual run in-cluster:
- snapshot 98c6fee6 written, listable via a fresh restic pod
- restic check --read-data-subset=5% clean
- resource requests dropped from the plan's 200m/256Mi to 100m/128Mi
because worker-2 has ~150m free CPU (Loki + Grafana + backup-* live
there); limits stay generous for pack/check bursts.
Runs in parallel with the legacy backup-volumes CronJob — DEV-482
step 6 will retire that job only after the restore drill passes.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add key restic-password to SealedSecret monitoring-s3-backup so the
upcoming restic CronJobs (DEV-485/486/487) can mount it as
RESTIC_PASSWORD without any further secret rotation. Value is a fresh
32-byte openssl-random base64 password, sealed strict-scope against
the running sealed-secrets-controller. Server-side dry-run of the
new SealedSecret reports "configured".
Bucket write probe against s3://basicstack-backup/ succeeded from an
in-cluster amazon/aws-cli pod using the existing access-key/secret-key
in the same SealedSecret (ls, put, stat, delete).
Recovery-key copy for Passbolt is held root-only on the control plane
at /root/dev-484/restic-password.txt pending the human-side Passbolt
paste; that step tracks as follow-up.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
RollingUpdate deadlocks the Loki rollout because loki-storage-encrypted
is a Hetzner CSI RWO volume — the new pod on worker-2 cannot attach the
PVC while the old pod on worker-1 still holds it. Recreate drains the
old pod first so the CSI detaches the volume cleanly.
Same pattern as the Harbor fix (commit 56e598f).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Bridge fix so DEV-478 weekly OS-update preflight can proceed while
DEV-482 (Option 4: restic -> Hetzner Object Storage) is designed and
rolled out. Captures the previously out-of-band Loki Deployment as a
manifest and adds nodeSelector kubernetes.io/hostname: k3s-worker-2 so
the loki-storage-encrypted RWO CSI volume co-locates with the
backup-volumes CronJob on worker-2, resolving the RWO/nodeSelector race.
This nodeSelector is temporary. It comes back out as part of DEV-482
step 6 when the legacy backup-volumes CronJob is retired.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Following the DEV-473 rocksdb-to-postgres migration, point Stalwart's bootstrap
config at the PostgreSQL store and expose PGPASSWORD to the container so the
`authSecret: EnvironmentVariable` lookup resolves.
- stalwart-bootstrap-config.yaml: config.json switches @type RocksDb -> PostgreSql
(host=stalwart-postgres, db=stalwart, authUsername=stalwart,
authSecret=EnvironmentVariable/PGPASSWORD, useTls=false).
- stalwart-fresh-deployment.yaml: adds PGPASSWORD env from
secret/stalwart-postgres-credentials.POSTGRES_PASSWORD on the stalwart container.
- stalwart-config.yaml: mirrors the switch in stalwart.toml as documentation
(the running pod uses `--config /etc/stalwart/config.json`, not the toml).
Board (DEV-476) explicitly asked for the manifest-level switch after the web UI
save failed with a `Permission denied` on the read-only bootstrap emptyDir.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The stalwart-backup CronJob has failed nightly since 2026-08-13, all with
FailureTarget=DeadlineExceeded. Root cause: the backup pod had no
scheduling constraint and got placed on a node different from stalwart-0.
The hcloud CSI block volume is RWO and can only be attached to one node,
so the backup pod stayed in ContainerCreating with FailedAttachVolume /
Multi-Attach until the 600s active deadline killed it.
- Add podAffinity requiredDuringScheduling on
app=stalwart,statefulset.kubernetes.io/pod-name=stalwart-0 with topology
key kubernetes.io/hostname so the backup pod always lands on the same
node. Same-node co-location lets both pods share the already-attached
block volume; the in-container script then scales stalwart-0 down,
backs up, and scales it back up as before.
- Raise activeDeadlineSeconds from 600s to 1800s as safety headroom
(successful runs are ~86s; the extra budget covers prune growth).
Verified: manual run of the patched CronJob completed in 86s and wrote
restic snapshot f76f534e (2026-08-15 12:07:30) to
s3://basicstack-backup/stalwart. stalwart-0 is back to Ready.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Phase 2 of the Stalwart RocksDB -> PostgreSQL migration (DEV-467).
Purely additive - does not touch the running stalwart-0 pod, its
ConfigMap, PVC, or bootstrap-config.
- stalwart-postgres.yaml: single-replica StatefulSet with a 10Gi
hcloud-volumes-encrypted PVC (subPath pgdata), ClusterIP Service
on 5432, nodeAffinity csi.hetzner.cloud/location=fsn1 so the DB
co-locates with stalwart-0 (which is fsn1-pinned by its PVC).
Resources match forgejo-postgres (250m/512Mi req, 500m/1Gi lim).
- stalwart-postgres-credentials-sealed.yaml: sealed secret with
POSTGRES_USER=stalwart, POSTGRES_DB=stalwart, POSTGRES_PASSWORD,
plus a copy of the password under stalwart-db-password for
Stalwart's [store.postgres] config in the Phase 4 cutover.
- stalwart-postgres-backup.yaml: daily CronJob at 02:30 UTC that
streams pg_dump | gzip into the existing stalwart-s3-backup restic
repo tagged stalwart-postgres, with independent retention keys
(14d/8w/6m) so it doesn't collide with the RocksDB snapshots.
activeDeadlineSeconds=1800 mirrors the DEV-464 sibling fix.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The original `rclone delete --min-age 7d` step ate the freshly uploaded
backup: rclone preserves each source file's mtime, and Prometheus TSDB
chunk/block files retain very old mtimes (weeks-old immutable blocks),
so filtering by file age deleted ~72% of the objects immediately after
sync (verified: 8.764 GiB destination reduced to 2.310 GiB / 37 objects
before we noticed).
Replaced the mtime prune with a prefix-name-based prune: every top-level
prefix is `prometheus-YYYYMMDD-HHMMSS`, so we parse the encoded date and
`rclone purge` whole prefixes older than 7 days. This keeps the latest
7 daily snapshots intact regardless of the Prometheus block ages.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
After the DEV-464 split, `monitoring/backup-volumes` only backs up
grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving
Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner
Cloud volume attached to whichever node currently runs the Prometheus
pod (typically k3s-worker-1), so it cannot join the shared backup-volumes
job without provoking Multi-Attach errors.
This introduces a dedicated `monitoring/prometheus-backup` CronJob that:
- Streams `prometheus-data-encrypted` to Hetzner S3 via rclone
(`basicstack-backup/prometheus/prometheus-<DATE>/`).
- Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO
PVC always attaches on the same node.
- Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline.
- Retains 7 days of dated backups (rclone delete --min-age 7d).
- Tolerates the expected TSDB compaction race (Prometheus deletes old
block dirs mid-copy): rclone's non-zero exit from those transient
errors is captured, then success is validated by comparing dest
bytes to source bytes (>= 80% and > 100 MiB floor).
S3 credentials are the same Hetzner Object Storage account used by
`opencloud-backup` and `stalwart-backup`, resealed for the `monitoring`
namespace as `SealedSecret monitoring-s3-backup`.
Verified with a manual job on k3s-worker-1 (see DEV-465 for logs).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Create the missing forgejo/platform-backup-data PVC that forgejo-backup
references (20Gi hcloud-volumes-encrypted).
- Record monitoring/backup-k8s-resources with a k3s-worker-2 nodeSelector
(backup-storage PVC is local-path pinned there), lower memory request
(128Mi) so it fits worker-2 pressure, and switch to alpine/k8s image
(bitnami/kubectl is no longer resolvable).
- Rewrite monitoring/backup-volumes to only back up grafana + loki
co-located with backup-storage on k3s-worker-2. Prometheus data
lives on k3s-worker-1 and is intentionally excluded here; a
dedicated Prometheus data backup follows in a separate ticket.
The three CronJobs previously left Pending/ContainerCreating pods that
blocked the OS-update health guard in DEV-463.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Investigation confirmed all SMTP/IMAP ports working correctly as of 21:00 UTC.
SMTP issue at 20:31 UTC coincided with LoadBalancer IP updates (10.42.1.1 → 178.105.17.239).
External port tests verify correct protocol greetings on all ports.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- infrastructure/OS_UPDATE_PROCEDURE.md: agent-facing rolling update
procedure (drain -> apt -> reboot -> verify -> uncordon -> health ->
next). Explicit MUST NOT list around k3s config, PVs, and manifests.
- infrastructure/OS_UPDATE_ROUTINE.md: describes the weekly Paperclip
routine (Sun 03:00 Europe/Berlin) that fires this procedure.
- infrastructure/scripts/os-update/: cluster-health.sh, update-node.sh,
os-update.sh, README. Enforces the same guardrails in code:
workers-first-then-CP, one node at a time, no --force drains, halts on
reboot/kubelet/health failure, never touches k3s config.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Documents the Pangolin + pangolin-kube-controller deployment: architecture,
per-file component map, the two SSO flows (dashboard login vs downstream
resource protection), the ingress-protection request flow, a step-by-step
runbook for adding a new protected ingress, troubleshooting rooted in the
DEV-457 findings, and hard "do not delete" notes on the two PVCs, the
sealed secrets, and the Traefik HelmChartConfig that Pangolin depends on.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Matches what was manually set on the drifted live Application so that
Argo CD reconciles from Forgejo automatically. Non-destructive
(prune: false).
Refs DEV-458.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Harbor's jobservice and registry Deployments have RWO PVCs (Hetzner
Cloud Volumes). The default RollingUpdate strategy deadlocks: the new
pod cannot attach the volume while the old pod still holds it on a
different node. This has left the harbor Argo CD Application in a
Degraded state since 2026-08-07.
The goharbor chart exposes a top-level updateStrategy specifically for
this case ("Set it as Recreate when RWM for volumes isn't supported").
Refs DEV-458.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Pangolin's IngressRoute 1-paperclip-router (pangolin ns) now owns the
paperclip.basicstack.de route (SSO gate via badger middleware). The legacy
Ingress at paperclip/paperclip bypassed Pangolin entirely and produced
"secret paperclip/paperclip-tls does not exist" noise in Traefik.
Cluster deletion follows in this heartbeat; Argo sync policy has no
automated prune, so removal from source is safe first.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds infrastructure/networking/traefik-helmchartconfig.yaml so the k3s
Traefik overrides (badger plugin, allowCrossNamespace, letsencrypt
resolver + persistent acme.json, non-root fsGroup) are tracked in git.
kube-system is not managed by ArgoCD in this cluster; kubectl apply of
this file is the manual reproducibility path.
Refs: DEV-455, DEV-457
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Traefik runs in kube-system, so the short DNS name "pangolin" that Pangolin's
default `internal_hostname` uses does not resolve. Set both
`server.internal_hostname` and `server.badger_override` to
`pangolin.pangolin.svc.cluster.local` so the badger middleware and internal
callbacks resolve from any namespace.
Fixes the HTTP 500 on `https://paperclip.basicstack.de/` after cross-namespace
IngressRoute wiring was fixed in DEV-457.
Refs: DEV-457
Co-Authored-By: Paperclip <noreply@paperclip.ing>
DEV-452. After the SQLite -> Postgres migration in DEV-451 the pangolin
initial setup was redone, so the controller's stale bearer token was
invalid. That was only half the problem: /api/v1/traefik-config is
served exclusively by Pangolin's internal API (port 3001), not the
external dashboard API (port 3000). Pointing a Bearer request at port
3000 always returned 401 because that path lives on the session-auth
router. This has been the underlying cause of the controller's
CrashLoopBackOff, not just the stale key.
Changes:
- pangolin-controller-api-key-sealed.yaml: reseal new bearer token
(kubeseal against sealed-secrets-controller in kube-system, includes
the Bearer prefix expected by the controller).
- pangolin-controller-config.yaml: CONFIG_ENDPOINT now targets
http://pangolin.pangolin.svc.cluster.local:3001/api/v1/traefik-config.
- pangolin-deployment.yaml: Service now exposes port 3001 as the
"internal" port so in-cluster clients (kube-controller) can reach it.
Ingress still only routes / and /api to ports 3002/3000; port 3001
is not published to the internet.
- pangolin-controller-deployment.yaml: replicas back to 1, dropped the
temporary "scaled to 0" comment block.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The DB switch (DEV-451, SQLite -> PostgreSQL) starts Pangolin from a
fresh schema with no admin user or API key. The controller's bearer
token in pangolin-controller-api-key SealedSecret was issued against
the old SQLite state and is now invalid (the controller was already
in CrashLoopBackOff for the same reason).
Scale to 0 replicas so the ArgoCD Application reports Healthy after
the DB migration. Follow-up work will run Pangolin initial setup,
issue a new controller API key, and re-seal the secret before scaling
back to 1.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Pangolin's PostgreSQL build requires postgres.connection_string in
config.yml — DATABASE_URL alone is not honored as an override, so the
container was crashing with "Postgres configuration is missing in the
configuration file".
Render the final config.yml at pod startup via a busybox init container
that substitutes the DATABASE_URL secret into a __DATABASE_URL__
placeholder in the ConfigMap template, then mount the rendered file
into the pangolin container.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Deploy a dedicated PostgreSQL 17 instance in the pangolin namespace
and switch Pangolin (fosrl/pangolin) to the postgresql-1.21.1 image
variant. The database URL is provided via a sealed secret and pulled
into the Pangolin container as DATABASE_URL, which the PostgreSQL
Pangolin build reads at startup.
- New: postgres.yaml (Deployment + PVC on hcloud-volumes-encrypted + Service)
- New: pangolin-postgres-secrets-sealed.yaml (postgres creds + connection string)
- pangolin-deployment.yaml: image -> fosrl/pangolin:postgresql-1.21.1,
DATABASE_URL from secret, /app/config PVC mount, init container waits
for postgres, removed obsolete DATABASE_PATH env
- pangolin-config.yaml: dropped SQLite database.path stanza
Fixes DEV-451.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The previous sealed secret was missing the "Bearer " prefix in the
auth-header value. The controller uses CONFIG_AUTH_HEADER directly
as the Authorization header, so it needs the complete value.
Sealed value now contains: Bearer abw3ud1ipko7ock.2al7fqzhbmfsqp3bm4byvynyx6jnqe733icjdprd
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The previous sealed secret was created with an incorrect API key.
Re-sealed with the correct API key: abw3ud1ipko7ock.2al7fqzhbmfsqp3bm4byvynyx6jnqe733icjdprd
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Sealed the correct auth header value from the controller config into the
sealed secret. This ensures the controller can authenticate properly with
the Pangolin API.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Update sealed API key (abw3ud1ipko7ock...) in controller secret
- Remove insecure plaintext API key from ConfigMap
- Controller now uses only the sealed secret for authentication
No changes to main Pangolin deployment or database configuration.
Safe to deploy - no risk of database reinitialization.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Update CONFIG_ENDPOINT to use port 3000 (API) instead of port 3002 (Web UI)
to resolve empty Traefik configuration issue.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The controller uses leader election with replicas=1. During RollingUpdate,
the new pod cannot acquire the leader lease (old pod holds it) and fails
health checks, causing indefinite restart loops.
Recreate strategy ensures the old pod terminates before the new one starts,
allowing clean leader election transitions.
Fixes: DEV-442
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds pangolin-controller-api-key sealed secret with Bearer token
for authenticating to Pangolin API. This enables the controller
to fetch Traefik configuration from Pangolin service.
Resolves: DEV-441
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This guide provides step-by-step instructions for configuring Pangolin
Community Edition to protect the paperclip.basicstack.de domain with
authentication and access control. It uses application-level policies
instead of enterprise-only global policies.
Related to DEV-400.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Add Hetzner Load Balancer (138.199.128.63) to network architecture diagram
- Update DNS configuration to show all domains pointing to Hetzner LB IP
- Update HTTP/HTTPS traffic flow to show traffic routing through Hetzner LB
- Update last modified date to 2026-08-02
Related to DEV-439.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added comprehensive documentation of the two-tier load balancing setup:
- Hetzner Cloud Load Balancer (external layer, managed by Hetzner CCM)
- Kubernetes LoadBalancer services (internal layer, k3s ServiceLB)
Key points documented:
- Traffic flow from external client through both LB layers to pod
- Why LoadBalancer service type is required (CCM integration)
- Historical context of the migration from hostPort to Hetzner LB
- Service definitions and port configurations
Updated:
- apps/stalwart/README.md: Added Network Architecture section
- infrastructure/networking/NETWORK_ARCHITECTURE.md: Enhanced Stalwart
section with two-tier architecture details and updated traffic flows
Resolves documentation gap identified in DEV-439.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Configure pangolin-kube-controller to authenticate with Pangolin API
using the provided API key.
Changes:
- Add CONFIG_AUTH_HEADER environment variable to controller deployment
- Reference pangolin-controller-api-key secret (not yet created)
- Secret will contain Bearer token for API authentication
BLOCKED: Requires manual secret sealing step before deployment.
To complete this deployment, run on a machine with cluster access:
kubectl create secret generic pangolin-controller-api-key \
--namespace=pangolin \
--from-literal=auth-header="Bearer 5qid06u9j325kpk.ywd3bpsx34dtxyczgatyxuoxkzwhie7d72k6v4hw" \
--dry-run=client -o yaml | \
kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \
--format=yaml > apps/pangolin/pangolin-controller-api-key-sealed.yaml
Then commit the sealed secret and push both files.
Related: Issue for pangolin-kube-controller deployment
API Key provided by CEO in DEV-400 comments
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add comprehensive Network Architecture section explaining:
- Two-tier load balancing (K8s LoadBalancer + Hetzner LB)
- Why LoadBalancer service type is required (Hetzner CCM auto-management)
- Traffic flow from external clients to Stalwart pods
- Warning against changing to NodePort (breaks CCM integration)
This documents the investigation from DEV-439, confirming that the current
LoadBalancer service setup is correct and necessary for our infrastructure.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
