Vikunja 2.6.0 deprecates `service.jwtsecret` in favor of `service.secret`;
switch the env var name (secret payload unchanged) and fix the admin-promotion
example in the README to the current CLI (`user set-admin --admin`).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Namespace-scoped Postgres 16 + unified Vikunja container behind Traefik
ingress at vikunja.basicstack.de. Both pods pinned to Hetzner fsn1 due to
their encrypted PVCs (10 Gi DB, 10 Gi attachments). Local auth disabled;
sign-in only via Pocket ID (client `Vikunja`, groups `vikunja_admins` /
`vikunja_members`). SMTP mailer left disabled pending mailbox provisioning.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Hand-rolled manifests under apps/kaneo/ + Argo Application:
- Dedicated PostgreSQL 16-alpine on hcloud-volumes-encrypted (20Gi),
pinned to fsn1 via csi.hetzner.cloud/location node affinity.
- Kaneo v2.22.0 (single ghcr.io/usekaneo/kaneo image; API + web on
5173). Pinned to fsn1 for consistency with the volume.
- Traefik ingress at kaneo.basicstack.de with letsencrypt-prod cert.
- Better Auth JWT secret and Postgres credentials as SealedSecrets.
- OIDC via Pocket ID using CUSTOM_OAUTH_*, restricted to the two
new Pocket ID groups kaneo-admins / kaneo-members (client id
d58ff9bb-dc98-49b4-9535-4af2aba66de2).
- Registration + password + email-OTP sign-in disabled; SSO only.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- New namespace grist with in-namespace PostgreSQL 16
- Ingress grist.basicstack.de + letsencrypt-prod (DNS-01 via Hetzner)
- Pocket ID OIDC login (client "Grist", group "grist")
- All PVCs on hcloud-volumes-encrypted
- ArgoCD application via apps/app-grist.yaml
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The scheduled backup pod could not start because worker-3 (where the
opencloud data PVC is attached and where podAffinity pins the backup)
had ~40Mi of memory left by requests after opencloud/tika were
co-scheduled. rclone actually uses well under 100Mi during the sync.
Drop cpu request 100m -> 50m and memory request 256Mi -> 64Mi. Limits
unchanged (500m / 512Mi) so it can still burst.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- provision-k3s-worker.sh now sets --node-ip=<private-ip> on the agent
install so k3s does not default to the public IP (DEV-224 root cause)
- K3S_OPERATIONS.md node table reflects the current cluster (cp-2/cp-3,
worker-4/5/6), documents the Hetzner Cloud Volume datacenter constraint,
and calls out the --node-ip requirement in the manual join steps
Co-Authored-By: Paperclip <noreply@paperclip.ing>
os-update.sh now walks all three CPs after workers, one at a time, etcd
followers first and leader last. Each CP is delegated to update-cp-node.sh
--run with ASSUME_YES=1; a health gate (cluster-health.sh +
etcdctl endpoint status) runs between CPs. Adds --skip-cps and --only-cps
flags; --only <node> and --start-from <node> now route to the correct list
automatically.
OS_UPDATE_PROCEDURE.md updated: CPs are no longer excluded, §8 renamed to
"Control plane handling" and describes the leader-last cycle + delegation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Mirrored docker.io/restic/restic:0.19.1 to
harbor.basicstack.de/library/restic:0.19.1 (crane in-cluster Job).
Updated all CronJob pins and the restore-drill/tag-bump docs.
Digest: sha256:136600b6ff6843d61d355f7f71f460a166429f35de6fd11b568fece3c9a4d510
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The 7.2.0 -> 7.4.0 rollout stalled because the default RollingUpdate
tried to start the new pod before terminating the old one, but they
both mount opencloud-data (RWO), causing Multi-Attach errors.
Switch to strategy: Recreate so the old pod is terminated first and
the volume detaches cleanly before the new pod starts. Same fix
pattern as bookstack (DEV-528) and other RWO-backed apps.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Skimmed 7.3.0 (2026-07-14) and 7.4.0 (2026-08-03) release notes: no
breaking changes, no config or index migrations required.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Minor version bump of the cluster OIDC provider. No breaking changes;
multi-client-secret support in 2.14 is additive; SameSite=Lax cookie
default in 2.13 is standard. DB snapshot captured pre-rollout.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Patch bump per DEV-520 plan. Only pangolin uses postgres:17.5*;
other apps (paperclip, directus, forgejo, stalwart) are on
postgres:16-alpine and are out of scope for this ticket.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The Job self-deletes after ttlSecondsAfterFinished=3600, causing ArgoCD
to report the app permanently OutOfSync. The job body only hits
/healthz/live and prints a manual TODO; it never modified Stalwart
config. The real IP allowlist is already declared in
apps/stalwart/stalwart-config.yaml under
[server.listener.*.security] allowed-ips.
Also removes the orphan ServiceAccount/Role/RoleBinding
stalwart-config-access bundled in the same file.
Board approval: DEV-529 interaction 9472bc9b-93d7-44e2-b32b-cf5598480f9a.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
BookStack is a single-replica Deployment backed by a ReadWriteOnce
PVC (`bookstack-config`). The default RollingUpdate strategy creates
the new pod before terminating the old one, and the two pods cannot
share the RWO volume — so every rollout deadlocks on
`Multi-Attach error for volume`.
Switching to `strategy: Recreate` terminates the old pod first,
releases the PVC, and then starts the new pod. Brief downtime is
expected on rollout, but rollouts actually complete.
Discovered while rolling out the `/status` probe fix (previous commit).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
BookStack's `/` returns 302 → /login, which caused Kubernetes to
raise ProbeWarning events on the readiness (and liveness) probes.
`/status` is BookStack's built-in monitoring endpoint: it returns
200 without a redirect and is designed for health checks.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Argo CD only substitutes $secret:key references from Secrets carrying
the label app.kubernetes.io/part-of: argocd. Without it, the server
logs "secret key does not exist in secret" and renders placeholders
verbatim into the OIDC redirect URL.
Adding the label to the SealedSecret template ensures sealed-secrets-
controller re-produces the Secret with the label on every restore, so
OIDC keeps working after DR / re-seal.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Post-migration verify uncovered that Pocket ID OIDC login was broken:
the argocd-server was rendering $oidc.pocketid.clientId literally into
the authorize URL instead of substituting the client id.
The Helm chart's default $key syntax looks in argocd-secret, but our
OIDC keys are held only in the SealedSecret-backed argocd-oidc-secret
Opaque secret. Pre-migration argocd-secret happened to contain byte-for-
byte copies of those keys (legacy install), which is why it worked.
Switch to Argo CD's $secret:key form so the values point at the correct
secret without duplicating sealed material into argocd-secret.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Argo CD server was returning HTTP 307 redirect loop behind Traefik
because it was serving HTTPS on port 8080 while the ingress terminates
TLS. Setting server.insecure so the server speaks plain HTTP internally.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Argo CD repo-server needs the forgejo.forgejo.svc.cluster.local host key
in its known_hosts to clone the git repos over SSH. Adding it to
configs.ssh.extraHosts so the helm-managed ssh-known-hosts-cm renders
the entry from git and does not drift when the argocd Application syncs.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Replace the vendored ~33k-line apps/argocd/argocd-install.yaml with the
argoproj/argo-helm chart argo-cd 10.4.0 (app v3.5.1). Values live in
apps/argocd/values.yaml; the local kustomize wrapper now only carries the
Traefik ingress and the sealed secrets. The root apps/app-argocd.yaml
Application becomes multi-source (chart + this repo as $values), enables
ServerSideApply + ApplyOutOfSyncOnly, and pins the resources-finalizer
explicitly.
Behavior-equivalent to the previous install: same URL, OIDC (Pocket ID),
argo_admins RBAC mapping, resource.exclusions list, and per-component
memory limits (DEV-281). Ingress is disabled in the chart; ours stays in
kustomize with cert-manager letsencrypt-prod annotations.
README.md updated with the Helm bump procedure. argocd-install.yaml
removed.
Verified locally:
helm template argocd argo/argo-cd --version 10.4.0 \
-f apps/argocd/values.yaml -n argocd
# renders 34k lines, image: quay.io/argoproj/argocd:v3.5.1
kustomize build apps/argocd/
# renders 1 Ingress + 3 SealedSecrets, no errors
Refs: DEV-521, plan DEV-519 §3, §8.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an
HA-aware procedure that covers all three CPs. Rationale: after DEV-510
the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs
are NoSchedule-tainted, so:
- Kine cascade is no longer the driving risk; embedded 3-node etcd
tolerates one member down. The kubectl-latency guardrail stays as a
soft signal.
- CPs host no StatefulSets or single-replica Deployments. The old
Phase C batched stateful eviction dance is obsolete.
- CP reboots no longer lose the api-server; two peers keep serving.
/livez is polled from the operator machine and cross-checked with
etcdctl endpoint status from a peer CP.
New CP ordering rule: one CP at a time, leader last, health gate
between CPs. Refuses to start if another CP is already cordoned.
Automation:
- New scripts/os-update/update-cp-node.sh <node> is the entry point.
Supports --add-swap / --preflight / --drain / --apt / --reboot /
--finalize / --run / --dry-run. Phase A also installs etcd-client
on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the
target. Peer picker prefers a CP that has etcdctl; refuses cp-1
target with a targeted error if no other CP has etcdctl yet.
- scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards
to update-cp-node.sh k3s-cp-1 "$@" for backwards compat.
- CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc.
- OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and
automation entry point list all point at the new HA-aware artifacts.
Verified: bash -n clean on both scripts; --dry-run walks all six
phases for each of cp-1/cp-2/cp-3; --preflight against live cluster
correctly aborts cp-1 with "install etcdctl on another CP first" and
cp-2/cp-3 with "run --add-swap first" (their current state).
Execution is board-approval-gated; DEV-497 (cp-1-only executor) is
superseded by a successor ticket.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Since DEV-510 landed HA control plane (2026-08-22), the cluster runs cp-1
plus cp-2/cp-3. The previous exclusion in os-update.sh matched `k3s-cp-1`
by hard-coded name only, which would have caused cp-2 and cp-3 to be
treated as regular fsn1 workers and drained/rebooted without the
CP-specific procedure.
Fix: select the CP list from `kubectl get nodes -l
node-role.kubernetes.io/control-plane` and skip any of those nodes. This
covers all present and future CPs automatically.
Also updated OS_UPDATE_PROCEDURE.md topology table and order rule to
document that all three CPs exist and are excluded from the weekly
cycle. The HA-aware CP OS-update procedure is a separate follow-up.
Verified on the current cluster:
[plan] EXCLUDING control-plane nodes: k3s-cp-1 k3s-cp-2 k3s-cp-3
[plan] ordered nodes (6): k3s-worker-4 k3s-update-runner k3s-worker-1
k3s-worker-2 k3s-worker-3 k3s-worker-5
Co-Authored-By: Paperclip <noreply@paperclip.ing>
New freshness (BackupForgejoStale, 28h) and integrity
(BackupForgejoCheckFailed) rules matching the sibling
loki/grafana/k8s-resources/prometheus alerts.
Metrics come from backup_forgejo_{timestamp_seconds,check_status,success}
emitted by apps/forgejo/forgejo-backup-cronjob.yaml.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The weekly rolling OS-update cycle has been silently stripping docker.io from
worker nodes (DEV-498), breaking the Forgejo runner whose hostPath mount for
/var/run/docker.sock requires the socket to exist. Fix in two layers:
- Defensive pin: update-node.sh now runs `apt-mark manual docker.io` in the
apt phase (whenever it is installed) so `apt-get autoremove --purge` cannot
silently drop it during subsequent upgrades.
- Post-reboot reconciliation: new `ensure-node-docker.sh` installs docker.io
if missing, enables + starts the systemd unit, waits for /var/run/docker.sock,
and re-applies the `basicstack.de/docker=true` label. Wired into
update-node.sh between kubelet-Ready and uncordon. No-op on nodes without
the label (safe for cp-1 and the update runner).
Verified idempotent against all 5 labeled workers; `apt-mark manual docker.io`
now set on every worker (survived across reboots by design).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Rolling OS updates removed docker.io from all workers, breaking the
runner's hostPath mount of /var/run/docker.sock (type=Socket check).
- Install docker.io on all 5 worker nodes and enable the systemd unit.
- Label the nodes basicstack.de/docker=true so the dependency is explicit.
- Add nodeSelector to the Deployment so scheduling fails fast on
non-labeled nodes instead of hanging in ContainerCreating.
- Document the node prerequisite in the README.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Follow-up to the cp-1 design commit: update the two remaining places in
OS_UPDATE_PROCEDURE.md that still said "cp-1 last" / "planned as a distinct
issue" so they now name CP1_UPDATE_PROCEDURE.md + update-cp-1.sh.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
cp-1 is excluded from os-update.sh because a one-shot drain would trigger
the kine cascade documented in DEV-495. This adds:
- CP1_UPDATE_PROCEDURE.md: swap-add (Phase A), preflight, batched stateful
eviction (Phase C), drain+apt (Phase D), reboot with external livez
monitor (Phase E), uncordon+verify (Phase F), and rollback paths.
- scripts/os-update/update-cp-1.sh: subcommand-per-phase runner with the
same /tmp/os-update-cp-1-<ts>.log contract as update-node.sh; supports
--dry-run, --add-swap, --preflight, --drain-stateful, --apt, --reboot,
--finalize, --run.
- os-update.sh: explicitly excludes k3s-cp-1 with a pointer to the cp-1
script; kine thundering-herd guardrails preserved.
- OS_UPDATE_PROCEDURE.md: cross-reference to the cp-1 procedure.
Execution requires separate board approval; this change is design +
dry-run artifact only.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add four hard rules to the OS_UPDATE_PROCEDURE:
1. Pre-plan drain order — no drain that evicts >3 StatefulSets at once
2. cp-1 must have swap before the cycle finishes (currently 0 swap, 3.7 GiB RAM)
3. Halt cycle if `kubectl get nodes` from cp-1 exceeds 5 s (kine slowness leading indicator)
4. cp-1 OS update is a separate design task, not part of standard os-update.sh cycle
Root cause reference: DEV-495 (worker-3 drain 2026-08-16 caused kine SQLite cascade
+ taint-eviction storm + near-OOM on cp-1; cluster self-recovered without operator
action after ~80 min).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Migrates the last plaintext-at-rest monitoring backup off rclone-sync
onto the same restic pipeline as loki/grafana/k8s-resources. Closes
the Hetzner Object Storage gap flagged in DEV-482 — no SSE-S3/SSE-KMS
means every object we push must be encrypted client-side.
- apps/monitoring/prometheus-backup-cronjob.yaml: pin the image to the
Harbor mirror `harbor.basicstack.de/library/restic:0.17.3` (DEV-493)
so the pull path matches the sibling CronJobs.
- docs/monitoring/restic-restore.md: append the DEV-492 drill entry —
cold restic init, 8.017 GiB → 2.441 GiB stored, `restic check`
clean, restore of `latest` into a scratch namespace, then
`promtool tsdb list` + `promtool tsdb analyze` against all 19
blocks (0 failed). Confirms the compaction-race mitigation
(`--exclude wal/*`/`chunks_head/*` + accept exit 3) does not leave
a corrupt snapshot. Also documents the DEV-492 Prometheus repo in
the layout table + per-repo restore commands.
- apps/monitoring/README.md: point the restore section at all four
repos and cite DEV-488 (loki/k8s) + DEV-492 (prometheus) drill
entries.
Definition of done (DEV-492):
- backup ships client-side-encrypted to Hetzner: ✅ (restic init
succeeded on the manual run; snapshot 71420465 written)
- restore drill promtool tsdb analyze clean: ✅ (19/19 blocks OK)
- OBSERVABILITY_BASELINE.md + apps/monitoring/README.md updated: ✅
- restic-password ownership documented: ✅ (README shared-SealedSecret
table already covered all four repos after this change)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Turns the four monitoring backup CronJobs' .prom output into scrapeable
Prometheus series so the DEV-490 alerts finally evaluate against live data.
- apps/observability/patches/node-exporter-textfile-collector.yaml:
strategic-merge patch on the kube-prometheus-stack node-exporter DS
that adds `--collector.textfile.directory=/host/textfile_collector`
and mounts `/var/lib/node_exporter/textfile_collector` read-only.
Chart isn't tracked in ArgoCD, so we keep the patch under version
control and re-apply after any helm upgrade (see patches/README.md).
- apps/monitoring/backup-{loki,grafana,k8s-resources,prometheus}-*-cronjob.yaml:
swap `emptyDir` /metrics for a `hostPath` on the same directory
(`DirectoryOrCreate`). Write via `.prom.tmp` + `mv` so node-exporter
never reads a truncated sample.
- apps/monitoring/backup-restic-alerts.yaml: `time() - max(...) > 28h`
for all four freshness alerts so a stale `.prom` left on a node the
job has since left does not fire the freshness pager.
- apps/monitoring/README.md: drop the "once wired" caveat; document
the on-node directory, atomic write, cross-node staleness rationale.
Verified end-to-end with a synthetic `kubectl create job
--from=cronjob/backup-k8s-resources`: pod ran on k3s-worker-4,
.prom file materialized in /var/lib/node_exporter/textfile_collector,
and Prometheus returned all four metric families
(`backup_k8s_resources_success=1`, `..._timestamp_seconds`,
`..._check_status=0`, `restic_repo_size_bytes{repo="k8s-resources"}=13692516`).
`time() - max(backup_k8s_resources_timestamp_seconds)` returned ~84s
against a fresh run. `promtool check rules` + `promtool test rules`
still pass (9 rules, 5 scenarios).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Point the three monitoring restic CronJobs at the Harbor-hosted
mirror at harbor.basicstack.de/library/restic:0.17.3 so the backup
pipeline is not gated by Docker Hub throttling or upstream retag.
- Mirrored docker.io/restic/restic:0.17.3 into the public
`library` project via a one-shot crane copy Job. Digest in
Harbor matches the upstream index (sha256:8f5a62b4…).
- Updated the three CronJobs
(backup-loki-restic, backup-grafana-restic,
backup-k8s-resources) to pull from Harbor.
- Documented the tag-bump procedure in
docs/monitoring/restic-restore.md (§ Tag-bump procedure) and
pointed the restore-drill pod at the same Harbor image so the
restore round-trip stays image-consistent with the writers.
Refs: [DEV-493](/DEV/issues/DEV-493), parent [DEV-482](/DEV/issues/DEV-482).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- New PrometheusRule apps/monitoring/backup-restic-alerts.yaml with
seven warning-level rules:
* BackupLokiStale / BackupGrafanaStale / BackupK8sResourcesStale
-> time() - backup_<kind>_timestamp_seconds > 28h
* BackupLokiCheckFailed / BackupGrafanaCheckFailed /
BackupK8sResourcesCheckFailed -> backup_<kind>_check_status != 0
* ResticRepoOversize -> restic_repo_size_bytes > 20 GiB (per-repo)
- Emit restic_repo_size_bytes{repo="<kind>"} from all three restic
CronJobs (loki, grafana, k8s-resources) via `restic stats --json
--mode raw-data` so ResticRepoOversize has data to match once the
textfile-collector scrape path is wired.
- Companion promtool unit test backup-restic-alerts.test.yaml with
five scenarios (fresh/stale, check pass/fail, oversize) -- verified
locally with promtool 2.53.1: SUCCESS.
- README.md: document the three restic CronJobs, the shared
SealedSecret keys (access-key/secret-key/endpoint/bucket/
restic-password), the emitted textfile metrics, and the alert list.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The restic pipeline (loki/grafana/k8s-resources) is proven end-to-end
after the DEV-488 restore drill, so the legacy rsync/tar pipeline into
the 100 Gi local-path `backup-storage` PVC is removed.
- Delete `apps/monitoring/backup-volumes-cronjob.yaml`.
- Remove the DEV-483 bridge `nodeSelector: k3s-worker-2` from
`apps/monitoring/loki-deployment.yaml`. Loki's data protection now
runs via `backup-loki-restic`, which follows the pod via podAffinity
regardless of which node the RWO CSI volume attaches on. The
`Recreate` rollout strategy stays — it is unrelated (avoids the
attach-deadlock during a rollout). Resolves the RWO/nodeSelector
attach race that was blocking DEV-478 weekly OS updates.
- Update `apps/monitoring/README.md` to drop the `backup-volumes`
section, link the restic restore runbook, and record the pin
removal.
- Clean stale coexistence comments in the restic/prometheus CronJob
manifests now that the legacy job is gone.
Cluster-side (already applied out-of-band, since these manifests are
`kubectl apply`-based, not Argo-managed):
- `kubectl -n monitoring delete cronjob backup-volumes` -> NotFound.
- `kubectl -n monitoring delete pvc backup-storage` -> gone; local-path
PV `pvc-d0db0ba9-8f89-4f66-9e65-d573ebe1085a` reclaimed automatically
(Delete policy). Two stale pre-DEV-487 `backup-k8s-resources` job
pods that still referenced the PVC were deleted to release the
`pvc-protection` finalizer.
- `kubectl -n monitoring apply -f loki-deployment.yaml` -> Recreate
rollout, new pod Ready in ~60s, no nodeSelector on the new spec.
- No `VolumeAttachment` for the retired PV.
- Restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`,
`backup-k8s-resources`, `prometheus-backup`) intact.
Pre-delete snapshots retained on k3s-cp-1 under
`/root/dev489-snapshots-20260816T155411Z/` for post-mortem.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Cover credentials projection via scratch ns + monitoring-s3-backup
copy, per-repo restore commands (loki / k8s-resources / grafana),
repo self-check + source-parity flow, cleanup, and rotation.
- Record first drill (2026-08-16): loki snapshot 97092888 and
k8s-resources snapshot 9b0155cf restored; sha256 parity vs live
loki-storage-encrypted PVC confirmed for all 4 files; cluster.yaml
sanity-checked. Outcome: PASS.
- Next drill target: 2026-11-16 (quarterly); include restic/grafana
once its first CronJob run has produced >= 1 snapshot.
Unblocks retirement of the legacy backup-volumes CronJob (DEV-489).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
