Container was repeatedly OOMKilled (exit 137) every ~5 s of live time since
2026-10-11 (pod age 6d23h, 1919 restarts). Live steady-state memory is now
~321 MiB with 1Gi/512Mi settings, so comfortable headroom. Patched the live
StatefulSet first to recover the control plane; this commit lets GitOps
retain the fix on the next reconcile.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Worker-2 CPU requests were pinned at 100% (2000m/2000m), leaving no room
for the daily opencloud-backup CronJob (needs 20m). Actual opencloud CPU
usage is ~3m — reserving 250m was 80x over actual. Reduce to 50m which is
still 15x headroom. Frees 200m on worker-2.
Backup CronJob is pinned to the opencloud node by RWO PVC + hard
podAffinity. Reducing opencloud's own CPU request is the cleanest
unblock — contained to the opencloud namespace, no cross-app blast
radius.
Related: DEV-593, DEV-464, DEV-481, DEV-514, DEV-582/584 (recurring
pattern of backup CronJob squeezed off worker-2).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds apps/monitoring/prometheus-deployment.yaml capturing the live PVC/Service/
Deployment for monitoring/prometheus. Two behaviour changes vs the previous
live state:
- strategy: RollingUpdate -> Recreate. prometheus-data-encrypted is RWO on
hcloud-volumes-encrypted; a rolling update deadlocks because the surge pod
cannot attach the PVC while the old pod holds it. This has been jamming the
weekly OS-update preflight (see DEV-582). Same fix pattern as loki
(loki-deployment.yaml) and opencloud (DEV-538).
- pod-template annotation backup.velero.io/backup-volumes-excludes:
prometheus-storage. Restores the annotation that rev 6 tried to add before
the RollingUpdate deadlock forced a rollback in DEV-582.
Applied and verified: rollout succeeded, /-/ready and /-/healthy both green,
pod running on k3s-worker-6 after PVC re-attach.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Velero cluster-all now covers cluster-manifest backup with two successive
Completed runs verified and one restore drill (openwebui) green (DEV-575).
- Delete apps/monitoring/backup-k8s-resources-cronjob.yaml
- Drop BackupK8sResourcesStale + BackupK8sResourcesCheckFailed from
backup-restic-alerts.yaml and matching series/cases from the promtool
test file (all 9 remaining rules pass `promtool test rules`)
- Update apps/monitoring/README.md + prometheus-backup-cronjob comments
- Runbook lives in BookStack: EN
https://books.basicstack.de/books/it-operations-manual/page/velero-backup-restore-runbook
DE
https://books.basicstack.de/books/it-betriebshandbuch/page/velero-backup-und-restore-runbook
Rollback: revert this commit — Argo restores the CronJob + alerts and
restic k8s-resources runs resume at the next 02:00 UTC schedule.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- CronJob at 0 4 1 * * restores the latest openwebui-manifests backup into
a transient restore-test-<ts> namespace, diffs core resource-kind counts
against the live openwebui namespace, and deletes the target namespace.
- ServiceAccount + ClusterRole scoped to velero.io + namespace + configmap
ops. Script enforces the 'restore-test-' name prefix before deleting a
namespace. Cluster-wide 'view' bound for the diff.
- Failure surfaces via kube_job_status_failed (DEV-574 alert stack) and via
ConfigMap velero-restore-drill-last-result written on every run.
- Optional Paperclip webhook Secret is sealed but unreachable from Hetzner
right now (home Paperclip is LAN-only); wired as optional secretKeyRef so
the drill runs regardless. Connectivity is a follow-up.
- Argo Application velero-restore-drill points at velero/restore-drill/.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds Prometheus scrape of Velero's :8085 metrics and four alerts
(VeleroBackupFailed, VeleroBackupPartial, VeleroBackupMissed,
VeleroStorageLocationUnavailable) labeled team=infra severity=warning.
VeleroStorageLocationUnavailable substitutes for the spec's
VeleroRepoUnhealthy since Velero v1.18.1 does not expose
velero_backup_repository_available (see comment in the rule).
Wired via a new ArgoCD Application (velero-monitoring) that syncs
only servicemonitor.yaml + prometheusrule.yaml from basicstack-repo/velero/.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- velero/schedules/: cluster-all @ 03:00 (30d TTL, manifests only),
per-namespace <ns>-manifests @ 03:15+n min (30 in-scope namespaces,
30d TTL, no fs backup), and per-namespace <ns>-data @ 03:45+n min
for the whitelist (pocket-id, vaultwarden, vikunja, passbolt;
14d TTL, defaultVolumesToFsBackup=true).
- All schedules exclude velero, kube-system, kube-node-lease,
kube-public so the Kopia password Secret never lands in S3.
- apps/app-velero-schedules.yaml: sibling Argo app syncing that
directory into namespace velero with prune+selfHeal.
- opencloud, harbor(registry+trivy): pod-template annotation
backup.velero.io/backup-volumes-excludes as a safety net so blob-
heavy PVCs are never fs-backed-up even under a manual defaultVolumes
run. nextcloud/monitoring workloads are not Argo-tracked yet — they
were patched live in the same commit but need a follow-up to source-
control them.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The prior sealed secret's access key lived in the personal `imicros.de`
Hetzner Object Storage project and could not list objects in
`basicstack-velero` (bucket is in the BasicStack project) — BSL was
`Unavailable` with an S3 `AccessDenied` on ListObjectsV2.
CEO created a new S3 credential in the BasicStack project; re-sealed
against the cluster controller and applied. BSL `default` now
`Available`.
Refs DEV-572.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- velero-plugin-for-aws v1.14.3 is not published on docker.io (only rc.1/rc.2). Downgrade to the latest GA tag v1.14.2 — still compatible with Velero 1.18.x.
- Drop node-agent memory request 256Mi -> 64Mi so the daemonset schedules on worker-1/2/3 (they sit at ~95% memory-request already).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds SealedSecret velero-s3-credentials in namespace velero, holding
the dedicated velero-prod Hetzner Object Storage access/secret key
(scoped only to the basicstack-velero bucket).
Sealed on cp-1 via /root/seal-velero-s3-credentials.sh with the
sealed-secrets-controller public cert; plaintext never left the host.
Only the in-cluster controller can decrypt.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the SealedSecret for the Kopia repository password that the Velero
BackupRepository will consume once the Argo app lands. Raw password is
held on cp-1 (root:root, 0600) pending the CEO handoff to Vaultwarden
and the offline safe copy (DEV-571).
Refs: DEV-569, DEV-571
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the Open WebUI stack at openwebui.basicstack.de:
- Namespace `openwebui`, ArgoCD Application with auto-sync + selfHeal
- Own Postgres 16 + pgvector (`pgvector/pgvector:pg16`, 20Gi encrypted PVC,
fsn1-pinned); VECTOR_DB=pgvector, extension auto-created
- Own Redis 7 (2Gi encrypted PVC, fsn1-pinned) used both as cache and as
websocket manager (db=1)
- Hetzner Object Storage bucket `openwebui-basicstack` (hel1, path-style)
for file uploads; shared project S3 credentials
- Traefik ingress + Let's Encrypt TLS
- Pocket ID client `Open WebUI` (PKCE on) restricted to groups
`openwebui_admins` and `openwebui_users`; `andreas.leinen@basicstack.de`
is member of both and gets admin via OAUTH_ADMIN_ROLES
- Password login disabled (`ENABLE_LOGIN_FORM=false`, `ENABLE_SIGNUP=false`)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Stalwart's port-25 relay rejects bare EHLO labels — needs at least one
dot. Vikunja doesn't call go-mail's WithHELO, so it sends
os.Hostname() as EHLO. Setting hostname/subdomain plus
setHostnameAsFQDN=true makes the pod's kernel hostname a valid FQDN
(vikunja.basicstack.vikunja.svc.cluster.local), which Stalwart
accepts.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Vikunja now sends reminder/notification emails through
`stalwart-smtp.stalwart.svc.cluster.local:25` using unauthenticated
relay from the pod-network CIDR that Stalwart already trusts. The
`vikunja@basicstack.de` principal exists in Stalwart but its password
is unknown and admin API access to reset it is currently broken
(admin/andreas/paperclip password auth returns `{"type":"failure"}` per
`stalwart-oidc-only-admin` note). Verified end-to-end: message from
`vikunja@basicstack.de` sent via port-25 relay arrived in the passbolt
mailbox.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the kaneo-smtp SealedSecret and points the Kaneo container at
mail.basicstack.de:587 (STARTTLS) for outbound workspace invites and
notifications. DISABLE_EMAIL_OTP_SIGN_IN stays true so sign-in remains
OIDC-only via Pocket ID.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Outline's OIDC plugin does not send `code_challenge`, so the Pocket ID
client must be created with `pkceEnabled: false` (default is `true`).
Login fails with `?notice=auth-error` and Pocket ID rejects
`/authorize` when PKCE is enforced.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Vikunja 2.x's frontend redirect helper and Go backend do not send
code_challenge. Pocket ID rejects the /authorize call with
'code_challenge parameter is missing' when the client has pkceEnabled=true.
The Vikunja client is confidential (isPublic=false) so client_secret still
authenticates the token exchange — turning PKCE off is safe and required.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add `vikunja_scope` to the OIDC scope list the Vikunja app requests
from Pocket ID, so the `vikunja_groups` custom claim is available for
Vikunja's automatic team provisioning.
The claim itself is set per Pocket ID user group (`vikunja_admins`,
`vikunja_members`) via `PUT /api/custom-claims/user-group/:id`.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Vikunja 2.6.0 deprecates `service.jwtsecret` in favor of `service.secret`;
switch the env var name (secret payload unchanged) and fix the admin-promotion
example in the README to the current CLI (`user set-admin --admin`).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Namespace-scoped Postgres 16 + unified Vikunja container behind Traefik
ingress at vikunja.basicstack.de. Both pods pinned to Hetzner fsn1 due to
their encrypted PVCs (10 Gi DB, 10 Gi attachments). Local auth disabled;
sign-in only via Pocket ID (client `Vikunja`, groups `vikunja_admins` /
`vikunja_members`). SMTP mailer left disabled pending mailbox provisioning.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Hand-rolled manifests under apps/kaneo/ + Argo Application:
- Dedicated PostgreSQL 16-alpine on hcloud-volumes-encrypted (20Gi),
pinned to fsn1 via csi.hetzner.cloud/location node affinity.
- Kaneo v2.22.0 (single ghcr.io/usekaneo/kaneo image; API + web on
5173). Pinned to fsn1 for consistency with the volume.
- Traefik ingress at kaneo.basicstack.de with letsencrypt-prod cert.
- Better Auth JWT secret and Postgres credentials as SealedSecrets.
- OIDC via Pocket ID using CUSTOM_OAUTH_*, restricted to the two
new Pocket ID groups kaneo-admins / kaneo-members (client id
d58ff9bb-dc98-49b4-9535-4af2aba66de2).
- Registration + password + email-OTP sign-in disabled; SSO only.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- New namespace grist with in-namespace PostgreSQL 16
- Ingress grist.basicstack.de + letsencrypt-prod (DNS-01 via Hetzner)
- Pocket ID OIDC login (client "Grist", group "grist")
- All PVCs on hcloud-volumes-encrypted
- ArgoCD application via apps/app-grist.yaml
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The scheduled backup pod could not start because worker-3 (where the
opencloud data PVC is attached and where podAffinity pins the backup)
had ~40Mi of memory left by requests after opencloud/tika were
co-scheduled. rclone actually uses well under 100Mi during the sync.
Drop cpu request 100m -> 50m and memory request 256Mi -> 64Mi. Limits
unchanged (500m / 512Mi) so it can still burst.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Mirrored docker.io/restic/restic:0.19.1 to
harbor.basicstack.de/library/restic:0.19.1 (crane in-cluster Job).
Updated all CronJob pins and the restore-drill/tag-bump docs.
Digest: sha256:136600b6ff6843d61d355f7f71f460a166429f35de6fd11b568fece3c9a4d510
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The 7.2.0 -> 7.4.0 rollout stalled because the default RollingUpdate
tried to start the new pod before terminating the old one, but they
both mount opencloud-data (RWO), causing Multi-Attach errors.
Switch to strategy: Recreate so the old pod is terminated first and
the volume detaches cleanly before the new pod starts. Same fix
pattern as bookstack (DEV-528) and other RWO-backed apps.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Skimmed 7.3.0 (2026-07-14) and 7.4.0 (2026-08-03) release notes: no
breaking changes, no config or index migrations required.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Minor version bump of the cluster OIDC provider. No breaking changes;
multi-client-secret support in 2.14 is additive; SameSite=Lax cookie
default in 2.13 is standard. DB snapshot captured pre-rollout.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Patch bump per DEV-520 plan. Only pangolin uses postgres:17.5*;
other apps (paperclip, directus, forgejo, stalwart) are on
postgres:16-alpine and are out of scope for this ticket.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The Job self-deletes after ttlSecondsAfterFinished=3600, causing ArgoCD
to report the app permanently OutOfSync. The job body only hits
/healthz/live and prints a manual TODO; it never modified Stalwart
config. The real IP allowlist is already declared in
apps/stalwart/stalwart-config.yaml under
[server.listener.*.security] allowed-ips.
Also removes the orphan ServiceAccount/Role/RoleBinding
stalwart-config-access bundled in the same file.
Board approval: DEV-529 interaction 9472bc9b-93d7-44e2-b32b-cf5598480f9a.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
BookStack is a single-replica Deployment backed by a ReadWriteOnce
PVC (`bookstack-config`). The default RollingUpdate strategy creates
the new pod before terminating the old one, and the two pods cannot
share the RWO volume — so every rollout deadlocks on
`Multi-Attach error for volume`.
Switching to `strategy: Recreate` terminates the old pod first,
releases the PVC, and then starts the new pod. Brief downtime is
expected on rollout, but rollouts actually complete.
Discovered while rolling out the `/status` probe fix (previous commit).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
