Container was repeatedly OOMKilled (exit 137) every ~5 s of live time since
2026-10-11 (pod age 6d23h, 1919 restarts). Live steady-state memory is now
~321 MiB with 1Gi/512Mi settings, so comfortable headroom. Patched the live
StatefulSet first to recover the control plane; this commit lets GitOps
retain the fix on the next reconcile.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The 180s post-uncordon health check in `update-node.sh` step 7 was single-shot.
Any transient yellow signal (Init:*, Running-but-not-Ready, deployment below
replicas during reschedule) at the exact moment of the check would FATAL the
whole cycle and require operator wake + manual `--start-from <next-node>`.
During DEV-593 this fired three times in one cycle (opencloud init, harbor
jobservice reinit) for conditions that cleared within seconds.
Wrap the health check in a bounded retry loop:
- First check at `POST_UNCORDON_WAIT_SECONDS` (default 180s, unchanged).
- Up to `HEALTH_MAX_RETRIES` (default 10) more attempts at
`HEALTH_RETRY_INTERVAL_SECONDS` (default 30s) = +5 min max.
- Short-circuit halt immediately on persistent pod errors
(CrashLoopBackOff, ImagePullBackOff, ErrImagePull,
CreateContainerConfigError, InvalidImageName) — these don't recover on
their own and shouldn't burn the retry budget.
- Each retry calls `cluster-health.sh` with `RETRY_ON_TRANSIENT=0` so the
outer loop controls cadence (no double 30s sleeps per attempt).
No k3s or manifest changes. Preflight path is untouched.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
DEV-586 (2026-09-27) aborted mid-drain because cp-1's pubkey had drifted out
of four nodes' /root/.ssh/authorized_keys — probably Hetzner cloud-init
re-syncing from the HC SSH-keys API on reboot (the operator key is not in
that keyset). Both the apt-phase SSH and the CP peer-probe SSH died once
they tried to reach the drifted targets, and recovery cost real time.
os-update.sh now runs a preflight against the fully-computed plan:
- Reads cp-1's pubkey once (fails loudly if operator can't reach cp-1).
- For every planned worker + CP: proves operator SSH works, then grep-checks
cp-1's pubkey in /root/.ssh/authorized_keys. Missing pubkeys are appended
via the operator's own SSH (auto-heal). Verify via grep after the write.
- Nodes still unreachable at the end block the whole cycle with the exact
fix command in the log — nothing gets drained until this passes.
- Preflight report: /tmp/os-update-<UTC-ts>/ssh-preflight.txt (ok/repaired/
problem lists) as audit trail.
- Escape hatch: --skip-ssh-preflight for bootstrap scenarios.
Also factors the node->ssh-endpoint case into a shared node_ssh_endpoint()
helper so the preflight, finalize/apt-history loop, and any future call site
can't drift apart.
Verified end-to-end against the live cluster: all planned nodes reachable
and cp-1's pubkey currently present on cp-2/cp-3, worker-1..6.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Worker-2 CPU requests were pinned at 100% (2000m/2000m), leaving no room
for the daily opencloud-backup CronJob (needs 20m). Actual opencloud CPU
usage is ~3m — reserving 250m was 80x over actual. Reduce to 50m which is
still 15x headroom. Frees 200m on worker-2.
Backup CronJob is pinned to the opencloud node by RWO PVC + hard
podAffinity. Reducing opencloud's own CPU request is the cleanest
unblock — contained to the opencloud namespace, no cross-app blast
radius.
Related: DEV-593, DEV-464, DEV-481, DEV-514, DEV-582/584 (recurring
pattern of backup CronJob squeezed off worker-2).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The 2026-10-01 monthly drill passed but the post-run webhook got
HTTP 401 from the home Paperclip. Root cause: the home server's
global actorMiddleware (/app/server/src/middleware/auth.ts) rejects
every Authorization: Bearer <...> that is not a board API key or an
agent JWT before per-route handlers run, so the trigger's own
`bearer` signingMode handler is unreachable with a 48-hex random
secret. Only publicMcpGatewayProtocolPath is exempted.
Fix:
- Trigger 6983798e-018c-459d-95ea-4b5959441e2d switched to
signingMode: github_hmac via PATCH (secret unchanged).
- post_webhook now computes HMAC-SHA256(body, secret) with openssl
(already in alpine/k8s) and sends X-Hub-Signature-256 instead of
Authorization: Bearer. HMAC headers bypass the Bearer branch of
the global middleware entirely.
Verified end-to-end: synthetic HMAC-signed body posted to
.../fire returned 202 issue_created and spawned DEV-591 under
parent DEV-578. SealedSecret is unchanged (same secret value
reused as the HMAC key).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds apps/monitoring/prometheus-deployment.yaml capturing the live PVC/Service/
Deployment for monitoring/prometheus. Two behaviour changes vs the previous
live state:
- strategy: RollingUpdate -> Recreate. prometheus-data-encrypted is RWO on
hcloud-volumes-encrypted; a rolling update deadlocks because the surge pod
cannot attach the PVC while the old pod holds it. This has been jamming the
weekly OS-update preflight (see DEV-582). Same fix pattern as loki
(loki-deployment.yaml) and opencloud (DEV-538).
- pod-template annotation backup.velero.io/backup-volumes-excludes:
prometheus-storage. Restores the annotation that rev 6 tried to add before
the RollingUpdate deadlock forced a rollback in DEV-582.
Applied and verified: rollout succeeded, /-/ready and /-/healthy both green,
pod running on k3s-worker-6 after PVC re-attach.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Velero cluster-all now covers cluster-manifest backup with two successive
Completed runs verified and one restore drill (openwebui) green (DEV-575).
- Delete apps/monitoring/backup-k8s-resources-cronjob.yaml
- Drop BackupK8sResourcesStale + BackupK8sResourcesCheckFailed from
backup-restic-alerts.yaml and matching series/cases from the promtool
test file (all 9 remaining rules pass `promtool test rules`)
- Update apps/monitoring/README.md + prometheus-backup-cronjob comments
- Runbook lives in BookStack: EN
https://books.basicstack.de/books/it-operations-manual/page/velero-backup-restore-runbook
DE
https://books.basicstack.de/books/it-betriebshandbuch/page/velero-backup-und-restore-runbook
Rollback: revert this commit — Argo restores the CronJob + alerts and
restic k8s-resources runs resume at the next 02:00 UTC schedule.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Public DNS for paperclip.home.imicros.de returns the LAN IP 10.0.0.2, so the
drill's webhook POST was landing with HTTP 000 from Hetzner. The home reverse
proxy at 46.225.155.185 already serves that vhost publicly, so pinning the
hostname to the home public IP in the CronJob's hostAliases lets the webhook
fire the Paperclip routine without any home-network or SealedSecret changes.
Verified end-to-end: bearer POST via --resolve returns HTTP 202 with
status=issue_created (run issue b07fc035 created under DEV-578).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- CronJob at 0 4 1 * * restores the latest openwebui-manifests backup into
a transient restore-test-<ts> namespace, diffs core resource-kind counts
against the live openwebui namespace, and deletes the target namespace.
- ServiceAccount + ClusterRole scoped to velero.io + namespace + configmap
ops. Script enforces the 'restore-test-' name prefix before deleting a
namespace. Cluster-wide 'view' bound for the diff.
- Failure surfaces via kube_job_status_failed (DEV-574 alert stack) and via
ConfigMap velero-restore-drill-last-result written on every run.
- Optional Paperclip webhook Secret is sealed but unreachable from Hetzner
right now (home Paperclip is LAN-only); wired as optional secretKeyRef so
the drill runs regardless. Connectivity is a follow-up.
- Argo Application velero-restore-drill points at velero/restore-drill/.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds Prometheus scrape of Velero's :8085 metrics and four alerts
(VeleroBackupFailed, VeleroBackupPartial, VeleroBackupMissed,
VeleroStorageLocationUnavailable) labeled team=infra severity=warning.
VeleroStorageLocationUnavailable substitutes for the spec's
VeleroRepoUnhealthy since Velero v1.18.1 does not expose
velero_backup_repository_available (see comment in the rule).
Wired via a new ArgoCD Application (velero-monitoring) that syncs
only servicemonitor.yaml + prometheusrule.yaml from basicstack-repo/velero/.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- velero/schedules/: cluster-all @ 03:00 (30d TTL, manifests only),
per-namespace <ns>-manifests @ 03:15+n min (30 in-scope namespaces,
30d TTL, no fs backup), and per-namespace <ns>-data @ 03:45+n min
for the whitelist (pocket-id, vaultwarden, vikunja, passbolt;
14d TTL, defaultVolumesToFsBackup=true).
- All schedules exclude velero, kube-system, kube-node-lease,
kube-public so the Kopia password Secret never lands in S3.
- apps/app-velero-schedules.yaml: sibling Argo app syncing that
directory into namespace velero with prune+selfHeal.
- opencloud, harbor(registry+trivy): pod-template annotation
backup.velero.io/backup-volumes-excludes as a safety net so blob-
heavy PVCs are never fs-backed-up even under a manual defaultVolumes
run. nextcloud/monitoring workloads are not Argo-tracked yet — they
were patched live in the same commit but need a follow-up to source-
control them.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The prior sealed secret's access key lived in the personal `imicros.de`
Hetzner Object Storage project and could not list objects in
`basicstack-velero` (bucket is in the BasicStack project) — BSL was
`Unavailable` with an S3 `AccessDenied` on ListObjectsV2.
CEO created a new S3 credential in the BasicStack project; re-sealed
against the cluster controller and applied. BSL `default` now
`Available`.
Refs DEV-572.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- velero-plugin-for-aws v1.14.3 is not published on docker.io (only rc.1/rc.2). Downgrade to the latest GA tag v1.14.2 — still compatible with Velero 1.18.x.
- Drop node-agent memory request 256Mi -> 64Mi so the daemonset schedules on worker-1/2/3 (they sit at ~95% memory-request already).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds SealedSecret velero-s3-credentials in namespace velero, holding
the dedicated velero-prod Hetzner Object Storage access/secret key
(scoped only to the basicstack-velero bucket).
Sealed on cp-1 via /root/seal-velero-s3-credentials.sh with the
sealed-secrets-controller public cert; plaintext never left the host.
Only the in-cluster controller can decrypt.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the SealedSecret for the Kopia repository password that the Velero
BackupRepository will consume once the Argo app lands. Raw password is
held on cp-1 (root:root, 0600) pending the CEO handoff to Vaultwarden
and the offline safe copy (DEV-571).
Refs: DEV-569, DEV-571
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the Open WebUI stack at openwebui.basicstack.de:
- Namespace `openwebui`, ArgoCD Application with auto-sync + selfHeal
- Own Postgres 16 + pgvector (`pgvector/pgvector:pg16`, 20Gi encrypted PVC,
fsn1-pinned); VECTOR_DB=pgvector, extension auto-created
- Own Redis 7 (2Gi encrypted PVC, fsn1-pinned) used both as cache and as
websocket manager (db=1)
- Hetzner Object Storage bucket `openwebui-basicstack` (hel1, path-style)
for file uploads; shared project S3 credentials
- Traefik ingress + Let's Encrypt TLS
- Pocket ID client `Open WebUI` (PKCE on) restricted to groups
`openwebui_admins` and `openwebui_users`; `andreas.leinen@basicstack.de`
is member of both and gets admin via OAUTH_ADMIN_ROLES
- Password login disabled (`ENABLE_LOGIN_FORM=false`, `ENABLE_SIGNUP=false`)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
k3s api-server /livez on this cluster is auth-gated and returns HTTP 401
without a bearer token. The Phase E reboot poll hardcoded HTTP 200, so
every CP reboot would time out at 10 min ("api-server did not return
within 600s") even though cp-2 was actually Ready and etcd-rejoined
within ~90 s.
Any HTTP response from the api-server (200/401/403) means it is up and
serving; only network-level failures (000, connection refused) count as
"not yet back". Added --max-time 5 to keep polling snappy immediately
after the reboot when TCP may accept-then-drop.
Verified during DEV-566 cycle:
curl -sk -o/dev/null -w '%{http_code}' https://<cp>:6443/livez -> 401
on cp-1, cp-2, cp-3 (all Ready).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
k3s-worker-6 was joined on 2026-08-30 but never added to the
node_ssh_target case in update-node.sh, so the weekly cycle would die
with 'unknown node k3s-worker-6' as soon as it reached that node. Map
it to its public IP (49.12.223.133) — same pattern as worker-3/4/5.
Also add the corresponding case in os-update.sh's apt-history digest.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Stalwart's port-25 relay rejects bare EHLO labels — needs at least one
dot. Vikunja doesn't call go-mail's WithHELO, so it sends
os.Hostname() as EHLO. Setting hostname/subdomain plus
setHostnameAsFQDN=true makes the pod's kernel hostname a valid FQDN
(vikunja.basicstack.vikunja.svc.cluster.local), which Stalwart
accepts.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Vikunja now sends reminder/notification emails through
`stalwart-smtp.stalwart.svc.cluster.local:25` using unauthenticated
relay from the pod-network CIDR that Stalwart already trusts. The
`vikunja@basicstack.de` principal exists in Stalwart but its password
is unknown and admin API access to reset it is currently broken
(admin/andreas/paperclip password auth returns `{"type":"failure"}` per
`stalwart-oidc-only-admin` note). Verified end-to-end: message from
`vikunja@basicstack.de` sent via port-25 relay arrived in the passbolt
mailbox.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the kaneo-smtp SealedSecret and points the Kaneo container at
mail.basicstack.de:587 (STARTTLS) for outbound workspace invites and
notifications. DISABLE_EMAIL_OTP_SIGN_IN stays true so sign-in remains
OIDC-only via Pocket ID.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Outline's OIDC plugin does not send `code_challenge`, so the Pocket ID
client must be created with `pkceEnabled: false` (default is `true`).
Login fails with `?notice=auth-error` and Pocket ID rejects
`/authorize` when PKCE is enforced.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Vikunja 2.x's frontend redirect helper and Go backend do not send
code_challenge. Pocket ID rejects the /authorize call with
'code_challenge parameter is missing' when the client has pkceEnabled=true.
The Vikunja client is confidential (isPublic=false) so client_secret still
authenticates the token exchange — turning PKCE off is safe and required.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add `vikunja_scope` to the OIDC scope list the Vikunja app requests
from Pocket ID, so the `vikunja_groups` custom claim is available for
Vikunja's automatic team provisioning.
The claim itself is set per Pocket ID user group (`vikunja_admins`,
`vikunja_members`) via `PUT /api/custom-claims/user-group/:id`.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Vikunja 2.6.0 deprecates `service.jwtsecret` in favor of `service.secret`;
switch the env var name (secret payload unchanged) and fix the admin-promotion
example in the README to the current CLI (`user set-admin --admin`).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Namespace-scoped Postgres 16 + unified Vikunja container behind Traefik
ingress at vikunja.basicstack.de. Both pods pinned to Hetzner fsn1 due to
their encrypted PVCs (10 Gi DB, 10 Gi attachments). Local auth disabled;
sign-in only via Pocket ID (client `Vikunja`, groups `vikunja_admins` /
`vikunja_members`). SMTP mailer left disabled pending mailbox provisioning.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Hand-rolled manifests under apps/kaneo/ + Argo Application:
- Dedicated PostgreSQL 16-alpine on hcloud-volumes-encrypted (20Gi),
pinned to fsn1 via csi.hetzner.cloud/location node affinity.
- Kaneo v2.22.0 (single ghcr.io/usekaneo/kaneo image; API + web on
5173). Pinned to fsn1 for consistency with the volume.
- Traefik ingress at kaneo.basicstack.de with letsencrypt-prod cert.
- Better Auth JWT secret and Postgres credentials as SealedSecrets.
- OIDC via Pocket ID using CUSTOM_OAUTH_*, restricted to the two
new Pocket ID groups kaneo-admins / kaneo-members (client id
d58ff9bb-dc98-49b4-9535-4af2aba66de2).
- Registration + password + email-OTP sign-in disabled; SSO only.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- New namespace grist with in-namespace PostgreSQL 16
- Ingress grist.basicstack.de + letsencrypt-prod (DNS-01 via Hetzner)
- Pocket ID OIDC login (client "Grist", group "grist")
- All PVCs on hcloud-volumes-encrypted
- ArgoCD application via apps/app-grist.yaml
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The scheduled backup pod could not start because worker-3 (where the
opencloud data PVC is attached and where podAffinity pins the backup)
had ~40Mi of memory left by requests after opencloud/tika were
co-scheduled. rclone actually uses well under 100Mi during the sync.
Drop cpu request 100m -> 50m and memory request 256Mi -> 64Mi. Limits
unchanged (500m / 512Mi) so it can still burst.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- provision-k3s-worker.sh now sets --node-ip=<private-ip> on the agent
install so k3s does not default to the public IP (DEV-224 root cause)
- K3S_OPERATIONS.md node table reflects the current cluster (cp-2/cp-3,
worker-4/5/6), documents the Hetzner Cloud Volume datacenter constraint,
and calls out the --node-ip requirement in the manual join steps
Co-Authored-By: Paperclip <noreply@paperclip.ing>
os-update.sh now walks all three CPs after workers, one at a time, etcd
followers first and leader last. Each CP is delegated to update-cp-node.sh
--run with ASSUME_YES=1; a health gate (cluster-health.sh +
etcdctl endpoint status) runs between CPs. Adds --skip-cps and --only-cps
flags; --only <node> and --start-from <node> now route to the correct list
automatically.
OS_UPDATE_PROCEDURE.md updated: CPs are no longer excluded, §8 renamed to
"Control plane handling" and describes the leader-last cycle + delegation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Mirrored docker.io/restic/restic:0.19.1 to
harbor.basicstack.de/library/restic:0.19.1 (crane in-cluster Job).
Updated all CronJob pins and the restore-drill/tag-bump docs.
Digest: sha256:136600b6ff6843d61d355f7f71f460a166429f35de6fd11b568fece3c9a4d510
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The 7.2.0 -> 7.4.0 rollout stalled because the default RollingUpdate
tried to start the new pod before terminating the old one, but they
both mount opencloud-data (RWO), causing Multi-Attach errors.
Switch to strategy: Recreate so the old pod is terminated first and
the volume detaches cleanly before the new pod starts. Same fix
pattern as bookstack (DEV-528) and other RWO-backed apps.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Skimmed 7.3.0 (2026-07-14) and 7.4.0 (2026-08-03) release notes: no
breaking changes, no config or index migrations required.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Minor version bump of the cluster OIDC provider. No breaking changes;
multi-client-secret support in 2.14 is additive; SameSite=Lax cookie
default in 2.13 is standard. DB snapshot captured pre-rollout.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Patch bump per DEV-520 plan. Only pangolin uses postgres:17.5*;
other apps (paperclip, directus, forgejo, stalwart) are on
postgres:16-alpine and are out of scope for this ticket.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
