Phase 2 of the Stalwart RocksDB -> PostgreSQL migration (DEV-467).
Purely additive - does not touch the running stalwart-0 pod, its
ConfigMap, PVC, or bootstrap-config.
- stalwart-postgres.yaml: single-replica StatefulSet with a 10Gi
hcloud-volumes-encrypted PVC (subPath pgdata), ClusterIP Service
on 5432, nodeAffinity csi.hetzner.cloud/location=fsn1 so the DB
co-locates with stalwart-0 (which is fsn1-pinned by its PVC).
Resources match forgejo-postgres (250m/512Mi req, 500m/1Gi lim).
- stalwart-postgres-credentials-sealed.yaml: sealed secret with
POSTGRES_USER=stalwart, POSTGRES_DB=stalwart, POSTGRES_PASSWORD,
plus a copy of the password under stalwart-db-password for
Stalwart's [store.postgres] config in the Phase 4 cutover.
- stalwart-postgres-backup.yaml: daily CronJob at 02:30 UTC that
streams pg_dump | gzip into the existing stalwart-s3-backup restic
repo tagged stalwart-postgres, with independent retention keys
(14d/8w/6m) so it doesn't collide with the RocksDB snapshots.
activeDeadlineSeconds=1800 mirrors the DEV-464 sibling fix.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The original `rclone delete --min-age 7d` step ate the freshly uploaded
backup: rclone preserves each source file's mtime, and Prometheus TSDB
chunk/block files retain very old mtimes (weeks-old immutable blocks),
so filtering by file age deleted ~72% of the objects immediately after
sync (verified: 8.764 GiB destination reduced to 2.310 GiB / 37 objects
before we noticed).
Replaced the mtime prune with a prefix-name-based prune: every top-level
prefix is `prometheus-YYYYMMDD-HHMMSS`, so we parse the encoded date and
`rclone purge` whole prefixes older than 7 days. This keeps the latest
7 daily snapshots intact regardless of the Prometheus block ages.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
After the DEV-464 split, `monitoring/backup-volumes` only backs up
grafana + loki (pinned to k3s-worker-2 with `backup-storage`), leaving
Prometheus data unbacked. `prometheus-data-encrypted` is an RWO Hetzner
Cloud volume attached to whichever node currently runs the Prometheus
pod (typically k3s-worker-1), so it cannot join the shared backup-volumes
job without provoking Multi-Attach errors.
This introduces a dedicated `monitoring/prometheus-backup` CronJob that:
- Streams `prometheus-data-encrypted` to Hetzner S3 via rclone
(`basicstack-backup/prometheus/prometheus-<DATE>/`).
- Uses `podAffinity` to co-schedule with the Prometheus pod so the RWO
PVC always attaches on the same node.
- Runs at 03:30 daily, `Forbid` concurrency, 60m hard deadline.
- Retains 7 days of dated backups (rclone delete --min-age 7d).
- Tolerates the expected TSDB compaction race (Prometheus deletes old
block dirs mid-copy): rclone's non-zero exit from those transient
errors is captured, then success is validated by comparing dest
bytes to source bytes (>= 80% and > 100 MiB floor).
S3 credentials are the same Hetzner Object Storage account used by
`opencloud-backup` and `stalwart-backup`, resealed for the `monitoring`
namespace as `SealedSecret monitoring-s3-backup`.
Verified with a manual job on k3s-worker-1 (see DEV-465 for logs).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Create the missing forgejo/platform-backup-data PVC that forgejo-backup
references (20Gi hcloud-volumes-encrypted).
- Record monitoring/backup-k8s-resources with a k3s-worker-2 nodeSelector
(backup-storage PVC is local-path pinned there), lower memory request
(128Mi) so it fits worker-2 pressure, and switch to alpine/k8s image
(bitnami/kubectl is no longer resolvable).
- Rewrite monitoring/backup-volumes to only back up grafana + loki
co-located with backup-storage on k3s-worker-2. Prometheus data
lives on k3s-worker-1 and is intentionally excluded here; a
dedicated Prometheus data backup follows in a separate ticket.
The three CronJobs previously left Pending/ContainerCreating pods that
blocked the OS-update health guard in DEV-463.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Investigation confirmed all SMTP/IMAP ports working correctly as of 21:00 UTC.
SMTP issue at 20:31 UTC coincided with LoadBalancer IP updates (10.42.1.1 → 178.105.17.239).
External port tests verify correct protocol greetings on all ports.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Documents the Pangolin + pangolin-kube-controller deployment: architecture,
per-file component map, the two SSO flows (dashboard login vs downstream
resource protection), the ingress-protection request flow, a step-by-step
runbook for adding a new protected ingress, troubleshooting rooted in the
DEV-457 findings, and hard "do not delete" notes on the two PVCs, the
sealed secrets, and the Traefik HelmChartConfig that Pangolin depends on.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Matches what was manually set on the drifted live Application so that
Argo CD reconciles from Forgejo automatically. Non-destructive
(prune: false).
Refs DEV-458.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Harbor's jobservice and registry Deployments have RWO PVCs (Hetzner
Cloud Volumes). The default RollingUpdate strategy deadlocks: the new
pod cannot attach the volume while the old pod still holds it on a
different node. This has left the harbor Argo CD Application in a
Degraded state since 2026-08-07.
The goharbor chart exposes a top-level updateStrategy specifically for
this case ("Set it as Recreate when RWM for volumes isn't supported").
Refs DEV-458.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Pangolin's IngressRoute 1-paperclip-router (pangolin ns) now owns the
paperclip.basicstack.de route (SSO gate via badger middleware). The legacy
Ingress at paperclip/paperclip bypassed Pangolin entirely and produced
"secret paperclip/paperclip-tls does not exist" noise in Traefik.
Cluster deletion follows in this heartbeat; Argo sync policy has no
automated prune, so removal from source is safe first.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Traefik runs in kube-system, so the short DNS name "pangolin" that Pangolin's
default `internal_hostname` uses does not resolve. Set both
`server.internal_hostname` and `server.badger_override` to
`pangolin.pangolin.svc.cluster.local` so the badger middleware and internal
callbacks resolve from any namespace.
Fixes the HTTP 500 on `https://paperclip.basicstack.de/` after cross-namespace
IngressRoute wiring was fixed in DEV-457.
Refs: DEV-457
Co-Authored-By: Paperclip <noreply@paperclip.ing>
DEV-452. After the SQLite -> Postgres migration in DEV-451 the pangolin
initial setup was redone, so the controller's stale bearer token was
invalid. That was only half the problem: /api/v1/traefik-config is
served exclusively by Pangolin's internal API (port 3001), not the
external dashboard API (port 3000). Pointing a Bearer request at port
3000 always returned 401 because that path lives on the session-auth
router. This has been the underlying cause of the controller's
CrashLoopBackOff, not just the stale key.
Changes:
- pangolin-controller-api-key-sealed.yaml: reseal new bearer token
(kubeseal against sealed-secrets-controller in kube-system, includes
the Bearer prefix expected by the controller).
- pangolin-controller-config.yaml: CONFIG_ENDPOINT now targets
http://pangolin.pangolin.svc.cluster.local:3001/api/v1/traefik-config.
- pangolin-deployment.yaml: Service now exposes port 3001 as the
"internal" port so in-cluster clients (kube-controller) can reach it.
Ingress still only routes / and /api to ports 3002/3000; port 3001
is not published to the internet.
- pangolin-controller-deployment.yaml: replicas back to 1, dropped the
temporary "scaled to 0" comment block.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The DB switch (DEV-451, SQLite -> PostgreSQL) starts Pangolin from a
fresh schema with no admin user or API key. The controller's bearer
token in pangolin-controller-api-key SealedSecret was issued against
the old SQLite state and is now invalid (the controller was already
in CrashLoopBackOff for the same reason).
Scale to 0 replicas so the ArgoCD Application reports Healthy after
the DB migration. Follow-up work will run Pangolin initial setup,
issue a new controller API key, and re-seal the secret before scaling
back to 1.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Pangolin's PostgreSQL build requires postgres.connection_string in
config.yml — DATABASE_URL alone is not honored as an override, so the
container was crashing with "Postgres configuration is missing in the
configuration file".
Render the final config.yml at pod startup via a busybox init container
that substitutes the DATABASE_URL secret into a __DATABASE_URL__
placeholder in the ConfigMap template, then mount the rendered file
into the pangolin container.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Deploy a dedicated PostgreSQL 17 instance in the pangolin namespace
and switch Pangolin (fosrl/pangolin) to the postgresql-1.21.1 image
variant. The database URL is provided via a sealed secret and pulled
into the Pangolin container as DATABASE_URL, which the PostgreSQL
Pangolin build reads at startup.
- New: postgres.yaml (Deployment + PVC on hcloud-volumes-encrypted + Service)
- New: pangolin-postgres-secrets-sealed.yaml (postgres creds + connection string)
- pangolin-deployment.yaml: image -> fosrl/pangolin:postgresql-1.21.1,
DATABASE_URL from secret, /app/config PVC mount, init container waits
for postgres, removed obsolete DATABASE_PATH env
- pangolin-config.yaml: dropped SQLite database.path stanza
Fixes DEV-451.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The previous sealed secret was missing the "Bearer " prefix in the
auth-header value. The controller uses CONFIG_AUTH_HEADER directly
as the Authorization header, so it needs the complete value.
Sealed value now contains: Bearer abw3ud1ipko7ock.2al7fqzhbmfsqp3bm4byvynyx6jnqe733icjdprd
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The previous sealed secret was created with an incorrect API key.
Re-sealed with the correct API key: abw3ud1ipko7ock.2al7fqzhbmfsqp3bm4byvynyx6jnqe733icjdprd
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Sealed the correct auth header value from the controller config into the
sealed secret. This ensures the controller can authenticate properly with
the Pangolin API.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Update sealed API key (abw3ud1ipko7ock...) in controller secret
- Remove insecure plaintext API key from ConfigMap
- Controller now uses only the sealed secret for authentication
No changes to main Pangolin deployment or database configuration.
Safe to deploy - no risk of database reinitialization.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Update CONFIG_ENDPOINT to use port 3000 (API) instead of port 3002 (Web UI)
to resolve empty Traefik configuration issue.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The controller uses leader election with replicas=1. During RollingUpdate,
the new pod cannot acquire the leader lease (old pod holds it) and fails
health checks, causing indefinite restart loops.
Recreate strategy ensures the old pod terminates before the new one starts,
allowing clean leader election transitions.
Fixes: DEV-442
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds pangolin-controller-api-key sealed secret with Bearer token
for authenticating to Pangolin API. This enables the controller
to fetch Traefik configuration from Pangolin service.
Resolves: DEV-441
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This guide provides step-by-step instructions for configuring Pangolin
Community Edition to protect the paperclip.basicstack.de domain with
authentication and access control. It uses application-level policies
instead of enterprise-only global policies.
Related to DEV-400.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added comprehensive documentation of the two-tier load balancing setup:
- Hetzner Cloud Load Balancer (external layer, managed by Hetzner CCM)
- Kubernetes LoadBalancer services (internal layer, k3s ServiceLB)
Key points documented:
- Traffic flow from external client through both LB layers to pod
- Why LoadBalancer service type is required (CCM integration)
- Historical context of the migration from hostPort to Hetzner LB
- Service definitions and port configurations
Updated:
- apps/stalwart/README.md: Added Network Architecture section
- infrastructure/networking/NETWORK_ARCHITECTURE.md: Enhanced Stalwart
section with two-tier architecture details and updated traffic flows
Resolves documentation gap identified in DEV-439.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Configure pangolin-kube-controller to authenticate with Pangolin API
using the provided API key.
Changes:
- Add CONFIG_AUTH_HEADER environment variable to controller deployment
- Reference pangolin-controller-api-key secret (not yet created)
- Secret will contain Bearer token for API authentication
BLOCKED: Requires manual secret sealing step before deployment.
To complete this deployment, run on a machine with cluster access:
kubectl create secret generic pangolin-controller-api-key \
--namespace=pangolin \
--from-literal=auth-header="Bearer 5qid06u9j325kpk.ywd3bpsx34dtxyczgatyxuoxkzwhie7d72k6v4hw" \
--dry-run=client -o yaml | \
kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \
--format=yaml > apps/pangolin/pangolin-controller-api-key-sealed.yaml
Then commit the sealed secret and push both files.
Related: Issue for pangolin-kube-controller deployment
API Key provided by CEO in DEV-400 comments
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add comprehensive Network Architecture section explaining:
- Two-tier load balancing (K8s LoadBalancer + Hetzner LB)
- Why LoadBalancer service type is required (Hetzner CCM auto-management)
- Traffic flow from external clients to Stalwart pods
- Warning against changing to NodePort (breaks CCM integration)
This documents the investigation from DEV-439, confirming that the current
LoadBalancer service setup is correct and necessary for our infrastructure.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Update liveness and readiness probe ports from 8080 to 9090 to match the
controller's METRICS_ADDR configuration. The controller is configured to
serve metrics on port 9090, but the probes were checking port 8080,
causing pods to remain in NotReady state despite the controller
functioning correctly.
Also update containerPort to 9090 for consistency.
Related: DEV-437
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Extend ClusterRole to include all Traefik CRDs that the controller manages:
- serverstransports, serverstransporttcps (for transport configuration)
- ingressroutetcps, ingressrouteudps (for TCP/UDP routing)
- middlewaretcps (for TCP middleware)
- tlsoptions, tlsstores (for TLS configuration)
This fixes the RBAC permission error preventing the controller from
reconciling Traefik configuration: "serverstransports.traefik.io is
forbidden: User \"system:serviceaccount:pangolin:pangolin-controller\"
cannot list resource \"serverstransports\""
Related: DEV-437
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Update controller configuration to fix 401 authentication errors:
- Add CONFIG_AUTH_HEADER with Bearer token for Pangolin API access
- Correct CONFIG_ENDPOINT port from 3000 to 3002 (matches Pangolin service)
- Remove duplicate controller manifests from controller/ subdirectory
The existing controller deployment was failing with 401 errors because
it lacked authentication credentials. This change adds the API key
provided in DEV-437 and corrects the internal service endpoint.
Related: DEV-437
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move Kubernetes controller manifests from kube-controller/ subdirectory
to the apps/pangolin/ root directory so Argo CD will sync them.
Files renamed with pangolin-controller- prefix:
- configmap.yaml -> pangolin-controller-config.yaml
- deployment.yaml -> pangolin-controller-deployment.yaml
- rbac.yaml -> pangolin-controller-rbac.yaml
- service.yaml -> pangolin-controller-service.yaml
Argo CD was not recursing into the subdirectory, so controller resources
were not being synced. Moving to root directory resolves this.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Use specific image version 0.1.0-alpha.1 instead of :latest
- Fix API endpoint port from 3001 to 3000 (correct Pangolin service port)
- Add CONFIG_ALLOW_INSECURE_HTTP for internal cluster HTTP communication
- Fix security context with numeric UIDs (65532) to avoid runAsNonRoot error
- Add required RBAC permissions for CRD and IngressClass discovery
Controller now successfully starts and acquires leader lease.
Next step: Configure API authentication (CONFIG_AUTH_HEADER) for
controller to access Pangolin API endpoint.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Create complete manifest set for Pangolin controller deployment:
- RBAC with ServiceAccount, ClusterRole, and ClusterRoleBinding
- ConfigMap with controller configuration (endpoint, namespace, leader election)
- Deployment with resource limits, health probes, and security context
- Service for metrics endpoint on port 8080
Controller will manage Traefik CRDs (IngressRoute, Middleware, TraefikService)
and sync configuration from Pangolin API.
Relates to DEV-397
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The Pangolin setup was failing with 404 on /api/v1/auth/set-server-admin
because the ingress was routing all traffic to port 3002 (Web UI), but
API endpoints are served by a separate API server on port 3000.
Changes:
- Added port 3000 (api) to deployment container ports
- Updated service to expose both port 3000 (api) and 3002 (http)
- Added /api path to ingress routing to port 3000
- Kept / path routing to port 3002 for Web UI
This allows the browser to make API calls to /api/* which will now
be routed to the API server, fixing the 404 error.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Fixes server action 404 error preventing initial setup.
Version 1.21.1 was released on 2026-07-30 and includes fixes
for Next.js server action routing issues.
Related: DEV-396
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Changed Traefik entrypoints from 'websecure' (HTTPS-only) to
'web,websecure' (HTTP + HTTPS) to allow Let's Encrypt HTTP-01
challenge verification on port 80.
The TLS section in the ingress will still redirect HTTP to HTTPS
for normal traffic after the certificate is issued.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The DNS-01 challenge has been stuck for 5+ days due to Hetzner DNS
webhook issues. Switched to the more reliable HTTP-01 challenge method
by using the new letsencrypt-http01 ClusterIssuer.
This resolves the missing certificate issue for pangolin.basicstack.de.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Test documentation for the ConfigMap subPath mount hardening deployed
in commit 291feb9.
Results:
- Pod deletion test: 3/3 passed with 0 restarts (69s, ~80s, 93s recovery)
- Alert test: Completed but found Alertmanager sends to null receiver
- Mail service health: Verified healthy after all tests
DEV-431.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Extends commit 291feb9 by adding stalwart-config (stalwart.toml) to the
copy-config initContainer. The previous fix only handled bootstrap-config
but missed stalwart-config, which was the root cause of CrashLoopBackOff
identified in DEV-426 stability testing.
Root cause: commit 693fcd3 introduced stalwart-config ConfigMap with
subPath mount. During pod restarts, Kubernetes subPath ConfigMap mounting
race leaves stalwart.toml empty/unparseable, causing:
"Failed to parse data store settings at /etc/stalwart/stalwart.toml:
expected value at line 1 column 1"
This completes the hardening by ensuring BOTH config files (config.json
and stalwart.toml) are atomically copied before Stalwart starts.
Fixes: DEV-433, DEV-431
Ref: stack.basicstack.de/apps/stalwart/STABILITY-VERIFICATION-2026-08-01.md
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Issue: During stability testing (DEV-426), pod restart hit a ConfigMap
subPath mounting race, leaving config.json empty/unparseable and causing
CrashLoopBackOff. This undermines the HA work.
Root cause: Kubernetes subPath ConfigMap mounts can race during pod
restart, resulting in empty or incomplete files before the container starts.
Changes:
- Add copy-config init-container that copies ConfigMap files to emptyDir
- Replace subPath mount with directory mount from emptyDir
- Config files are now guaranteed to be complete before Stalwart starts
- Eliminates the ConfigMap subPath mounting race entirely
This is a critical hardening fix for production mail server stability.
Resolves DEV-431.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This reverts commit c060c83. The TOML configuration approach caused
Stalwart to fail to start due to configuration parsing errors.
Will implement a different approach that doesn't require a full
configuration file rewrite.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Stalwart was blocking the HTTP port (8080) from Traefik's internal IP
(10.244.2.227), causing 502 errors when accessing mail.basicstack.de.
Changes:
- Added complete Stalwart TOML configuration (stalwart-config.yaml)
- Configured HTTP listener security to allow internal pod network (10.244.0.0/16)
- Updated StatefulSet to use the new configuration file
- This allows Traefik ingress to reach the Stalwart web UI backend
The fix is non-destructive:
- PVC data is preserved
- Rolling update will restart the pod with new config
- Only security setting is changed (adding allowed IPs)
Fixes: DEV-422
Co-Authored-By: Paperclip <noreply@paperclip.ing>
OpenCloud has been migrated to use the built-in IDM service which
provides an internal LDAP server (localhost:9236). The external
OpenLDAP deployment is no longer needed.
Changes:
- Remove openldap-deployment.yaml and related configuration files
- Remove openldap-admin-sealed.yaml (admin credentials)
- Remove ldap-init-structure.ldif (initialization script)
- Remove opencloud-ldap-schema files (schema configuration)
- Remove OPENLDAP_DEPLOYMENT.md documentation
- Remove OPENLDAP_ADMIN_PASSWORD env var from OpenCloud deployment
The OpenCloud deployment now relies solely on the built-in IDM
service for user and group storage via its internal LDAP interface.
Resolves: DEV-421
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The pod was crash-looping because Stalwart's security configuration
blocks the kubelet's IP (10.244.4.1) from accessing the HTTP health
endpoints. The kubelet's health checks were failing, causing the
startup probe to fail after 6 attempts, leading to pod restarts.
Changed all three health probes (startup, liveness, readiness) from
httpGet to exec with curl localhost. This bypasses the IP blocking
since the health check runs from inside the container using localhost,
which is not subject to Stalwart's external IP blocking rules.
This fix is non-destructive to Stalwart's configuration and state.
The pod will restart once with the new probe configuration, but no
data or configuration will be lost.
Root cause: Stalwart logs showed "Blocked IP address (security.ip-blocked)
listenerId=http, remoteIp=10.244.4.1" followed by "Shutting down Stalwart
Server (server.shutdown) causedBy=SIGTERM" in a repeating pattern.
Fixes: DEV-420
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Pangolin runs three servers:
- Dashboard API on port 3000
- Internal API on port 3001
- Web UI on port 3002
The current configuration was routing traffic to port 3001 (Internal API), which
returns 404 for the root path, causing health probe failures and restart loops.
Changes:
- Update containerPort from 3001 to 3002
- Update livenessProbe to check port 3002
- Update readinessProbe to check port 3002
- Update Service to expose port 3002
- Update Ingress to route to port 3002
This allows health probes to pass (port 3002 returns 200 for /) and routes
web traffic to the correct Web UI server.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
