Update liveness and readiness probe ports from 8080 to 9090 to match the
controller's METRICS_ADDR configuration. The controller is configured to
serve metrics on port 9090, but the probes were checking port 8080,
causing pods to remain in NotReady state despite the controller
functioning correctly.
Also update containerPort to 9090 for consistency.
Related: DEV-437
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Extend ClusterRole to include all Traefik CRDs that the controller manages:
- serverstransports, serverstransporttcps (for transport configuration)
- ingressroutetcps, ingressrouteudps (for TCP/UDP routing)
- middlewaretcps (for TCP middleware)
- tlsoptions, tlsstores (for TLS configuration)
This fixes the RBAC permission error preventing the controller from
reconciling Traefik configuration: "serverstransports.traefik.io is
forbidden: User \"system:serviceaccount:pangolin:pangolin-controller\"
cannot list resource \"serverstransports\""
Related: DEV-437
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Update controller configuration to fix 401 authentication errors:
- Add CONFIG_AUTH_HEADER with Bearer token for Pangolin API access
- Correct CONFIG_ENDPOINT port from 3000 to 3002 (matches Pangolin service)
- Remove duplicate controller manifests from controller/ subdirectory
The existing controller deployment was failing with 401 errors because
it lacked authentication credentials. This change adds the API key
provided in DEV-437 and corrects the internal service endpoint.
Related: DEV-437
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move Kubernetes controller manifests from kube-controller/ subdirectory
to the apps/pangolin/ root directory so Argo CD will sync them.
Files renamed with pangolin-controller- prefix:
- configmap.yaml -> pangolin-controller-config.yaml
- deployment.yaml -> pangolin-controller-deployment.yaml
- rbac.yaml -> pangolin-controller-rbac.yaml
- service.yaml -> pangolin-controller-service.yaml
Argo CD was not recursing into the subdirectory, so controller resources
were not being synced. Moving to root directory resolves this.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Use specific image version 0.1.0-alpha.1 instead of :latest
- Fix API endpoint port from 3001 to 3000 (correct Pangolin service port)
- Add CONFIG_ALLOW_INSECURE_HTTP for internal cluster HTTP communication
- Fix security context with numeric UIDs (65532) to avoid runAsNonRoot error
- Add required RBAC permissions for CRD and IngressClass discovery
Controller now successfully starts and acquires leader lease.
Next step: Configure API authentication (CONFIG_AUTH_HEADER) for
controller to access Pangolin API endpoint.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Create complete manifest set for Pangolin controller deployment:
- RBAC with ServiceAccount, ClusterRole, and ClusterRoleBinding
- ConfigMap with controller configuration (endpoint, namespace, leader election)
- Deployment with resource limits, health probes, and security context
- Service for metrics endpoint on port 8080
Controller will manage Traefik CRDs (IngressRoute, Middleware, TraefikService)
and sync configuration from Pangolin API.
Relates to DEV-397
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The Pangolin setup was failing with 404 on /api/v1/auth/set-server-admin
because the ingress was routing all traffic to port 3002 (Web UI), but
API endpoints are served by a separate API server on port 3000.
Changes:
- Added port 3000 (api) to deployment container ports
- Updated service to expose both port 3000 (api) and 3002 (http)
- Added /api path to ingress routing to port 3000
- Kept / path routing to port 3002 for Web UI
This allows the browser to make API calls to /api/* which will now
be routed to the API server, fixing the 404 error.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Fixes server action 404 error preventing initial setup.
Version 1.21.1 was released on 2026-07-30 and includes fixes
for Next.js server action routing issues.
Related: DEV-396
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Changed Traefik entrypoints from 'websecure' (HTTPS-only) to
'web,websecure' (HTTP + HTTPS) to allow Let's Encrypt HTTP-01
challenge verification on port 80.
The TLS section in the ingress will still redirect HTTP to HTTPS
for normal traffic after the certificate is issued.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The DNS-01 challenge has been stuck for 5+ days due to Hetzner DNS
webhook issues. Switched to the more reliable HTTP-01 challenge method
by using the new letsencrypt-http01 ClusterIssuer.
This resolves the missing certificate issue for pangolin.basicstack.de.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Test documentation for the ConfigMap subPath mount hardening deployed
in commit 291feb9.
Results:
- Pod deletion test: 3/3 passed with 0 restarts (69s, ~80s, 93s recovery)
- Alert test: Completed but found Alertmanager sends to null receiver
- Mail service health: Verified healthy after all tests
DEV-431.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Extends commit 291feb9 by adding stalwart-config (stalwart.toml) to the
copy-config initContainer. The previous fix only handled bootstrap-config
but missed stalwart-config, which was the root cause of CrashLoopBackOff
identified in DEV-426 stability testing.
Root cause: commit 693fcd3 introduced stalwart-config ConfigMap with
subPath mount. During pod restarts, Kubernetes subPath ConfigMap mounting
race leaves stalwart.toml empty/unparseable, causing:
"Failed to parse data store settings at /etc/stalwart/stalwart.toml:
expected value at line 1 column 1"
This completes the hardening by ensuring BOTH config files (config.json
and stalwart.toml) are atomically copied before Stalwart starts.
Fixes: DEV-433, DEV-431
Ref: stack.basicstack.de/apps/stalwart/STABILITY-VERIFICATION-2026-08-01.md
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Issue: During stability testing (DEV-426), pod restart hit a ConfigMap
subPath mounting race, leaving config.json empty/unparseable and causing
CrashLoopBackOff. This undermines the HA work.
Root cause: Kubernetes subPath ConfigMap mounts can race during pod
restart, resulting in empty or incomplete files before the container starts.
Changes:
- Add copy-config init-container that copies ConfigMap files to emptyDir
- Replace subPath mount with directory mount from emptyDir
- Config files are now guaranteed to be complete before Stalwart starts
- Eliminates the ConfigMap subPath mounting race entirely
This is a critical hardening fix for production mail server stability.
Resolves DEV-431.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This reverts commit c060c83. The TOML configuration approach caused
Stalwart to fail to start due to configuration parsing errors.
Will implement a different approach that doesn't require a full
configuration file rewrite.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Stalwart was blocking the HTTP port (8080) from Traefik's internal IP
(10.244.2.227), causing 502 errors when accessing mail.basicstack.de.
Changes:
- Added complete Stalwart TOML configuration (stalwart-config.yaml)
- Configured HTTP listener security to allow internal pod network (10.244.0.0/16)
- Updated StatefulSet to use the new configuration file
- This allows Traefik ingress to reach the Stalwart web UI backend
The fix is non-destructive:
- PVC data is preserved
- Rolling update will restart the pod with new config
- Only security setting is changed (adding allowed IPs)
Fixes: DEV-422
Co-Authored-By: Paperclip <noreply@paperclip.ing>
OpenCloud has been migrated to use the built-in IDM service which
provides an internal LDAP server (localhost:9236). The external
OpenLDAP deployment is no longer needed.
Changes:
- Remove openldap-deployment.yaml and related configuration files
- Remove openldap-admin-sealed.yaml (admin credentials)
- Remove ldap-init-structure.ldif (initialization script)
- Remove opencloud-ldap-schema files (schema configuration)
- Remove OPENLDAP_DEPLOYMENT.md documentation
- Remove OPENLDAP_ADMIN_PASSWORD env var from OpenCloud deployment
The OpenCloud deployment now relies solely on the built-in IDM
service for user and group storage via its internal LDAP interface.
Resolves: DEV-421
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The pod was crash-looping because Stalwart's security configuration
blocks the kubelet's IP (10.244.4.1) from accessing the HTTP health
endpoints. The kubelet's health checks were failing, causing the
startup probe to fail after 6 attempts, leading to pod restarts.
Changed all three health probes (startup, liveness, readiness) from
httpGet to exec with curl localhost. This bypasses the IP blocking
since the health check runs from inside the container using localhost,
which is not subject to Stalwart's external IP blocking rules.
This fix is non-destructive to Stalwart's configuration and state.
The pod will restart once with the new probe configuration, but no
data or configuration will be lost.
Root cause: Stalwart logs showed "Blocked IP address (security.ip-blocked)
listenerId=http, remoteIp=10.244.4.1" followed by "Shutting down Stalwart
Server (server.shutdown) causedBy=SIGTERM" in a repeating pattern.
Fixes: DEV-420
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Pangolin runs three servers:
- Dashboard API on port 3000
- Internal API on port 3001
- Web UI on port 3002
The current configuration was routing traffic to port 3001 (Internal API), which
returns 404 for the root path, causing health probe failures and restart loops.
Changes:
- Update containerPort from 3001 to 3002
- Update livenessProbe to check port 3002
- Update readinessProbe to check port 3002
- Update Service to expose port 3002
- Update Ingress to route to port 3002
This allows health probes to pass (port 3002 returns 200 for /) and routes
web traffic to the correct Web UI server.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
CONTEXT: DEV-403 - Stalwart pod not starting after ConfigMap revert
ROOT CAUSE:
- The stalwart-bootstrap-config ConfigMap was manually created on July 5
- It was added to GitOps at commit 191614f (12:27 today)
- It was reverted at commit fcc5042 (12:56 today)
- ArgoCD pruned the ConfigMap from the cluster after the revert
- Stalwart deployment requires this ConfigMap to start
RECOVERY:
This commit restores the ConfigMap to Git so ArgoCD can recreate it.
The ConfigMap content is identical to the working version from before.
This is a non-destructive recovery - no data or accounts are affected.
The actual Stalwart data on PVC stalwart-data remains intact.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The stalwart-bootstrap-config ConfigMap was previously created manually
via kubectl and not tracked in Git. This adds it to the repository for
proper GitOps management via Argo CD.
Related to DEV-403 - Stalwart sync hanging issue.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Mount config file to /app/config/config.toml instead of /app/config.toml as the application expects it in the config directory.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Pangolin requires a config.toml file to start. Added ConfigMap with basic configuration including URL, database path, and OIDC settings. Updated deployment to mount the config file.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Cluster nodes are at 93-99% memory capacity. Reduced requests from 512Mi to 256Mi and CPU from 200m to 100m to allow pod scheduling.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Created apps/pangolin/ directory with namespace.yaml defining the pangolin namespace.
Configured DNS A record for pangolin.basicstack.de → 178.105.17.239 (cluster ingress IP).
Updated DNS_REQUIREMENTS.md to document the new Pangolin service.
This completes Phase 1 of the Pangolin deployment (DEV-390):
- Repository structure created with namespace definition
- DNS record configured and verified in Hetzner zone
- Documentation updated
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Change PVC storage class from local-path to hcloud-volumes-encrypted
- Extend initContainer to create full directory structure including instances/default/data/run-logs
- This fixes permission errors when agents try to create run logs
Resolves: DEV-388
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The /api/health endpoint returns 403 when accessed externally but works from localhost. Changed readiness and liveness probes to use exec with wget from inside the container to work around this restriction.
Resolves: DEV-387
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Change health check from /health to /api/health to match the actual API endpoint. The server is healthy but the readiness/liveness probes were failing because they were checking the wrong path.
Resolves: DEV-387
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Change paperclip deployment to use paperclip-data-encrypted PVC (preserves existing data)
- Remove pvc.yaml (use existing PVC instead of creating new one)
- Remove postgres StatefulSet and service (use existing postgres deployment)
This preserves the existing 43-day-old data in the encrypted volumes instead of creating a fresh deployment.
Resolves: DEV-387
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Reduce resource requests from 500m/1Gi to 200m/512Mi to fit available cluster capacity
- Reduce resource limits from 2000m/4Gi to 1000m/2Gi for better resource sharing
- Add IgnoreExtraneous annotation to paperclip-secrets SealedSecret to fix ArgoCD degraded status
Resolves: DEV-387
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add all necessary Kubernetes manifests for Paperclip deployment:
- deployment.yaml: Main Paperclip application deployment
- service.yaml: ClusterIP service for Paperclip (port 3100)
- ingress.yaml: Traefik ingress for paperclip.home.imicros.de
- pvc.yaml: 50Gi persistent volume for Paperclip data
- postgres-statefulset.yaml: PostgreSQL 16 StatefulSet
- postgres-service.yaml: PostgreSQL service
This completes the GitOps configuration for Paperclip in the
apps/paperclip directory. The ArgoCD application at apps/app-paperclip.yaml
is already configured to deploy from this path.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move backup resources from separate 'backup' namespace to 'forgejo':
- Change CronJob namespace from 'backup' to 'forgejo'
- Reuse existing forgejo-postgres-secret instead of duplicate backup secret
- Remove backup-namespace.yaml (no longer needed)
- Remove forgejo-backup-secret-sealed.yaml (using existing secret)
This simplifies the architecture by keeping all Forgejo-related
resources in a single namespace and eliminating credential duplication.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move backup configuration from separate app to forgejo folder:
- Move forgejo-backup-cronjob.yaml to apps/forgejo/
- Move forgejo-backup-secret-sealed.yaml to apps/forgejo/
- Add backup-namespace.yaml to ensure backup namespace is created
- Remove apps/backup/ folder
- Remove apps/app-backup.yaml Argo CD application
This consolidates the backup configuration into the main Forgejo
application, eliminating the need for a separate Argo CD app.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add IgnoreExtraneous annotation to MariaDB sealed secret template metadata.
This prevents ArgoCD from seeing the unsealed secret (created by sealed-secrets controller) as an extraneous resource.
Same fix pattern as DEV-377, DEV-378, and DEV-379.
Resolves DEV-385
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This file was replaced by passbolt-ingress-fixed.yaml in the Passbolt
certificate fix (DEV-382). The new ingress uses cert-manager annotations
instead of Traefik-specific TLS configuration.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The ingress was configured with ingressClassName: nginx, but the cluster
only has Traefik ingress controller installed. This caused the browser to
show Traefik's default certificate instead of the Let's Encrypt certificate.
Changed to ingressClassName: traefik to match all other ingresses in the
cluster (argocd, forgejo, directus, etc.)
Fixes: DEV-382
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Add all Kubernetes manifest files for Passbolt deployment
- Add MariaDB deployment, service, PVC manifests
- Add Passbolt deployment, service, PVC, ingress manifests
- Add namespace manifest
- Update passbolt-secret to include SMTP authentication
- Add README with initial admin user setup instructions
The configuration was extracted from the running cluster and organized
into separate manifest files for better maintainability.
SMTP is configured to use Stalwart mail server with:
- Username: passbolt
- Email: passbolt@basicstack.de
- Host: stalwart-mail.stalwart.svc.cluster.local
Co-Authored-By: Paperclip <noreply@paperclip.ing>
