Pangolin requires a config.toml file to start. Added ConfigMap with basic configuration including URL, database path, and OIDC settings. Updated deployment to mount the config file.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Cluster nodes are at 93-99% memory capacity. Reduced requests from 512Mi to 256Mi and CPU from 200m to 100m to allow pod scheduling.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Created apps/pangolin/ directory with namespace.yaml defining the pangolin namespace.
Configured DNS A record for pangolin.basicstack.de → 178.105.17.239 (cluster ingress IP).
Updated DNS_REQUIREMENTS.md to document the new Pangolin service.
This completes Phase 1 of the Pangolin deployment (DEV-390):
- Repository structure created with namespace definition
- DNS record configured and verified in Hetzner zone
- Documentation updated
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Change PVC storage class from local-path to hcloud-volumes-encrypted
- Extend initContainer to create full directory structure including instances/default/data/run-logs
- This fixes permission errors when agents try to create run logs
Resolves: DEV-388
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The /api/health endpoint returns 403 when accessed externally but works from localhost. Changed readiness and liveness probes to use exec with wget from inside the container to work around this restriction.
Resolves: DEV-387
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Change health check from /health to /api/health to match the actual API endpoint. The server is healthy but the readiness/liveness probes were failing because they were checking the wrong path.
Resolves: DEV-387
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Change paperclip deployment to use paperclip-data-encrypted PVC (preserves existing data)
- Remove pvc.yaml (use existing PVC instead of creating new one)
- Remove postgres StatefulSet and service (use existing postgres deployment)
This preserves the existing 43-day-old data in the encrypted volumes instead of creating a fresh deployment.
Resolves: DEV-387
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Reduce resource requests from 500m/1Gi to 200m/512Mi to fit available cluster capacity
- Reduce resource limits from 2000m/4Gi to 1000m/2Gi for better resource sharing
- Add IgnoreExtraneous annotation to paperclip-secrets SealedSecret to fix ArgoCD degraded status
Resolves: DEV-387
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add all necessary Kubernetes manifests for Paperclip deployment:
- deployment.yaml: Main Paperclip application deployment
- service.yaml: ClusterIP service for Paperclip (port 3100)
- ingress.yaml: Traefik ingress for paperclip.home.imicros.de
- pvc.yaml: 50Gi persistent volume for Paperclip data
- postgres-statefulset.yaml: PostgreSQL 16 StatefulSet
- postgres-service.yaml: PostgreSQL service
This completes the GitOps configuration for Paperclip in the
apps/paperclip directory. The ArgoCD application at apps/app-paperclip.yaml
is already configured to deploy from this path.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move backup resources from separate 'backup' namespace to 'forgejo':
- Change CronJob namespace from 'backup' to 'forgejo'
- Reuse existing forgejo-postgres-secret instead of duplicate backup secret
- Remove backup-namespace.yaml (no longer needed)
- Remove forgejo-backup-secret-sealed.yaml (using existing secret)
This simplifies the architecture by keeping all Forgejo-related
resources in a single namespace and eliminating credential duplication.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move backup configuration from separate app to forgejo folder:
- Move forgejo-backup-cronjob.yaml to apps/forgejo/
- Move forgejo-backup-secret-sealed.yaml to apps/forgejo/
- Add backup-namespace.yaml to ensure backup namespace is created
- Remove apps/backup/ folder
- Remove apps/app-backup.yaml Argo CD application
This consolidates the backup configuration into the main Forgejo
application, eliminating the need for a separate Argo CD app.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add IgnoreExtraneous annotation to MariaDB sealed secret template metadata.
This prevents ArgoCD from seeing the unsealed secret (created by sealed-secrets controller) as an extraneous resource.
Same fix pattern as DEV-377, DEV-378, and DEV-379.
Resolves DEV-385
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This file was replaced by passbolt-ingress-fixed.yaml in the Passbolt
certificate fix (DEV-382). The new ingress uses cert-manager annotations
instead of Traefik-specific TLS configuration.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The ingress was configured with ingressClassName: nginx, but the cluster
only has Traefik ingress controller installed. This caused the browser to
show Traefik's default certificate instead of the Let's Encrypt certificate.
Changed to ingressClassName: traefik to match all other ingresses in the
cluster (argocd, forgejo, directus, etc.)
Fixes: DEV-382
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Add all Kubernetes manifest files for Passbolt deployment
- Add MariaDB deployment, service, PVC manifests
- Add Passbolt deployment, service, PVC, ingress manifests
- Add namespace manifest
- Update passbolt-secret to include SMTP authentication
- Add README with initial admin user setup instructions
The configuration was extracted from the running cluster and organized
into separate manifest files for better maintainability.
SMTP is configured to use Stalwart mail server with:
- Username: passbolt
- Email: passbolt@basicstack.de
- Host: stalwart-mail.stalwart.svc.cluster.local
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Change deployment strategy from RollingUpdate (default) to Recreate
to resolve volume attachment conflict. The forgejo-data PVC uses
ReadWriteOnce access mode, which only allows attachment to one pod
at a time. RollingUpdate tries to start new pod before terminating
old pod, causing multi-attach error.
With Recreate strategy, old pod terminates first, volume detaches,
then new pod starts and successfully attaches the volume.
Resolves: DEV-380
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add argocd.argoproj.io/compare-options: IgnoreExtraneous annotation to
both Stalwart sealed secret templates (stalwart-admin-credentials and
stalwart-s3-backup) to prevent ArgoCD from seeing the unsealed secrets
(created by sealed-secrets controller) as extraneous.
This is the same fix applied in DEV-377 (Directus) and DEV-378 (Harbor).
Resolves: DEV-379
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add argocd.argoproj.io/compare-options: IgnoreExtraneous annotation to
harbor-secrets sealed secret template to prevent ArgoCD from seeing the
unsealed secret (created by sealed-secrets controller) as extraneous.
This is the same fix applied in DEV-377 for Directus and previously in
DEV-289, DEV-290 for other services.
Resolves: DEV-378
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This tells Argo CD to ignore the unsealed secrets created by the sealed-secrets controller, preventing the health status from being degraded.
This is the same fix applied previously in DEV-289, DEV-290, and to Pocket ID for the same issue.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Changed liveness probe from HTTP /server/health to TCP socket check
and readiness probe to use root path instead. The /server/health
endpoint in Directus v12+ requires authentication by default,
causing 403 responses that triggered pod restarts.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the Directus license key (DP3WZ-8DT8F-79FME-5HJ1A-E4EMJ) as a sealed secret and configures the deployment to use it via the LICENSE_KEY environment variable.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Moved Directus deployment configuration from k8s cluster to Git repository:
- Main Directus deployment updated to v12.1.1
- PostgreSQL deployment (postgres:16-alpine)
- Services for both Directus and PostgreSQL
- Ingress with TLS/cert-manager
- Backup CronJob (daily at 2 AM)
- PVCs for database and uploads
This enables GitOps management via Argo CD.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Dozzle pod was crashing with:
pods.metrics.k8s.io is forbidden: User "system:serviceaccount:dozzle:dozzle"
cannot list resource "pods" in API group "metrics.k8s.io" at the cluster scope
Added permission for the metrics.k8s.io API group to allow Dozzle to collect
pod metrics for its monitoring dashboard.
Fixes: DEV-372
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The websocket connection drops were caused by oauth2-proxy's upstream-timeout
being set to only 30s. Websocket connections are long-lived and need much
longer timeouts. Increased to 3600s (1 hour) per Traefik websocket best practices.
Fixes: DEV-371
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add Traefik annotations for websocket support:
- Enable sticky sessions to ensure websocket connections go to same pod
- Add custom sticky cookie name for better tracking
- Pass host header for proper routing
Resolves DEV-371
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Changed both stalwart-smtp and stalwart-imap services from
externalTrafficPolicy: Local to externalTrafficPolicy: Cluster.
This enables all 4 k3s nodes to appear healthy in Hetzner Load
Balancer health checks, instead of only k3s-worker-1 (where the
single Stalwart pod runs).
With Cluster policy, traffic arriving at any node will be
forwarded via kube-proxy to k3s-worker-1 where the pod is running.
Verification:
- All 4 nodes now show in service EXTERNAL-IP field
- Mail ports (25, 587, 993) accessible from all nodes
- Connectivity tested from k3s-worker-1 and k3s-worker-2
Fixes: DEV-368
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add securityContext configuration to the database StatefulSet to ensure
PostgreSQL can access its data directory with correct permissions.
The issue was that the PVC mount permissions did not match PostgreSQL's
requirements (u=rwx/0700 or u=rwx,g=rx/0750). This fix sets:
- runAsUser: 999 (postgres user)
- fsGroup: 999 (postgres group)
- fsGroupChangePolicy: OnRootMismatch (only change ownership when needed)
This resolves the crash loop where harbor-database-0 failed with:
'data directory has invalid permissions'
Fixes: DEV-364
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Drop legacy insecure mail ports per DEV-359 approval:
- Remove port 465 (SMTPS) from SMTP service - enforce STARTTLS on 587
- Remove port 143 (IMAP) from IMAP service - enforce TLS on 993
This reduces attack surface and enforces secure mail protocols.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: Dozzle was being rate-limited by the Kubernetes API when
fetching logs from many containers. The aggressive 1-second health check
timeouts caused the container to fail probes and restart repeatedly.
Changes:
- Increased all probe timeouts from 1s to 5s to handle API rate limiting
- Increased liveness probe period from 10s to 30s to reduce check frequency
- Increased readiness probe period from 5s to 10s
- Increased startup probe failure threshold from 15 to 20 (60s total startup time)
- Increased CPU limit from 500m to 1000m and request from 100m to 250m
- Increased memory limit from 512Mi to 1Gi and request from 128Mi to 256Mi
- Pinned image version to v10.6.10 instead of :latest
This prevents unnecessary restarts when Dozzle is waiting on rate-limited
Kubernetes API calls, fixing both the initial 503 errors and the connection
drops after prolonged use.
Resolves: DEV-351
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Fixes DEV-350. During pod startup, Dozzle takes ~11 seconds to start
accepting connections, but oauth2-proxy can receive and proxy requests
immediately. This causes "connection refused" errors when users access
the UI right after a pod restart.
Solution:
- Add startupProbe with 30s timeout (15 failures × 2s) to give Dozzle
time to start without failing readiness
- Add readinessProbe to prevent traffic routing until Dozzle is ready
- Add livenessProbe to restart container if Dozzle becomes unhealthy
All probes use the /healthcheck endpoint on port 8080.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The oauth2-proxy container was failing with "unknown flag: --proxy-buffer-size".
This flag is not valid for oauth2-proxy and has been removed.
Fixes DEV-349
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added oauth2-proxy flags to properly handle Server-Sent Events (SSE):
- --flush-interval=1s: Flush buffered data regularly for SSE
- --upstream-timeout=30s: Allow longer streaming connections
- --proxy-buffer-size=8k: Reduce buffering latency
Simplified ingress configuration to standard Traefik setup.
OAuth2-proxy was buffering responses by default, which broke the
WebSocket/SSE endpoints used for real-time log streaming. The
flush-interval flag ensures data is streamed properly.
Related: DEV-349
Co-Authored-By: Paperclip <noreply@paperclip.ing>
oauth2-proxy was resolving localhost to IPv6 [::1]:8080 instead of
IPv4 127.0.0.1:8080, causing connection refused errors on the streaming
endpoints (/api/events/stream and /api/hosts/.../logs/stream).
Changed upstream URL from http://localhost:8080 to http://127.0.0.1:8080
to force IPv4 resolution.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added WebSocket support configuration to both oauth2-proxy and Traefik ingress:
- Enable explicit WebSocket proxying in oauth2-proxy (--proxy-websockets=true)
- Pass Host header through oauth2-proxy (--pass-host-header=true)
- Add WebSocket upgrade headers to Traefik ingress annotations
This fixes the "UI cannot connect to API" error by ensuring real-time
log streaming WebSocket connections work through the authentication proxy.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add Traefik WebSocket annotations to ingress to enable real-time log
streaming. Without these, the UI cannot establish WebSocket connections
to the API backend.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Dozzle running in k8s mode requires permission to list nodes at cluster
scope. Without this permission, the pod fails with error:
"nodes is forbidden: User \"system:serviceaccount:dozzle:dozzle\" cannot
list resource \"nodes\" in API group \"\" at the cluster scope"
This change adds the nodes resource with get, list, and watch verbs to
the ClusterRole, allowing Dozzle to discover all nodes in the cluster
when running in k8s/swarm mode.
Resolves: DEV-349
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Trying to resolve "Could not connect to any Docker Engine" error by letting Dozzle auto-detect its environment without explicit flags.
Current blocker: Dozzle v10.6.10 is not auto-detecting Kubernetes mode despite having proper RBAC and service account configured.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Revert to :latest image (v8.9.4 doesn't exist) and add --agent flag to help Dozzle detect and work in Kubernetes mode.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Switch from :latest to v8.9.4 for stability and add --no-analytics flag to improve startup behavior in Kubernetes.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
