Changed liveness probe from HTTP /server/health to TCP socket check
and readiness probe to use root path instead. The /server/health
endpoint in Directus v12+ requires authentication by default,
causing 403 responses that triggered pod restarts.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the Directus license key (DP3WZ-8DT8F-79FME-5HJ1A-E4EMJ) as a sealed secret and configures the deployment to use it via the LICENSE_KEY environment variable.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Moved Directus deployment configuration from k8s cluster to Git repository:
- Main Directus deployment updated to v12.1.1
- PostgreSQL deployment (postgres:16-alpine)
- Services for both Directus and PostgreSQL
- Ingress with TLS/cert-manager
- Backup CronJob (daily at 2 AM)
- PVCs for database and uploads
This enables GitOps management via Argo CD.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Dozzle pod was crashing with:
pods.metrics.k8s.io is forbidden: User "system:serviceaccount:dozzle:dozzle"
cannot list resource "pods" in API group "metrics.k8s.io" at the cluster scope
Added permission for the metrics.k8s.io API group to allow Dozzle to collect
pod metrics for its monitoring dashboard.
Fixes: DEV-372
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The websocket connection drops were caused by oauth2-proxy's upstream-timeout
being set to only 30s. Websocket connections are long-lived and need much
longer timeouts. Increased to 3600s (1 hour) per Traefik websocket best practices.
Fixes: DEV-371
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add Traefik annotations for websocket support:
- Enable sticky sessions to ensure websocket connections go to same pod
- Add custom sticky cookie name for better tracking
- Pass host header for proper routing
Resolves DEV-371
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Changed both stalwart-smtp and stalwart-imap services from
externalTrafficPolicy: Local to externalTrafficPolicy: Cluster.
This enables all 4 k3s nodes to appear healthy in Hetzner Load
Balancer health checks, instead of only k3s-worker-1 (where the
single Stalwart pod runs).
With Cluster policy, traffic arriving at any node will be
forwarded via kube-proxy to k3s-worker-1 where the pod is running.
Verification:
- All 4 nodes now show in service EXTERNAL-IP field
- Mail ports (25, 587, 993) accessible from all nodes
- Connectivity tested from k3s-worker-1 and k3s-worker-2
Fixes: DEV-368
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add securityContext configuration to the database StatefulSet to ensure
PostgreSQL can access its data directory with correct permissions.
The issue was that the PVC mount permissions did not match PostgreSQL's
requirements (u=rwx/0700 or u=rwx,g=rx/0750). This fix sets:
- runAsUser: 999 (postgres user)
- fsGroup: 999 (postgres group)
- fsGroupChangePolicy: OnRootMismatch (only change ownership when needed)
This resolves the crash loop where harbor-database-0 failed with:
'data directory has invalid permissions'
Fixes: DEV-364
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Drop legacy insecure mail ports per DEV-359 approval:
- Remove port 465 (SMTPS) from SMTP service - enforce STARTTLS on 587
- Remove port 143 (IMAP) from IMAP service - enforce TLS on 993
This reduces attack surface and enforces secure mail protocols.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: Dozzle was being rate-limited by the Kubernetes API when
fetching logs from many containers. The aggressive 1-second health check
timeouts caused the container to fail probes and restart repeatedly.
Changes:
- Increased all probe timeouts from 1s to 5s to handle API rate limiting
- Increased liveness probe period from 10s to 30s to reduce check frequency
- Increased readiness probe period from 5s to 10s
- Increased startup probe failure threshold from 15 to 20 (60s total startup time)
- Increased CPU limit from 500m to 1000m and request from 100m to 250m
- Increased memory limit from 512Mi to 1Gi and request from 128Mi to 256Mi
- Pinned image version to v10.6.10 instead of :latest
This prevents unnecessary restarts when Dozzle is waiting on rate-limited
Kubernetes API calls, fixing both the initial 503 errors and the connection
drops after prolonged use.
Resolves: DEV-351
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Fixes DEV-350. During pod startup, Dozzle takes ~11 seconds to start
accepting connections, but oauth2-proxy can receive and proxy requests
immediately. This causes "connection refused" errors when users access
the UI right after a pod restart.
Solution:
- Add startupProbe with 30s timeout (15 failures × 2s) to give Dozzle
time to start without failing readiness
- Add readinessProbe to prevent traffic routing until Dozzle is ready
- Add livenessProbe to restart container if Dozzle becomes unhealthy
All probes use the /healthcheck endpoint on port 8080.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The oauth2-proxy container was failing with "unknown flag: --proxy-buffer-size".
This flag is not valid for oauth2-proxy and has been removed.
Fixes DEV-349
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added oauth2-proxy flags to properly handle Server-Sent Events (SSE):
- --flush-interval=1s: Flush buffered data regularly for SSE
- --upstream-timeout=30s: Allow longer streaming connections
- --proxy-buffer-size=8k: Reduce buffering latency
Simplified ingress configuration to standard Traefik setup.
OAuth2-proxy was buffering responses by default, which broke the
WebSocket/SSE endpoints used for real-time log streaming. The
flush-interval flag ensures data is streamed properly.
Related: DEV-349
Co-Authored-By: Paperclip <noreply@paperclip.ing>
oauth2-proxy was resolving localhost to IPv6 [::1]:8080 instead of
IPv4 127.0.0.1:8080, causing connection refused errors on the streaming
endpoints (/api/events/stream and /api/hosts/.../logs/stream).
Changed upstream URL from http://localhost:8080 to http://127.0.0.1:8080
to force IPv4 resolution.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added WebSocket support configuration to both oauth2-proxy and Traefik ingress:
- Enable explicit WebSocket proxying in oauth2-proxy (--proxy-websockets=true)
- Pass Host header through oauth2-proxy (--pass-host-header=true)
- Add WebSocket upgrade headers to Traefik ingress annotations
This fixes the "UI cannot connect to API" error by ensuring real-time
log streaming WebSocket connections work through the authentication proxy.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add Traefik WebSocket annotations to ingress to enable real-time log
streaming. Without these, the UI cannot establish WebSocket connections
to the API backend.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Dozzle running in k8s mode requires permission to list nodes at cluster
scope. Without this permission, the pod fails with error:
"nodes is forbidden: User \"system:serviceaccount:dozzle:dozzle\" cannot
list resource \"nodes\" in API group \"\" at the cluster scope"
This change adds the nodes resource with get, list, and watch verbs to
the ClusterRole, allowing Dozzle to discover all nodes in the cluster
when running in k8s/swarm mode.
Resolves: DEV-349
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Trying to resolve "Could not connect to any Docker Engine" error by letting Dozzle auto-detect its environment without explicit flags.
Current blocker: Dozzle v10.6.10 is not auto-detecting Kubernetes mode despite having proper RBAC and service account configured.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Revert to :latest image (v8.9.4 doesn't exist) and add --agent flag to help Dozzle detect and work in Kubernetes mode.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Switch from :latest to v8.9.4 for stability and add --no-analytics flag to improve startup behavior in Kubernetes.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Switch from direct OIDC (not supported by Dozzle) to forward-proxy authentication using oauth2-proxy as a sidecar container.
Changes:
- Add oauth2-proxy sidecar container for OIDC authentication
- Configure Dozzle to use forward-proxy auth with user headers
- Update service and ingress to route to oauth2-proxy (port 4180)
- Add cookie-secret to sealed secret for oauth2-proxy session management
- Update documentation to reflect oauth2-proxy architecture
The oauth2-proxy authenticates users via Pocket ID and forwards requests to Dozzle with X-Forwarded-User, X-Forwarded-Email, and X-Forwarded-Preferred-Username headers.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Deploy Dozzle (https://dozzle.dev/) for real-time container log viewing in the k8s cluster.
Configuration:
- Namespace: dozzle
- Domain: dozzle.basicstack.de
- Storage: 1Gi PVC with hcloud-volumes-encrypted storage class
- Authentication: Pocket ID OIDC integration
- RBAC: Cluster-wide pod log access via service account
- Strategy: Recreate with ReadWriteOnce PVC for persistent settings
Created Pocket ID OIDC client:
- Client ID: 179c13f2-d251-4e1e-b1a0-c070df350c4e
- Callback URL: https://dozzle.basicstack.de/oauth/callback
ArgoCD application configured with automated sync and self-heal.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move basicstack.org deployment manifests from basicstack.org/k8s to
stack.basicstack.de/apps/basicstack-org to centralize infrastructure
configuration and use Harbor registry.
Changes:
- Created apps/basicstack-org/ with deployment manifests
- Updated Argo CD app to point to stack.basicstack.de repo
- Deployment now uses Harbor registry at 10.106.73.119:5000
- Set imagePullPolicy: Always for continuous deployment
Related: DEV-336
Co-Authored-By: Paperclip <noreply@paperclip.ing>
dind sidecar had networking issue: host-networked job containers can't reach
the dind daemon inside the pod. Back to host socket with docker binary
bind-mounted into job containers via container.options.
The forgejo-runner image doesn't include Docker CLI by default, causing
the startup script to hang waiting for Docker to be ready. Install
docker-cli via apk during startup to enable Docker socket access.
This completes the DinD setup for the runner.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Replaces the Deployment-based runner with a StatefulSet using a proper
Docker-in-Docker sidecar for improved isolation and state management.
Key changes:
- StatefulSet deployment for stable pod identity and persistent storage
- Docker-in-Docker init container (privileged) providing isolated Docker daemon
- Two persistent volumes: runner-data (1Gi) for config, docker-data (20Gi) for image cache
- Uses hcloud-volumes storage class for durability
- Each replica gets its own Docker daemon and image cache
This addresses the workflow failure where job containers (node:24-bookworm)
couldn't access Docker because the CLI was missing. The DinD sidecar provides
a complete Docker environment for all job containers.
Fixes: DEV-335
Co-Authored-By: Paperclip <noreply@paperclip.ing>
act_runner v4.0.1 automatically propagates the docker socket when
DOCKER_HOST env var is set. Adding the socket via 'container.options'
caused duplicate bind mounts which made Docker fail container creation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Job containers need access to the host Docker socket to run docker
build/push commands. Adding -v /var/run/docker.sock:/var/run/docker.sock
to container options so docker CLI in job containers can reach the
host Docker daemon.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Docker containers launched by the runner on the host cannot resolve
forgejo.forgejo.svc.cluster.local (k8s cluster DNS is not available
from host Docker containers). Switching to https://forgejo.basicstack.de
for runner registration so GITHUB_SERVER_URL in job context uses the
externally reachable URL, fixing git fetch in actions/checkout.
Also adds detection to force re-registration if a pod starts with the
old internal-URL .runner file still present.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Remove --user 1000:1000 option to allow workflow containers to run as root.
This is needed for the Docker CLI installation step which requires root
permissions for apt-get commands.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Change runner container network from bridge to host to allow workflow
containers to resolve Kubernetes service DNS names. With bridge network,
containers couldn't resolve forgejo.forgejo.svc.cluster.local.
Using host network gives containers access to the cluster's DNS resolver.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add --add-host mapping for forgejo.forgejo.svc.cluster.local to the
Forgejo service ClusterIP (10.102.251.20) in the runner container options.
This fixes the DNS resolution issue where workflow containers created by
the runner (via Docker-in-Docker) cannot resolve Kubernetes service DNS
names, causing the actions/checkout step to fail with "Could not resolve
host: forgejo.forgejo.svc.cluster.local".
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Run the forgejo-runner container as root (uid 0) to allow access to the
Docker socket. This is required for Docker-in-Docker functionality needed
by Forgejo Actions workflows.
Without this, the runner fails with "permission denied" when trying to
access /var/run/docker.sock.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Create sealed secret for forgejo-runner-token to enable Forgejo Actions
runner deployment. This token allows the runner to register with the
Forgejo instance and execute CI/CD workflows.
Fixes basicstack.org workflow failures (DEV-335) caused by missing runner.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Create runner deployment with:
- ServiceAccount and RBAC for runner pod
- ConfigMap for runner configuration
- Deployment using code.forgejo.org/forgejo/runner:4.0.1
- Argo CD application for automated deployment
Note: Runner requires a sealed secret with registration token.
See apps/forgejo-runner/README.md for setup instructions.
Part of DEV-334 CI/CD workflow implementation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add FORGEJO__actions__ENABLED and DEFAULT_ACTIONS_URL environment
variables to enable Actions/CI workflows in Forgejo.
This is required for DEV-334 CI/CD workflow implementation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move Helm chart configuration from inline values in app-harbor.yaml to
separate files in apps/harbor/ subdirectory, following the same pattern
as forgejo and other apps.
Changes:
- Create apps/harbor/Chart.yaml defining dependency on Harbor Helm chart
- Create apps/harbor/values.yaml with all Helm values configuration
- Update app-harbor.yaml to use git path source instead of direct Helm chart
- Add apps/harbor/README.md documenting OIDC setup procedure
OIDC authentication must be configured via Harbor UI after deployment,
as the Helm chart does not support OIDC configuration at deployment time.
The README provides step-by-step instructions for Pocket ID integration.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Created Argo CD Application for Harbor (app-harbor.yaml)
- Configured Harbor Helm chart with:
- Ingress at harbor.basicstack.de with TLS via cert-manager
- PVCs using hcloud-volumes-encrypted storage class
- OIDC authentication via Pocket ID
- Resource limits for all components
- Created sealed secret with Harbor admin password, database password, and OIDC client secret
- Configured DNS A record for harbor.basicstack.de -> 178.105.17.239
- Created Pocket ID OIDC client for Harbor with callback URL
Harbor will be available at https://harbor.basicstack.de after Argo CD sync.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
