act_runner v4.0.1 automatically propagates the docker socket when
DOCKER_HOST env var is set. Adding the socket via 'container.options'
caused duplicate bind mounts which made Docker fail container creation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Job containers need access to the host Docker socket to run docker
build/push commands. Adding -v /var/run/docker.sock:/var/run/docker.sock
to container options so docker CLI in job containers can reach the
host Docker daemon.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Docker containers launched by the runner on the host cannot resolve
forgejo.forgejo.svc.cluster.local (k8s cluster DNS is not available
from host Docker containers). Switching to https://forgejo.basicstack.de
for runner registration so GITHUB_SERVER_URL in job context uses the
externally reachable URL, fixing git fetch in actions/checkout.
Also adds detection to force re-registration if a pod starts with the
old internal-URL .runner file still present.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Remove --user 1000:1000 option to allow workflow containers to run as root.
This is needed for the Docker CLI installation step which requires root
permissions for apt-get commands.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Change runner container network from bridge to host to allow workflow
containers to resolve Kubernetes service DNS names. With bridge network,
containers couldn't resolve forgejo.forgejo.svc.cluster.local.
Using host network gives containers access to the cluster's DNS resolver.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add --add-host mapping for forgejo.forgejo.svc.cluster.local to the
Forgejo service ClusterIP (10.102.251.20) in the runner container options.
This fixes the DNS resolution issue where workflow containers created by
the runner (via Docker-in-Docker) cannot resolve Kubernetes service DNS
names, causing the actions/checkout step to fail with "Could not resolve
host: forgejo.forgejo.svc.cluster.local".
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Run the forgejo-runner container as root (uid 0) to allow access to the
Docker socket. This is required for Docker-in-Docker functionality needed
by Forgejo Actions workflows.
Without this, the runner fails with "permission denied" when trying to
access /var/run/docker.sock.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Create sealed secret for forgejo-runner-token to enable Forgejo Actions
runner deployment. This token allows the runner to register with the
Forgejo instance and execute CI/CD workflows.
Fixes basicstack.org workflow failures (DEV-335) caused by missing runner.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Create runner deployment with:
- ServiceAccount and RBAC for runner pod
- ConfigMap for runner configuration
- Deployment using code.forgejo.org/forgejo/runner:4.0.1
- Argo CD application for automated deployment
Note: Runner requires a sealed secret with registration token.
See apps/forgejo-runner/README.md for setup instructions.
Part of DEV-334 CI/CD workflow implementation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add FORGEJO__actions__ENABLED and DEFAULT_ACTIONS_URL environment
variables to enable Actions/CI workflows in Forgejo.
This is required for DEV-334 CI/CD workflow implementation.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move Helm chart configuration from inline values in app-harbor.yaml to
separate files in apps/harbor/ subdirectory, following the same pattern
as forgejo and other apps.
Changes:
- Create apps/harbor/Chart.yaml defining dependency on Harbor Helm chart
- Create apps/harbor/values.yaml with all Helm values configuration
- Update app-harbor.yaml to use git path source instead of direct Helm chart
- Add apps/harbor/README.md documenting OIDC setup procedure
OIDC authentication must be configured via Harbor UI after deployment,
as the Helm chart does not support OIDC configuration at deployment time.
The README provides step-by-step instructions for Pocket ID integration.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Created Argo CD Application for Harbor (app-harbor.yaml)
- Configured Harbor Helm chart with:
- Ingress at harbor.basicstack.de with TLS via cert-manager
- PVCs using hcloud-volumes-encrypted storage class
- OIDC authentication via Pocket ID
- Resource limits for all components
- Created sealed secret with Harbor admin password, database password, and OIDC client secret
- Configured DNS A record for harbor.basicstack.de -> 178.105.17.239
- Created Pocket ID OIDC client for Harbor with callback URL
Harbor will be available at https://harbor.basicstack.de after Argo CD sync.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Set HEADLAMP_CONFIG_BASE_URL to https://headlamp.basicstack.de to ensure
the OIDC callback URL is generated correctly with HTTPS scheme. When
running behind Traefik without explicit base URL, Headlamp may generate
http:// callback URLs instead of https://, causing OIDC flow failures.
Also added traefik.ingress.kubernetes.io/preserve-host annotation to
ensure proper header forwarding.
Fixes: DEV-324
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Headlamp's BASE_URL config expects a path prefix (e.g., "/headlamp") or empty
string, not a full URL. Since Headlamp is deployed at the domain root, BASE_URL
should be empty.
The OIDC redirect URIs in Pocket ID are correctly configured and the user has
been added to the headlamp group, which should resolve the authentication issue.
Related to DEV-324
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Set HEADLAMP_CONFIG_BASE_URL to https://headlamp.basicstack.de (was empty)
- Fix environment variable names to use underscores (OIDC_IDP_ISSUER_URL, etc.)
- Update image version to v0.43.0 to match deployed version
The missing BASE_URL caused the OIDC redirect flow to fail, resulting in the
auth popup closing immediately. Environment variable name corrections ensure
consistency with Headlamp's expected configuration format.
Resolves DEV-324
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add clear documentation for both OIDC and token-based authentication.
Include troubleshooting section for common 403 errors caused by
using tokens from wrong namespace.
Fixes issue where token was generated from kube-system instead of
headlamp namespace, causing 403 errors on metrics API.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Deploy Headlamp v0.27.0 in dedicated namespace
- Configure OIDC authentication with Pocket ID
- Grant cluster-admin access via service account
- Expose via Traefik ingress at headlamp.basicstack.de
- Seal OIDC client credentials
- Enable access for andreas.leinen@basicstack.de and admin@basicstack.de
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Start new 7-day verification window: 2026-07-13 21:07 UTC → 2026-07-20 21:07 UTC
- CEO confirmed Web-UI SMTP and IMAP working
- Restarting verification per Option A (clean 7-day window)
- Baseline status: all services verified working
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Verified all infrastructure working correctly (SMTP/IMAP/LoadBalancer)
- Identified user SMTP issue as client configuration (port 143 vs 587)
- Documented proposed verification window restart (2026-07-13 to 2026-07-20)
- Original window had 2+ days with broken LoadBalancer (10.42.1.1 vs 178.105.17.239)
Related: DEV-300
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Investigation revealed that k3s ServiceLB uses hostPort bindings on nodes'
public IPs, not the misleading EXTERNAL-IP VIP (10.42.1.1) shown in kubectl.
External connectivity was working correctly the entire time. The confusion
was caused by misinterpreting the flannel pod network IP in the EXTERNAL-IP
field as meaning external traffic couldn't reach the service.
Verified all SMTP/IMAP ports responding correctly with proper Stalwart banners.
Related: DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: ArgoCD continuously reverts the LoadBalancer services to use
externalTrafficPolicy: Cluster (the k8s default), causing k3s to assign
internal flannel VXLAN IPs (10.42.1.x) instead of the node public IPs.
With externalTrafficPolicy: Cluster, traffic can be routed to any node,
and k3s's service controller assigns the flannel overlay IPs. This breaks
external connectivity because those IPs are not routable from outside.
With externalTrafficPolicy: Local, traffic is only routed to pods on the
same node, and k3s assigns the node's actual public IP to the LoadBalancer.
This was the missing piece from the reliability hardening in commit b0f2acf.
Without this in git, any manual kubectl patch is reverted by ArgoCD sync.
Evidence: stalwart-smtp and stalwart-imap both showing LoadBalancer IPs:
10.42.1.1, 10.42.1.2, 10.42.1.3, 10.42.1.5 (internal flannel IPs)
Related: DEV-230, DEV-231, DEV-233, DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: k3s ServiceLB assigns internal VIPs (10.42.1.x) that are not
publicly routable. External traffic to mail.basicstack.de cannot reach the
SMTP/IMAP services.
Investigation shows:
- Web-UI works (goes through Traefik IngressRoute)
- SMTP/IMAP ports have correct firewall rules
- iptables DNAT rules exist but don't help external traffic
- Internal connectivity works correctly
- LoadBalancer 'external IPs' are actually pod network IPs
Three solution options documented:
A. NodePort + iptables REDIRECT (recommended)
B. Deploy MetalLB for true LoadBalancer IPs
C. Hetzner Cloud LoadBalancers (not recommended, cost)
Awaiting approval on approach before implementing fix.
Related: DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This document establishes operational rules and diagnostic playbooks for
Stalwart stability monitoring and incident response.
Key sections:
- Critical operational rules (repository as single point of truth)
- Requirement to read git history before any investigation
- Current 7-day stability check procedures (DEV-235)
- Known issues and their fixes (k3s networking, commit b0f2acf)
- Diagnostic playbook for SMTP/IMAP/Web-UI issues
- Issue documentation template
- Evidence requirements for verification completion
This ensures all future investigations follow documented procedures and
avoid repeated debugging cycles.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Configure Argo CD to automatically sync bookstack from the repository,
ensuring the repository remains the single source of truth.
Changes:
- Added automated sync policy with prune and selfHeal
- prune: removes resources no longer in repo
- selfHeal: reverts manual cluster changes back to repo state
This resolves the manual sync requirement and ensures continuous
synchronization without data loss.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The mysql-data-encrypted PVC exists in the cluster and is working correctly,
but attempting to manage it through Argo CD causes sync failures because the
PVC is already bound and its spec is immutable. Leaving it unmanaged while
documenting its existence.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The MySQL deployment was referencing a PVC that wasn't defined in the
manifests. This PVC exists in the cluster but wasn't tracked by Argo CD.
Adding the definition ensures Argo CD can properly manage all resources
and prevents sync issues.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the sealedsecrets.bitnami.com/managed: "true" label to all three
argocd sealed secrets (argocd-oidc-secret, repo-basicstack-org,
repo-stack-basicstack-de) to fix Health Degraded status in Argo CD.
This follows the same fix pattern applied in DEV-291, DEV-290, and DEV-289.
Closes DEV-293
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This tells Argo CD to ignore the unsealed secret created by the sealed-secrets controller, preventing the health status from being degraded.
This is the same fix applied in DEV-289 and DEV-290 for Forgejo and Argo.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Documents the complete migration process, safety measures, and verification steps
for the Pocket ID Argo CD migration.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Migrated Pocket ID deployment from manual kubectl to GitOps. All existing resources
will be adopted by Argo CD without recreation, preserving data and configuration.
Key safety measures:
- PVC has Delete=false annotation to prevent data loss
- Existing sealed secret reused for ENCRYPTION_KEY
- TLS certificate managed by cert-manager (not in Git)
- All OIDC clients and user data preserved in existing PVC
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds sealedsecrets.bitnami.com/managed annotation to ensure proper
lifecycle management by the sealed-secrets controller. This is a
non-destructive fix that improves secret synchronization without
touching encrypted data or affecting access.
Related: DEV-290, DEV-289
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Migrate all Forgejo deployment resources from cluster to Git repository:
- Forgejo Deployment with PostgreSQL backend
- Forgejo Service and Ingress with TLS
- Forgejo PersistentVolumeClaim (5Gi)
- PostgreSQL StatefulSet and Service
- Backup CronJob (daily at 03:00 UTC, 14-day retention)
All existing PVCs (forgejo-data, postgres-data-forgejo-postgres-0) and
Pocket-ID SSO configuration are preserved. No data loss expected.
The Argo CD Application (app-forgejo.yaml) is already configured to
sync from apps/forgejo/ path in this repository.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This adds the sealedsecrets.bitnami.com/managed: "true" label to the
mysql sealed secret to fix Health Degraded status in Argo CD.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The deployment was using client ID "bookstack" which does not exist in
Pocket ID. Updated to use the actual client ID from Pocket ID.
This fixes the "Record not found" error when attempting to login via
Pocket ID.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Fix Ingress rule host from bookstack.basicstack.de to books.basicstack.de
- Update OIDC issuer from pocket-id.basicstack.de to auth.basicstack.de
- Enable AUTH_AUTO_INITIATE for seamless SSO
- Update OIDC_NAME from "Pocket-ID" to "Pocket ID"
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Updated APP_URL environment variable and Ingress configuration to use the correct hostname books.basicstack.de.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Change MySQL from StatefulSet to Deployment to match existing setup
- Use existing PVC mysql-data-encrypted (contains live data) instead of creating new mysql-data
- Add PersistentVolumeClaim for bookstack-config to persist APP_KEY and configuration
- Remove duplicate MySQL resource definitions
- Align with current working deployment while making Forgejo the source of truth
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added comprehensive Kubernetes deployment for Bookstack including:
- MySQL StatefulSet with PersistentVolumeClaim (10Gi encrypted storage)
- MySQL Service (ClusterIP)
- Bookstack Deployment with OIDC/Pocket-ID integration
- Bookstack Service (ClusterIP)
- Ingress with TLS certificate (bookstack.basicstack.de)
The deployment uses existing sealed secrets for MySQL credentials and
Bookstack OIDC client secret. Bookstack is configured with proper
health checks and resource limits.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
