Investigation revealed that k3s ServiceLB uses hostPort bindings on nodes'
public IPs, not the misleading EXTERNAL-IP VIP (10.42.1.1) shown in kubectl.
External connectivity was working correctly the entire time. The confusion
was caused by misinterpreting the flannel pod network IP in the EXTERNAL-IP
field as meaning external traffic couldn't reach the service.
Verified all SMTP/IMAP ports responding correctly with proper Stalwart banners.
Related: DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: ArgoCD continuously reverts the LoadBalancer services to use
externalTrafficPolicy: Cluster (the k8s default), causing k3s to assign
internal flannel VXLAN IPs (10.42.1.x) instead of the node public IPs.
With externalTrafficPolicy: Cluster, traffic can be routed to any node,
and k3s's service controller assigns the flannel overlay IPs. This breaks
external connectivity because those IPs are not routable from outside.
With externalTrafficPolicy: Local, traffic is only routed to pods on the
same node, and k3s assigns the node's actual public IP to the LoadBalancer.
This was the missing piece from the reliability hardening in commit b0f2acf.
Without this in git, any manual kubectl patch is reverted by ArgoCD sync.
Evidence: stalwart-smtp and stalwart-imap both showing LoadBalancer IPs:
10.42.1.1, 10.42.1.2, 10.42.1.3, 10.42.1.5 (internal flannel IPs)
Related: DEV-230, DEV-231, DEV-233, DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: k3s ServiceLB assigns internal VIPs (10.42.1.x) that are not
publicly routable. External traffic to mail.basicstack.de cannot reach the
SMTP/IMAP services.
Investigation shows:
- Web-UI works (goes through Traefik IngressRoute)
- SMTP/IMAP ports have correct firewall rules
- iptables DNAT rules exist but don't help external traffic
- Internal connectivity works correctly
- LoadBalancer 'external IPs' are actually pod network IPs
Three solution options documented:
A. NodePort + iptables REDIRECT (recommended)
B. Deploy MetalLB for true LoadBalancer IPs
C. Hetzner Cloud LoadBalancers (not recommended, cost)
Awaiting approval on approach before implementing fix.
Related: DEV-235
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This document establishes operational rules and diagnostic playbooks for
Stalwart stability monitoring and incident response.
Key sections:
- Critical operational rules (repository as single point of truth)
- Requirement to read git history before any investigation
- Current 7-day stability check procedures (DEV-235)
- Known issues and their fixes (k3s networking, commit b0f2acf)
- Diagnostic playbook for SMTP/IMAP/Web-UI issues
- Issue documentation template
- Evidence requirements for verification completion
This ensures all future investigations follow documented procedures and
avoid repeated debugging cycles.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Configure Argo CD to automatically sync bookstack from the repository,
ensuring the repository remains the single source of truth.
Changes:
- Added automated sync policy with prune and selfHeal
- prune: removes resources no longer in repo
- selfHeal: reverts manual cluster changes back to repo state
This resolves the manual sync requirement and ensures continuous
synchronization without data loss.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The mysql-data-encrypted PVC exists in the cluster and is working correctly,
but attempting to manage it through Argo CD causes sync failures because the
PVC is already bound and its spec is immutable. Leaving it unmanaged while
documenting its existence.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The MySQL deployment was referencing a PVC that wasn't defined in the
manifests. This PVC exists in the cluster but wasn't tracked by Argo CD.
Adding the definition ensures Argo CD can properly manage all resources
and prevents sync issues.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds the sealedsecrets.bitnami.com/managed: "true" label to all three
argocd sealed secrets (argocd-oidc-secret, repo-basicstack-org,
repo-stack-basicstack-de) to fix Health Degraded status in Argo CD.
This follows the same fix pattern applied in DEV-291, DEV-290, and DEV-289.
Closes DEV-293
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This tells Argo CD to ignore the unsealed secret created by the sealed-secrets controller, preventing the health status from being degraded.
This is the same fix applied in DEV-289 and DEV-290 for Forgejo and Argo.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Documents the complete migration process, safety measures, and verification steps
for the Pocket ID Argo CD migration.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Migrated Pocket ID deployment from manual kubectl to GitOps. All existing resources
will be adopted by Argo CD without recreation, preserving data and configuration.
Key safety measures:
- PVC has Delete=false annotation to prevent data loss
- Existing sealed secret reused for ENCRYPTION_KEY
- TLS certificate managed by cert-manager (not in Git)
- All OIDC clients and user data preserved in existing PVC
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Adds sealedsecrets.bitnami.com/managed annotation to ensure proper
lifecycle management by the sealed-secrets controller. This is a
non-destructive fix that improves secret synchronization without
touching encrypted data or affecting access.
Related: DEV-290, DEV-289
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Migrate all Forgejo deployment resources from cluster to Git repository:
- Forgejo Deployment with PostgreSQL backend
- Forgejo Service and Ingress with TLS
- Forgejo PersistentVolumeClaim (5Gi)
- PostgreSQL StatefulSet and Service
- Backup CronJob (daily at 03:00 UTC, 14-day retention)
All existing PVCs (forgejo-data, postgres-data-forgejo-postgres-0) and
Pocket-ID SSO configuration are preserved. No data loss expected.
The Argo CD Application (app-forgejo.yaml) is already configured to
sync from apps/forgejo/ path in this repository.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This adds the sealedsecrets.bitnami.com/managed: "true" label to the
mysql sealed secret to fix Health Degraded status in Argo CD.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The deployment was using client ID "bookstack" which does not exist in
Pocket ID. Updated to use the actual client ID from Pocket ID.
This fixes the "Record not found" error when attempting to login via
Pocket ID.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Fix Ingress rule host from bookstack.basicstack.de to books.basicstack.de
- Update OIDC issuer from pocket-id.basicstack.de to auth.basicstack.de
- Enable AUTH_AUTO_INITIATE for seamless SSO
- Update OIDC_NAME from "Pocket-ID" to "Pocket ID"
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Updated APP_URL environment variable and Ingress configuration to use the correct hostname books.basicstack.de.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Change MySQL from StatefulSet to Deployment to match existing setup
- Use existing PVC mysql-data-encrypted (contains live data) instead of creating new mysql-data
- Add PersistentVolumeClaim for bookstack-config to persist APP_KEY and configuration
- Remove duplicate MySQL resource definitions
- Align with current working deployment while making Forgejo the source of truth
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Added comprehensive Kubernetes deployment for Bookstack including:
- MySQL StatefulSet with PersistentVolumeClaim (10Gi encrypted storage)
- MySQL Service (ClusterIP)
- Bookstack Deployment with OIDC/Pocket-ID integration
- Bookstack Service (ClusterIP)
- Ingress with TLS certificate (bookstack.basicstack.de)
The deployment uses existing sealed secrets for MySQL credentials and
Bookstack OIDC client secret. Bookstack is configured with proper
health checks and resource limits.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The old recursive directory sync has been replaced with individual
Application manifests. The Application resource was deleted from the
cluster and this file is no longer needed.
Related: DEV-287
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Migrated from single recursive directory sync to individual Application
manifests for better control and visibility.
Changes:
- Created app-*.yaml for each application (10 apps)
- Disabled auto-sync on all apps for safe migration
- Disabled old app-stack-basicstack-de.yaml complex sync
- Updated app-argocd.yaml to disable auto-sync during migration
- Added ARGOCD-MIGRATION.md with migration plan and next steps
- Updated apps/README.md to document new structure
Each app now needs:
1. Manual sync verification
2. Health check
3. Auto-sync enablement (tracked in separate tasks)
Related: DEV-287
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Re-sealed the three Argo CD secrets with namespace: argocd instead of
namespace: paperclip to match the kustomization.yaml deployment target.
The sealed-secrets controller was failing to unseal these because they
were encrypted for the 'paperclip' namespace but Kustomize was deploying
them to the 'argocd' namespace. Sealed secrets use strict scope by
default, so namespace/name must match exactly.
Fixed files:
- apps/argocd/argocd-oidc-secret-sealed.yaml
- apps/argocd/repo-basicstack-org-secret-sealed.yaml
- apps/argocd/repo-stack-basicstack-de-secret-sealed.yaml
Root cause: DEV-284 investigation revealed controller error logs showing
"no key could decrypt secret" for all three Argo CD sealed secrets.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
When Argo CD manages its own install manifest, it overwrites the live
argocd-ssh-known-hosts-cm with the upstream defaults (GitHub/GitLab/etc),
losing the Forgejo internal service key. Add the forgejo.forgejo.svc.cluster.local
host key so self-management doesn't break repo access after each sync.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The Argo CD CRDs exceed the 262144-byte annotation limit that client-side
apply imposes via last-applied-configuration. Server-side apply bypasses
this limit and is required for Argo CD to manage its own large CRDs.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The argocd-cm document had two separate data: sections — one at the top
with resource customizations and one at the bottom with url/oidc.config.
This is invalid YAML and caused kustomize build to fail with
"mapping key data already defined". Merged url and oidc.config into the
single data: block to fix the argocd Application sync error.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Removing the exclusion caused the root app to apply argocd raw manifests
directly (bypassing kustomize), breaking namespace resolution. The correct
self-management design is: app-argocd.yaml Application is picked up by the
root app, which then delegates to the argocd Application managing apps/argocd/
via kustomize. The exclusion of argocd/** prevents double-applying resources.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Remove the 'argocd/**' exclusion from the root stack Application so that
Argo CD manages its own configuration via the app-of-apps pattern.
The apps/argocd/ directory already contains all manifests (install, ingress,
sealed secrets) and app-argocd.yaml defines the dedicated argocd Application.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move metadata.name and app.kubernetes.io/part-of label from data section
to proper metadata section. This was causing kustomize build failures.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Change from multiline exclude with **/ pattern to simple single-line
'argocd/**' pattern relative to apps/ path.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Move all Application manifests (app-*.yaml) out of apps/argocd/ to apps/
to avoid chicken-and-egg issue where Applications couldn't update themselves.
Architecture:
- apps/app-stack-basicstack-de.yaml: manages apps/** excluding argocd/**
- apps/app-argocd.yaml: manages apps/argocd/** via kustomize
- apps/app-basicstack-org.yaml: manages basicstack.org repo
This enables full self-management: all Applications can sync their own
configurations from git.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Fully exclude argocd directory from stack Application to prevent
namespace errors. The argocd Application (apps/app-argocd.yaml) handles
all argocd resources via kustomize.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Add test annotation to argocd-cm ConfigMap to verify that Argo CD
automatically syncs changes to its own configuration.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This allows the stack Application to manage Argo CD's own configuration
via GitOps, implementing the app-of-apps pattern.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Creates the Argo CD Application that manages all cluster infrastructure
applications from the stack.basicstack.de repository. Configuration:
- Points to cluster-internal Forgejo (forgejo.forgejo.svc.cluster.local)
- Excludes argocd directory to prevent self-management
- Enables selfHeal for automatic sync of changes
- Disables prune for manual deletion approval
- Deployed to the cluster and verified working
All existing services (Stalwart, Directus, Forgejo, BookStack, Pocket ID,
OpenCloud, Passbolt) remain operational. Some pods restarted due to applying
pending git changes (expected GitOps behavior).
Note: Warning about duplicate opencloud namespace definition in
openldap-deployment.yaml and opencloud-deployment.yaml - does not affect
functionality, can be addressed in follow-up cleanup.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Downloaded upstream Argo CD install.yaml from stable branch
- Customized for basicstack.de cluster:
- Disabled dex deployment (using Pocket ID OIDC instead)
- Added --insecure flag to argocd-server (TLS at Traefik ingress)
- Configured OIDC integration with auth.basicstack.de
- Added RBAC policy mapping argo_admins group to admin role
- Mounted OIDC client credentials from argocd-oidc-secret
- Created Traefik ingress for argo.basicstack.de with cert-manager TLS
Sealed secrets for OIDC and repository credentials already in place.
Ready for deployment per DEV-249 plan.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Created three SealedSecrets for Argo CD integration:
- argocd-oidc-secret: Pocket ID OIDC client credentials
- repo-stack-basicstack-de: SSH deploy key for stack repository
- repo-basicstack-org: SSH deploy key for basicstack.org repository
Repository secrets include argocd.argoproj.io/secret-type label
for automatic Argo CD discovery.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Use existing basicstack-backup bucket with opencloud/ prefix
- Configure endpoint: https://hel1.your-objectstorage.com
- Add pod affinity to run on same node as OpenCloud (RWO volume requirement)
- Schedule: daily at 2:00 AM UTC
- Retention: last 7 days
- First backup verified successful: 110.611 MiB transferred
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Document all four OIDC clients for OpenCloud:
- Web application (UUID-based client ID)
- Desktop client (OpenCloudDesktop)
- Android mobile (OpenCloudAndroid)
- iOS mobile (OpenCloudIOS)
Includes configuration details, security notes, and troubleshooting.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Implement scale-down/backup/scale-up pattern to work around PVC access mode limitation.
Changes:
- Add RBAC (ServiceAccount, Role, RoleBinding) with statefulsets/scale and pods permissions
- Switch to alpine:3.19 base image with kubectl and restic
- Scale down StatefulSet to 0 replicas before backup
- Run restic backup while pod is stopped
- Scale back up to 1 replica with error handling
- Add cleanup trap to ensure scale-up even on failure
- Set 10-minute timeout and backoff limit
Tested successfully: backup completes in ~32 seconds with minimal downtime.
Resolves DEV-236.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.
Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures
Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service
Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.
Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231
Co-Authored-By: Paperclip <noreply@paperclip.ing>
OpenCloud's IDM service provides an internal LDAP server on localhost:9125.
Updated configuration to connect to IDM's LDAP instead of external OpenLDAP:
- Base DN: o=libregraph-idm (IDM's base DN)
- Graph binds as uid=libregraph,ou=sysusers,o=libregraph-idm
- Users/Groups bind as uid=reva,ou=sysusers,o=libregraph-idm
- Uses IDM service user passwords from secrets
Co-Authored-By: Paperclip <noreply@paperclip.ing>
