Document all four OIDC clients for OpenCloud:
- Web application (UUID-based client ID)
- Desktop client (OpenCloudDesktop)
- Android mobile (OpenCloudAndroid)
- iOS mobile (OpenCloudIOS)
Includes configuration details, security notes, and troubleshooting.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Implement scale-down/backup/scale-up pattern to work around PVC access mode limitation.
Changes:
- Add RBAC (ServiceAccount, Role, RoleBinding) with statefulsets/scale and pods permissions
- Switch to alpine:3.19 base image with kubectl and restic
- Scale down StatefulSet to 0 replicas before backup
- Run restic backup while pod is stopped
- Scale back up to 1 replica with error handling
- Add cleanup trap to ensure scale-up even on failure
- Set 10-minute timeout and backoff limit
Tested successfully: backup completes in ~32 seconds with minimal downtime.
Resolves DEV-236.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.
Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures
Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service
Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.
Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Created comprehensive network documentation for BasicStack k3s cluster:
- NETWORK_ARCHITECTURE.md: Complete network architecture with diagrams,
node configuration, CNI (Flannel) details, ingress/LoadBalancer setup,
DNS configuration, TLS certificates, network policies, traffic flows,
and troubleshooting procedures
- DNS_REQUIREMENTS.md: Complete DNS record requirements for all services
including A records, MX records, SPF, DKIM, DMARC, and PTR records
- NETWORK_VERIFICATION.md: Verification report documenting current state
of all network components with findings and recommendations
Updated infrastructure README with links to new network documentation.
Key findings:
- All worker nodes correctly configured with --node-ip set to private IPs
- Flannel VXLAN properly configured with public IP annotations
- Traefik ingress controller operational
- 16/17 TLS certificates valid (registry-tls needs investigation)
- 3 LoadBalancer services properly configured
- Network policies securing database services
Addresses DEV-225: Verify and document k3s cluster network configuration
Related: DEV-224 (node-ip configuration), DEV-223 (DNS issues)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Add CLUSTER_ACCESS.md with comprehensive cluster access guide
- Fix Service CIDR in K3S_OPERATIONS.md (10.43.0.0/16, not 10.96.0.0/12)
- Document API server instability fix (cluster-cidr configuration)
- Add troubleshooting section for CIDR mismatch issues
- Update change history with cluster update details
Co-Authored-By: Paperclip <noreply@paperclip.ing>
OpenCloud's IDM service provides an internal LDAP server on localhost:9125.
Updated configuration to connect to IDM's LDAP instead of external OpenLDAP:
- Base DN: o=libregraph-idm (IDM's base DN)
- Graph binds as uid=libregraph,ou=sysusers,o=libregraph-idm
- Users/Groups bind as uid=reva,ou=sysusers,o=libregraph-idm
- Uses IDM service user passwords from secrets
Co-Authored-By: Paperclip <noreply@paperclip.ing>
When using OpenCloud's built-in IDM service, the graph, users, and groups
services should not use external LDAP. Commented out all LDAP-specific
environment variables that were causing startup failures.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Enable IDM service by removing it from OC_EXCLUDE_RUN_SERVICES
- Remove external LDAP configuration from configmap (graph, users, groups)
- Built-in IDM will handle user/group storage internally
- OIDC auto-provisioning via Pocket ID remains unchanged
This allows OpenCloud to save new users using its internal IDM service.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The postStart lifecycle hook caused osixia/openldap to crash on startup:
its init script does chown -R on /container/service/slapd/assets/, and
the ConfigMap subPath mount there is read-only, killing the container.
Remove the postStart hook and the schema volume mount from the OpenLDAP
deployment. Add a standalone Kubernetes Job (opencloud-ldap-schema-job.yaml)
that connects via network LDAP as cn=admin,cn=config and loads the schema
after OpenLDAP is confirmed ready. The Job is idempotent (skips if the
schema already exists) and retries up to 10 times on failure.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
User creation failed with "openCloudUUID: attribute type undefined" because
OpenLDAP was missing the OpenCloud schema (OIDs under 1.3.6.1.4.1.63016).
Changes:
- Add opencloud-ldap-schema.yaml ConfigMap with the official OpenCloud LDAP
schema defining openCloudUUID, openCloudUser, openCloudExternalIdentity,
openCloudUserEnabled, openCloudUserType, openCloudLastSignInTimestamp
- Mount the ConfigMap into the OpenLDAP pod
- Add lifecycle postStart hook to load schema via ldapadd -Y EXTERNAL -H ldapi:///
(idempotent: skips if already loaded)
- Re-exclude IDM in OpenCloud deployment (external LDAP handles user storage)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Graph service was routing auto-provisioning to external OpenLDAP which lacks
the OpenCloud schema (openCloudUUID: attribute type undefined). The explicit
graph.identity.ldap, users.drivers.ldap, and groups.drivers.ldap sections in
opencloud.yaml overrode the default IDM LDAP, causing user creation to fail.
Remove all external LDAP sections so Graph/Users/Groups default to IDM's
internal LDAP, which has the full OpenCloud schema. IDM is already running.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The /etc/opencloud mount is read-only (ConfigMap), so opencloud init
cannot write the generated IDM password config. IDM fails to start with
"password of service user IDM has not been set properly".
Add the idm.service_user_passwords section to opencloud.yaml, referencing
the OC_IDM_* env vars that are already populated from the sealed secret.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
After Pocket ID OIDC flow completed, users got "Nicht angemeldet" because:
1. PROXY_ROLE_ASSIGNMENT_DRIVER=oidc required an 'opencloud_role' OIDC claim
that Pocket ID wasn't sending → users got no role → login rejected
2. IDM was excluded, removing the internal user store that auto-provisioning
needs to create user accounts when they first log in
Fixes:
- Switch to PROXY_ROLE_ASSIGNMENT_DRIVER=default so all OIDC-authenticated
users automatically receive the standard user role
- Re-enable IDM service so auto-provisioned accounts have a working user store
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Browser was blocking fetch of https://auth.basicstack.de/.well-known/openid-configuration
due to missing connect-src directive in Content-Security-Policy.
Adds csp.yaml to the ConfigMap (mounted at /etc/opencloud/csp.yaml) with
extended connect-src that includes auth.basicstack.de and WebSocket origins.
Sets PROXY_CSP_CONFIG_FILE_LOCATION env var so the proxy service picks it up.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Removed auth-basic configuration and environment variables:
- Removed auth-basic section from ConfigMap
- Removed AUTH_BASIC_LOG_LEVEL, OC_AUTH_BASIC_LDAP_BIND_PASSWORD,
and AUTH_BASIC_AUTH_PROVIDERS_LDAP_BIND_PASSWORD env vars
- Added auth-basic to OC_EXCLUDE_RUN_SERVICES
Result: OpenCloud pod running healthy (1/1 Ready)
- All services listening and operational
- https://opencloud.basicstack.de/ responding with HTTP 200
- No auth-basic bind_password errors
Ready for OIDC authentication testing with Pocket ID.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Changed from alpine+envsubst to busybox with simple shell-based
variable expansion using sed. This avoids permission issues with
apk and works with non-root security context.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Mounting opencloud-config ConfigMap to /etc/opencloud caused pod crashes
because the YAML contains placeholders like ${ENV_VAR} that aren't expanded.
Issue: OpenCloud expects actual values in config files, but ConfigMaps
don't perform environment variable substitution.
Solution: Use environment variables only (highest precedence per docs).
The current deployment with ENV vars works correctly - no file config needed.
Alternative: If file config is required, use an init container to:
1. Read the ConfigMap template
2. Substitute environment variables
3. Write the expanded config to /etc/opencloud
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Added volume mount for opencloud-config ConfigMap at /etc/opencloud
so OpenCloud services can read the opencloud.yaml configuration file.
This follows OpenCloud's standard configuration pattern:
- opencloud.yaml provides global defaults
- Environment variables override file configuration
- ConfigMap mounted at /etc/opencloud (container default)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
OIDC redirect still not working after multiple configuration attempts.
Enabled basic auth so user can test LDAP authentication.
Added user: andreas.leinen@basicstack.de
Password: OpenCloud2024!
TODO: Investigate why OIDC redirect isn't happening despite correct
configuration in deployment and configmap.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Problem: Frontend still not redirecting to OIDC after all config changes.
Created detailed troubleshooting doc covering:
- All 6 commits of configuration changes applied
- What's verified working (OIDC config, IDM, no local users)
- Current behavior vs expected behavior
- Possible root causes
- Next investigation steps
Suggests further investigation needed:
- Check OpenCloud web frontend source code
- Test OIDC flow with curl/API
- Consult OpenCloud community/support
- Browser dev tools debugging
All backend config appears correct. Issue likely in frontend SPA logic
or undocumented configuration requirement for OIDC-only mode.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Problem: Auth-service requires OC_JWT_SECRET to mint/validate JWT tokens.
Root cause: Only OC_TOKEN_MANAGER_JWT_SECRET was set, not OC_JWT_SECRET.
According to OpenCloud docs, auth-service needs:
- OC_JWT_SECRET or AUTH_SERVICE_JWT_SECRET
Changes:
- Added OC_JWT_SECRET environment variable
- Points to same jwt-secret as OC_TOKEN_MANAGER_JWT_SECRET
Why: Auth-service uses OC_JWT_SECRET to validate tokens from OIDC flow.
Without it, authentication fails silently.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Problem: Removing init entirely broke OpenCloud - needs init for config setup.
Solution: Keep 'opencloud init' but remove '--admin-password' flag.
Changes:
- Restored 'opencloud init' command (needed for jwt_secret and config)
- Removed '--admin-password' flag (prevents local admin user creation)
- IDM_CREATE_DEMO_USERS=false still set (prevents demo users)
Result: Init runs to setup config, but no local users are created.
Only OIDC auto-provisioned users will exist.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Problem: Frontend shows login page because local admin user exists.
Root cause: 'opencloud init --admin-password' creates local admin on every start.
Changes:
- Removed 'opencloud init' command from container startup
- Changed to direct 'opencloud server' execution
- Added IDM_CREATE_DEMO_USERS=false to prevent demo user creation
Why: With external OIDC, we don't need local users. The frontend detects
local users and shows a password login page. By removing local user creation,
the frontend will only offer OIDC authentication.
Auto-provisioning will create users in IDM on first OIDC login.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Problem: Login still showing local page instead of OIDC redirect.
Root cause: Excluded both IDP and IDM, but IDM is needed for user storage.
Architecture Fix:
- IDP (Identity Provider) = EXCLUDED (auth handled by Pocket ID OIDC)
- IDM (Identity Management) = KEPT (provides LDAP storage for users/groups)
- Proxy auto-provisioning = creates users in IDM LDAP on first OIDC login
Changes:
- deployment: OC_EXCLUDE_RUN_SERVICES changed from "search,idp,idm" to "search,idp"
- configmap: Re-enabled IDM service configuration
- configmap: Restored graph.identity.ldap (points to internal IDM)
- configmap: Restored users/groups LDAP drivers (connect to internal IDM)
Flow: User → OIDC (Pocket ID) → Proxy auto-provision → IDM LDAP → User created
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Problem: Users still routed to /login page instead of OIDC redirect.
Root cause: auto_provision_accounts was disabled in proxy OIDC config.
Changes:
- proxy.oidc.auto_provision_accounts: true (enable auto-provisioning)
- proxy.oidc.user_oidc_claim: email (user identification claim)
- proxy.oidc.role_assignment: Added OIDC role mapper for groups
- proxy.enable_basic_auth: false (explicitly disable basic auth)
Result: Users will be auto-created on first OIDC login with group-based roles.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Problem: OpenCloud was configured for internal authentication but login wasn't working.
Changes:
- Exclude internal IDP/IDM services (OC_EXCLUDE_RUN_SERVICES=search,idp,idm)
- Configure external OIDC with Pocket ID (auth.basicstack.de)
- Disable basic authentication (PROXY_ENABLE_BASIC_AUTH=false)
- Configure web service OIDC client settings
- Remove internal demo user creation (IDM_CREATE_DEMO_USERS)
Result: Users will authenticate via Pocket ID only, no internal auth methods.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Changed from random password to fixed password 'OpenCloud2024!'
to provide consistent login credentials.
Admin credentials:
- Username: admin
- Password: OpenCloud2024!
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Removed read-only ConfigMap mount that prevented initialization
- Added 'opencloud init' to startup command to auto-generate config
- Config is now generated in /etc/opencloud at container startup
- Admin user is automatically created with random password
- Service users are created by init process
- Fixes HTTP 500 error on login
This allows OpenCloud to properly initialize its internal IDM/IDP
services with the necessary service users for inter-service auth.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Removed all Pocket ID OIDC environment variables. OpenCloud now uses
its internal IDP for authentication instead of external OIDC provider.
The frontend config.json now correctly points to the internal IDP:
- Authority: https://opencloud.basicstack.de (was: auth.basicstack.de)
- Client ID: web (internal IDP client)
This fixes the login redirect issue where users were being sent to
the login page but couldn't see the login form.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Enable internal IDP service (remove from OC_EXCLUDE_RUN_SERVICES)
- Enable demo users (IDM_CREATE_DEMO_USERS=true)
- Enable basic auth (PROXY_ENABLE_BASIC_AUTH=true)
- Remove opencloud init command (not needed with ConfigMap)
- Remove conflicting admin user overrides
OpenCloud now uses internal user management instead of external OIDC.
Demo users will be created on first startup (admin, einstein, marie).
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Created automated daily backup system using rclone and Kubernetes CronJob.
Features:
- Daily backups at 2 AM UTC
- 7-day retention policy
- Backs up data directory and configuration
- Uses Hetzner S3-compatible Object Storage
- Read-only access to OpenCloud volumes
Files:
- backup-cronjob.yaml: CronJob for automated backups
- BACKUP.md: Complete setup and restore documentation
Requires:
- Hetzner Object Storage bucket credentials (sealed secret)
- S3 access key/secret to be provided
Once credentials are configured, backups will run automatically.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
