stack.basicstack.de/apps/pangolin/README.md

Pangolin

Pangolin is BasicStack's self-hosted reverse proxy and SSO gateway. It sits in front of selected cluster services (e.g. paperclip.basicstack.de) and forces every request through Pocket-ID before it reaches the backend. Pangolin does not terminate TLS or handle the raw HTTP request path itself — it publishes a Traefik dynamic config, and the pangolin-kube-controller translates that config into Traefik CRDs that our existing Traefik ingress controller enforces.

Public URL: https://pangolin.basicstack.de — App source: https://github.com/fosrl/pangolin

Architecture

Browser ──HTTPS──► Traefik (ns=kube-system)
                    │
                    │   IngressRoute + Middleware (badger) + TraefikService
                    │   ▲  managed by pangolin-kube-controller (ns=pangolin)
                    │   │
                    │   │ polls /api/v1/traefik-config every ~15s
                    │   │
                    ▼   │
              Pangolin API (ns=pangolin)
              ├─ port 3000  external dashboard API  (session auth, behind Ingress)
              ├─ port 3001  internal traefik-config API  (Bearer auth, cluster-internal only)
              └─ port 3002  UI HTTP                (behind Ingress)
                    │
                    ├── PostgreSQL (ns=pangolin, pangolin-postgres)
                    └── OIDC provider: Pocket-ID at https://auth.basicstack.de

Auth flow for a protected resource (e.g. paperclip.basicstack.de):
   Browser ──► Traefik ──► badger middleware ──callback──► Pangolin API (3001)
                                                              │
                                                              ▼
                                                        Pocket-ID / session

Two long-running workloads live in the pangolin namespace:

• Pangolin — the app itself: dashboard, resource/target CRUD, OIDC handshake, session cookies, and the internal /api/v1/traefik-config endpoint that publishes the current desired Traefik state.
• pangolin-kube-controller — a Kubernetes controller (leader-elected) that polls the Pangolin API, then upserts Traefik IngressRoute, Middleware, TraefikService, etc. via server-side apply into the pangolin namespace.

The controller only writes CRDs into pangolin; the backing Service typically lives in the target app's own namespace. Traefik is started with --providers.kubernetescrd.allowCrossNamespace=true so those cross-namespace refs resolve.

Components (files in this directory)

File	Purpose
namespace.yaml	Creates the pangolin namespace.
pangolin-config.yaml	Pangolin config.yml template (ConfigMap). __DATABASE_URL__ is rendered at startup by the render-config init container from the sealed secret.
pangolin-deployment.yaml	Pangolin app Deployment, Service (ports 3000/3001/3002), Ingress, and the pangolin-data PVC.
pangolin-secrets-sealed.yaml	Sealed OIDC client id/secret and local admin password.
pangolin-postgres-secrets-sealed.yaml	Sealed Postgres credentials plus the rendered DATABASE_URL.
postgres.yaml	Single-replica Postgres 17.5 + pangolin-postgres-data PVC.
pangolin-controller-config.yaml	pangolin-kube-controller env: CONFIG_ENDPOINT, target namespace, log level.
pangolin-controller-deployment.yaml	Controller Deployment (image fosrl/pangolin-kube-controller:0.1.0-alpha.1).
pangolin-controller-rbac.yaml	ServiceAccount + ClusterRole/Binding: CRUD on Traefik CRDs and leases.
pangolin-controller-service.yaml	ClusterIP for controller metrics.
pangolin-controller-api-key-sealed.yaml	Sealed Authorization header the controller sends to the internal API.

ArgoCD Application: apps/app-pangolin.yaml — sync policy is automated: { prune: true, selfHeal: true }. Any local edit must land in git.

SSO integration (Pocket-ID)

Pangolin authenticates users against Pocket-ID via OIDC. Two independent SSO flows exist and should not be confused:

1. Logging into the Pangolin dashboard. Configured in pangolin-config.yaml (oidc.*) and via env in pangolin-deployment.yaml:
   • Issuer: https://auth.basicstack.de
   • Callback URL: https://pangolin.basicstack.de/auth/callback
   • Scopes: openid profile email groups
   • Client id/secret: pangolin-secrets sealed secret keys oidc-client-id / oidc-client-secret.
2. Protecting a downstream resource (e.g. paperclip.basicstack.de). Handled by the badger Traefik middleware that the controller emits. The middleware calls back into Pangolin's internal API (port 3001) to check the session cookie, and if unauthenticated redirects the browser through Pangolin's own OIDC flow. There is nothing to configure per-resource in Kubernetes — the resource's auth policy is set inside the Pangolin UI.

The Pocket-ID client is provisioned separately (see pocket-id docs). Do not reconfigure the OIDC client from the Pangolin side once it is working.

Ingress protection workflow

The end-to-end flow for a protected hostname like paperclip.basicstack.de:

1. Operator creates a Resource in the Pangolin UI (hostname, backend Service, port, auth policy).
2. Pangolin persists it in Postgres and exposes the new desired state at http://pangolin.pangolin.svc.cluster.local:3001/api/v1/traefik-config.
3. pangolin-kube-controller polls that endpoint on a ~15s loop.
4. On change, the controller server-side-applies into ns=pangolin:
   • IngressRoute for the hostname (with tls.certResolver: letsencrypt).
   • Middleware/badger (session check + OIDC redirect).
   • TraefikService/1-<resource>-service pointing at the backend Service in the app's namespace, port from the DB.
5. Traefik picks the new CRDs up and starts serving the route.
6. First request → redirected to Pocket-ID → session cookie → subsequent requests are proxied to the backend Service.

Adding a new protected ingress

Prereqs the target app must already satisfy:

• A Kubernetes Service in its own namespace, listening on a known port.
• A DNS record for the hostname pointing at the cluster ingress IP.
• No competing plain Ingress in another namespace for the same hostname. If one exists (e.g. a legacy direct ingress), delete it first — Traefik will otherwise route the pre-existing ingress and Pangolin's IngressRoute never wins.

Steps:

1. Log in to https://pangolin.basicstack.de (SSO via Pocket-ID, or local admin from the pangolin-secrets sealed secret).

2. Create the Resource.

   • Type: HTTP/HTTPS.
   • Domain: the exact FQDN (e.g. newapp.basicstack.de).
   • Base domain: basicstack.de.

3. Target.

   • Target type: HTTP.
   • Backend URL: http://<service>.<namespace>.svc.cluster.local:<port> — the port here is written to the targets.port column in Pangolin's Postgres and used by the controller when it generates the TraefikService. If the Service port later changes, update the target port in the UI (or via SQL: UPDATE targets SET port=<N> WHERE "targetId"=<id>;).
   • Path match: /, strip prefix off.

4. TLS. Enable TLS, force HTTPS redirect. The Pangolin-generated IngressRoute hardcodes tls.certResolver: letsencrypt, which is provisioned on our Traefik install (HTTP-01, persistent /data PVC). Do not disable that resolver in the Traefik HelmChartConfig without also removing every Pangolin-generated route — Traefik will otherwise fail with nonexistent certificate resolver.

5. Auth. Enable authentication, method SSO/OIDC. In Community Edition (what we run) there are no shared policies or group rules — add authorised users individually by email. If you need group-based rules, that's an Enterprise feature.

6. Save — controller applies CRDs within a poll interval (~15s).

7. Verify.

   kubectl -n pangolin get ingressroute,middleware,traefikservice \
     | grep -i <resource-name>
   kubectl -n pangolin logs -l app=pangolin-controller --tail=50
   # HTTP smoke test
   curl -sSI https://<host>/                # expect 302 to Pocket-ID
   

Troubleshooting

• badger middleware returns HTTP 500. The middleware is trying to call apiBaseUrl and either the URL is not an FQDN or Pangolin isn't reachable from kube-system. Confirm pangolin-config.yaml sets server.internal_hostname and server.badger_override to pangolin.pangolin.svc.cluster.local — the short name pangolin does not resolve from Traefik's namespace.
• Controller logs conflict with "controller" using traefik.io/v1alpha1: .spec…. A pre-existing CRD was created with the Update verb and the current controller uses server-side Apply. Delete the offending Middleware/TraefikService/IngressRoute once — the controller recreates it via SSA and the conflict is gone.
• Traefik logs service port not found: <N>. The port stored in Pangolin's targets table doesn't match the Kubernetes Service port. Fix in the UI, or: UPDATE targets SET port=<N> WHERE "targetId"=<id>; (identifier case matters, double-quote it).
• Traefik logs nonexistent certificate resolver: letsencrypt. The letsencrypt resolver isn't loaded in Traefik. Every Pangolin IngressRoute references it — either define the resolver in Traefik (see infrastructure/networking/traefik-helmchartconfig.yaml) or override traefik.cert_resolver in pangolin-config.yaml. Grep the Traefik startup log for unable to get ACME account; that appears when the resolver was skipped entirely.
• Route doesn't resolve cross-namespace. Traefik must run with --providers.kubernetescrd.allowCrossNamespace=true. Pangolin's CRDs live in pangolin; backend Services usually don't.
• CRDs never appear after saving a Resource.

  kubectl -n pangolin get pods -l app=pangolin-controller
  kubectl -n pangolin logs  -l app=pangolin-controller --tail=200
  kubectl -n pangolin get configmap pangolin-controller-config -o yaml
  

  Common causes: controller CrashLoop, wrong CONFIG_ENDPOINT, missing auth header sealed secret, RBAC missing after a controller image upgrade.
• Login loop / redirects to Pocket-ID forever. Clear cookies for both pangolin.basicstack.de and auth.basicstack.de, confirm the OIDC callback URL matches the Pocket-ID client, and check Pangolin logs for OIDC errors: kubectl -n pangolin logs -l app=pangolin.
• Pangolin pod stuck in Init:0/2. The wait-for-postgres init container is looping on pg_isready. Check pangolin-postgres health and the pangolin-postgres-secrets connection details.

Critical notes — DO NOT DELETE

Pangolin holds live user/account/config state. Losing a PVC here means re-provisioning every protected resource, every target port, every user grant, and re-doing the OIDC client trust from scratch — plus temporary outages on every downstream site that Pangolin fronts.

• pangolin-postgres-data PVC (ns=pangolin, 10 Gi, hcloud-volumes-encrypted). Backs Postgres. Contains resources, targets, users, sessions, and everything the controller reads. Annotated with argocd.argoproj.io/sync-options: Delete=false — do not remove that annotation, and do not delete the PVC as a "reset" step. Snapshot the volume before any migration or destructive DB operation.
• pangolin-data PVC (ns=pangolin, 10 Gi, hcloud-volumes-encrypted). Runtime files under /app/config. Also marked Delete=false. Do not delete.
• Sealed secrets (pangolin-secrets, pangolin-postgres-secrets, pangolin-controller-api-key) — rotating them out of order will lock the controller out of the internal API or take Pangolin off Postgres. Rotate one secret at a time and verify the corresponding pod comes back before proceeding.
• pangolin-config ConfigMap. Editing server.internal_hostname / server.badger_override back to the default short hostname will break the badger middleware cluster-wide (see Troubleshooting).
• The Traefik HelmChartConfig in kube-system (infrastructure/networking/traefik-helmchartconfig.yaml) — Pangolin depends on allowCrossNamespace=true and on the letsencrypt resolver being loaded with a persistent /data PVC. Removing either silently breaks every Pangolin-managed route.

If access to Pangolin is broken, restore from a snapshot first; do not "reset" or re-provision the admin/OIDC layer. That is the standing rule for every stateful service in this cluster.

Related

• History and gotchas: internal memory pangolin-kube-integration (from DEV-457).
• Configuration walkthrough for the paperclip.basicstack.de resource: pangolin-paperclip-configuration-guide-community.md.
• Pangolin docs: https://docs.pangolin.net/
• pangolin-kube-controller: https://github.com/fosrl/pangolin-kube-controller