# Pangolin Pangolin is BasicStack's self-hosted reverse proxy and SSO gateway. It sits in front of selected cluster services (e.g. `paperclip.basicstack.de`) and forces every request through Pocket-ID before it reaches the backend. Pangolin does not terminate TLS or handle the raw HTTP request path itself — it publishes a Traefik dynamic config, and the `pangolin-kube-controller` translates that config into Traefik CRDs that our existing Traefik ingress controller enforces. Public URL: — App source: ## Architecture ``` Browser ──HTTPS──► Traefik (ns=kube-system) │ │ IngressRoute + Middleware (badger) + TraefikService │ ▲ managed by pangolin-kube-controller (ns=pangolin) │ │ │ │ polls /api/v1/traefik-config every ~15s │ │ ▼ │ Pangolin API (ns=pangolin) ├─ port 3000 external dashboard API (session auth, behind Ingress) ├─ port 3001 internal traefik-config API (Bearer auth, cluster-internal only) └─ port 3002 UI HTTP (behind Ingress) │ ├── PostgreSQL (ns=pangolin, pangolin-postgres) └── OIDC provider: Pocket-ID at https://auth.basicstack.de Auth flow for a protected resource (e.g. paperclip.basicstack.de): Browser ──► Traefik ──► badger middleware ──callback──► Pangolin API (3001) │ ▼ Pocket-ID / session ``` Two long-running workloads live in the `pangolin` namespace: - **Pangolin** — the app itself: dashboard, resource/target CRUD, OIDC handshake, session cookies, and the internal `/api/v1/traefik-config` endpoint that publishes the current desired Traefik state. - **pangolin-kube-controller** — a Kubernetes controller (leader-elected) that polls the Pangolin API, then upserts Traefik `IngressRoute`, `Middleware`, `TraefikService`, etc. via server-side apply into the `pangolin` namespace. The controller only writes CRDs into `pangolin`; the backing Service typically lives in the target app's own namespace. Traefik is started with `--providers.kubernetescrd.allowCrossNamespace=true` so those cross-namespace refs resolve. ## Components (files in this directory) | File | Purpose | | --- | --- | | `namespace.yaml` | Creates the `pangolin` namespace. | | `pangolin-config.yaml` | Pangolin `config.yml` template (ConfigMap). `__DATABASE_URL__` is rendered at startup by the `render-config` init container from the sealed secret. | | `pangolin-deployment.yaml` | Pangolin app Deployment, Service (ports 3000/3001/3002), Ingress, and the `pangolin-data` PVC. | | `pangolin-secrets-sealed.yaml` | Sealed OIDC client id/secret and local admin password. | | `pangolin-postgres-secrets-sealed.yaml` | Sealed Postgres credentials plus the rendered `DATABASE_URL`. | | `postgres.yaml` | Single-replica Postgres 17.5 + `pangolin-postgres-data` PVC. | | `pangolin-controller-config.yaml` | `pangolin-kube-controller` env: `CONFIG_ENDPOINT`, target namespace, log level. | | `pangolin-controller-deployment.yaml` | Controller Deployment (image `fosrl/pangolin-kube-controller:0.1.0-alpha.1`). | | `pangolin-controller-rbac.yaml` | ServiceAccount + ClusterRole/Binding: CRUD on Traefik CRDs and leases. | | `pangolin-controller-service.yaml` | ClusterIP for controller metrics. | | `pangolin-controller-api-key-sealed.yaml` | Sealed `Authorization` header the controller sends to the internal API. | ArgoCD Application: [`apps/app-pangolin.yaml`](../app-pangolin.yaml) — sync policy is `automated: { prune: true, selfHeal: true }`. Any local edit must land in git. ## SSO integration (Pocket-ID) Pangolin authenticates users against Pocket-ID via OIDC. Two independent SSO flows exist and should not be confused: 1. **Logging into the Pangolin dashboard.** Configured in `pangolin-config.yaml` (`oidc.*`) and via env in `pangolin-deployment.yaml`: - Issuer: `https://auth.basicstack.de` - Callback URL: `https://pangolin.basicstack.de/auth/callback` - Scopes: `openid profile email groups` - Client id/secret: `pangolin-secrets` sealed secret keys `oidc-client-id` / `oidc-client-secret`. 2. **Protecting a downstream resource (e.g. paperclip.basicstack.de).** Handled by the badger Traefik middleware that the controller emits. The middleware calls back into Pangolin's internal API (port 3001) to check the session cookie, and if unauthenticated redirects the browser through Pangolin's own OIDC flow. There is nothing to configure per-resource in Kubernetes — the resource's auth policy is set inside the Pangolin UI. The Pocket-ID client is provisioned separately (see [pocket-id docs](https://pocket-id.org/docs/client-examples/pangolin)). Do not reconfigure the OIDC client from the Pangolin side once it is working. ## Ingress protection workflow The end-to-end flow for a protected hostname like `paperclip.basicstack.de`: 1. Operator creates a **Resource** in the Pangolin UI (hostname, backend Service, port, auth policy). 2. Pangolin persists it in Postgres and exposes the new desired state at `http://pangolin.pangolin.svc.cluster.local:3001/api/v1/traefik-config`. 3. `pangolin-kube-controller` polls that endpoint on a ~15s loop. 4. On change, the controller server-side-applies into `ns=pangolin`: - `IngressRoute` for the hostname (with `tls.certResolver: letsencrypt`). - `Middleware/badger` (session check + OIDC redirect). - `TraefikService/1--service` pointing at the backend Service in the app's namespace, port from the DB. 5. Traefik picks the new CRDs up and starts serving the route. 6. First request → redirected to Pocket-ID → session cookie → subsequent requests are proxied to the backend Service. ## Adding a new protected ingress Prereqs the target app must already satisfy: - A Kubernetes `Service` in its own namespace, listening on a known port. - A DNS record for the hostname pointing at the cluster ingress IP. - No competing plain `Ingress` in another namespace for the same hostname. If one exists (e.g. a legacy direct ingress), delete it first — Traefik will otherwise route the pre-existing ingress and Pangolin's IngressRoute never wins. Steps: 1. **Log in** to (SSO via Pocket-ID, or local admin from the `pangolin-secrets` sealed secret). 2. **Create the Resource.** - Type: HTTP/HTTPS. - Domain: the exact FQDN (e.g. `newapp.basicstack.de`). - Base domain: `basicstack.de`. 3. **Target.** - Target type: HTTP. - Backend URL: `http://..svc.cluster.local:` — the port here is written to the `targets.port` column in Pangolin's Postgres and used by the controller when it generates the `TraefikService`. If the Service port later changes, update the target port in the UI (or via SQL: `UPDATE targets SET port= WHERE "targetId"=;`). - Path match: `/`, strip prefix off. 4. **TLS.** Enable TLS, force HTTPS redirect. The Pangolin-generated IngressRoute hardcodes `tls.certResolver: letsencrypt`, which is provisioned on our Traefik install (HTTP-01, persistent `/data` PVC). Do not disable that resolver in the Traefik `HelmChartConfig` without also removing every Pangolin-generated route — Traefik will otherwise fail with `nonexistent certificate resolver`. 5. **Auth.** Enable authentication, method SSO/OIDC. In Community Edition (what we run) there are no shared policies or group rules — add authorised users individually by email. If you need group-based rules, that's an Enterprise feature. 6. **Save** — controller applies CRDs within a poll interval (~15s). 7. **Verify.** ```bash kubectl -n pangolin get ingressroute,middleware,traefikservice \ | grep -i kubectl -n pangolin logs -l app=pangolin-controller --tail=50 # HTTP smoke test curl -sSI https:/// # expect 302 to Pocket-ID ``` ## Troubleshooting - **badger middleware returns HTTP 500.** The middleware is trying to call `apiBaseUrl` and either the URL is not an FQDN or Pangolin isn't reachable from `kube-system`. Confirm `pangolin-config.yaml` sets `server.internal_hostname` and `server.badger_override` to `pangolin.pangolin.svc.cluster.local` — the short name `pangolin` does not resolve from Traefik's namespace. - **Controller logs `conflict with "controller" using traefik.io/v1alpha1: .spec…`.** A pre-existing CRD was created with the `Update` verb and the current controller uses server-side `Apply`. Delete the offending `Middleware/TraefikService/IngressRoute` once — the controller recreates it via SSA and the conflict is gone. - **Traefik logs `service port not found: `.** The port stored in Pangolin's `targets` table doesn't match the Kubernetes Service port. Fix in the UI, or: `UPDATE targets SET port= WHERE "targetId"=;` (identifier case matters, double-quote it). - **Traefik logs `nonexistent certificate resolver: letsencrypt`.** The `letsencrypt` resolver isn't loaded in Traefik. Every Pangolin IngressRoute references it — either define the resolver in Traefik (see `infrastructure/networking/traefik-helmchartconfig.yaml`) or override `traefik.cert_resolver` in `pangolin-config.yaml`. Grep the Traefik startup log for `unable to get ACME account`; that appears when the resolver was skipped entirely. - **Route doesn't resolve cross-namespace.** Traefik must run with `--providers.kubernetescrd.allowCrossNamespace=true`. Pangolin's CRDs live in `pangolin`; backend Services usually don't. - **CRDs never appear after saving a Resource.** ```bash kubectl -n pangolin get pods -l app=pangolin-controller kubectl -n pangolin logs -l app=pangolin-controller --tail=200 kubectl -n pangolin get configmap pangolin-controller-config -o yaml ``` Common causes: controller CrashLoop, wrong `CONFIG_ENDPOINT`, missing auth header sealed secret, RBAC missing after a controller image upgrade. - **Login loop / redirects to Pocket-ID forever.** Clear cookies for both `pangolin.basicstack.de` and `auth.basicstack.de`, confirm the OIDC callback URL matches the Pocket-ID client, and check Pangolin logs for OIDC errors: `kubectl -n pangolin logs -l app=pangolin`. - **Pangolin pod stuck in `Init:0/2`.** The `wait-for-postgres` init container is looping on `pg_isready`. Check `pangolin-postgres` health and the `pangolin-postgres-secrets` connection details. ## Critical notes — DO NOT DELETE Pangolin holds live user/account/config state. Losing a PVC here means re-provisioning every protected resource, every target port, every user grant, and re-doing the OIDC client trust from scratch — plus temporary outages on every downstream site that Pangolin fronts. - **`pangolin-postgres-data` PVC (ns=pangolin, 10 Gi, `hcloud-volumes-encrypted`).** Backs Postgres. Contains resources, targets, users, sessions, and everything the controller reads. Annotated with `argocd.argoproj.io/sync-options: Delete=false` — do not remove that annotation, and do not delete the PVC as a "reset" step. Snapshot the volume before any migration or destructive DB operation. - **`pangolin-data` PVC (ns=pangolin, 10 Gi, `hcloud-volumes-encrypted`).** Runtime files under `/app/config`. Also marked `Delete=false`. Do not delete. - **Sealed secrets** (`pangolin-secrets`, `pangolin-postgres-secrets`, `pangolin-controller-api-key`) — rotating them out of order will lock the controller out of the internal API or take Pangolin off Postgres. Rotate one secret at a time and verify the corresponding pod comes back before proceeding. - **`pangolin-config` ConfigMap.** Editing `server.internal_hostname` / `server.badger_override` back to the default short hostname will break the badger middleware cluster-wide (see Troubleshooting). - **The Traefik `HelmChartConfig`** in `kube-system` (`infrastructure/networking/traefik-helmchartconfig.yaml`) — Pangolin depends on `allowCrossNamespace=true` and on the `letsencrypt` resolver being loaded with a persistent `/data` PVC. Removing either silently breaks every Pangolin-managed route. If access to Pangolin is broken, **restore from a snapshot first; do not "reset" or re-provision** the admin/OIDC layer. That is the standing rule for every stateful service in this cluster. ## Related - History and gotchas: internal memory `pangolin-kube-integration` (from DEV-457). - Configuration walkthrough for the `paperclip.basicstack.de` resource: [`pangolin-paperclip-configuration-guide-community.md`](pangolin-paperclip-configuration-guide-community.md). - Pangolin docs: - pangolin-kube-controller: