diff --git a/apps/pangolin/README.md b/apps/pangolin/README.md new file mode 100644 index 0000000..0d9796c --- /dev/null +++ b/apps/pangolin/README.md @@ -0,0 +1,240 @@ +# Pangolin + +Pangolin is BasicStack's self-hosted reverse proxy and SSO gateway. It sits in front of +selected cluster services (e.g. `paperclip.basicstack.de`) and forces every request through +Pocket-ID before it reaches the backend. Pangolin does not terminate TLS or handle the +raw HTTP request path itself — it publishes a Traefik dynamic config, and the +`pangolin-kube-controller` translates that config into Traefik CRDs that our existing +Traefik ingress controller enforces. + +Public URL: — App source: + +## Architecture + +``` +Browser ──HTTPS──► Traefik (ns=kube-system) + │ + │ IngressRoute + Middleware (badger) + TraefikService + │ ▲ managed by pangolin-kube-controller (ns=pangolin) + │ │ + │ │ polls /api/v1/traefik-config every ~15s + │ │ + ▼ │ + Pangolin API (ns=pangolin) + ├─ port 3000 external dashboard API (session auth, behind Ingress) + ├─ port 3001 internal traefik-config API (Bearer auth, cluster-internal only) + └─ port 3002 UI HTTP (behind Ingress) + │ + ├── PostgreSQL (ns=pangolin, pangolin-postgres) + └── OIDC provider: Pocket-ID at https://auth.basicstack.de + +Auth flow for a protected resource (e.g. paperclip.basicstack.de): + Browser ──► Traefik ──► badger middleware ──callback──► Pangolin API (3001) + │ + ▼ + Pocket-ID / session +``` + +Two long-running workloads live in the `pangolin` namespace: + +- **Pangolin** — the app itself: dashboard, resource/target CRUD, OIDC handshake, + session cookies, and the internal `/api/v1/traefik-config` endpoint that + publishes the current desired Traefik state. +- **pangolin-kube-controller** — a Kubernetes controller (leader-elected) that + polls the Pangolin API, then upserts Traefik `IngressRoute`, `Middleware`, + `TraefikService`, etc. via server-side apply into the `pangolin` namespace. + +The controller only writes CRDs into `pangolin`; the backing Service typically +lives in the target app's own namespace. Traefik is started with +`--providers.kubernetescrd.allowCrossNamespace=true` so those cross-namespace +refs resolve. + +## Components (files in this directory) + +| File | Purpose | +| --- | --- | +| `namespace.yaml` | Creates the `pangolin` namespace. | +| `pangolin-config.yaml` | Pangolin `config.yml` template (ConfigMap). `__DATABASE_URL__` is rendered at startup by the `render-config` init container from the sealed secret. | +| `pangolin-deployment.yaml` | Pangolin app Deployment, Service (ports 3000/3001/3002), Ingress, and the `pangolin-data` PVC. | +| `pangolin-secrets-sealed.yaml` | Sealed OIDC client id/secret and local admin password. | +| `pangolin-postgres-secrets-sealed.yaml` | Sealed Postgres credentials plus the rendered `DATABASE_URL`. | +| `postgres.yaml` | Single-replica Postgres 17.5 + `pangolin-postgres-data` PVC. | +| `pangolin-controller-config.yaml` | `pangolin-kube-controller` env: `CONFIG_ENDPOINT`, target namespace, log level. | +| `pangolin-controller-deployment.yaml` | Controller Deployment (image `fosrl/pangolin-kube-controller:0.1.0-alpha.1`). | +| `pangolin-controller-rbac.yaml` | ServiceAccount + ClusterRole/Binding: CRUD on Traefik CRDs and leases. | +| `pangolin-controller-service.yaml` | ClusterIP for controller metrics. | +| `pangolin-controller-api-key-sealed.yaml` | Sealed `Authorization` header the controller sends to the internal API. | + +ArgoCD Application: [`apps/app-pangolin.yaml`](../app-pangolin.yaml) — sync policy is +`automated: { prune: true, selfHeal: true }`. Any local edit must land in git. + +## SSO integration (Pocket-ID) + +Pangolin authenticates users against Pocket-ID via OIDC. Two independent SSO +flows exist and should not be confused: + +1. **Logging into the Pangolin dashboard.** Configured in + `pangolin-config.yaml` (`oidc.*`) and via env in `pangolin-deployment.yaml`: + - Issuer: `https://auth.basicstack.de` + - Callback URL: `https://pangolin.basicstack.de/auth/callback` + - Scopes: `openid profile email groups` + - Client id/secret: `pangolin-secrets` sealed secret keys + `oidc-client-id` / `oidc-client-secret`. +2. **Protecting a downstream resource (e.g. paperclip.basicstack.de).** + Handled by the badger Traefik middleware that the controller emits. + The middleware calls back into Pangolin's internal API (port 3001) to + check the session cookie, and if unauthenticated redirects the browser + through Pangolin's own OIDC flow. There is nothing to configure per-resource + in Kubernetes — the resource's auth policy is set inside the Pangolin UI. + +The Pocket-ID client is provisioned separately (see [pocket-id docs](https://pocket-id.org/docs/client-examples/pangolin)). +Do not reconfigure the OIDC client from the Pangolin side once it is working. + +## Ingress protection workflow + +The end-to-end flow for a protected hostname like `paperclip.basicstack.de`: + +1. Operator creates a **Resource** in the Pangolin UI (hostname, backend + Service, port, auth policy). +2. Pangolin persists it in Postgres and exposes the new desired state at + `http://pangolin.pangolin.svc.cluster.local:3001/api/v1/traefik-config`. +3. `pangolin-kube-controller` polls that endpoint on a ~15s loop. +4. On change, the controller server-side-applies into `ns=pangolin`: + - `IngressRoute` for the hostname (with `tls.certResolver: letsencrypt`). + - `Middleware/badger` (session check + OIDC redirect). + - `TraefikService/1--service` pointing at the backend + Service in the app's namespace, port from the DB. +5. Traefik picks the new CRDs up and starts serving the route. +6. First request → redirected to Pocket-ID → session cookie → subsequent + requests are proxied to the backend Service. + +## Adding a new protected ingress + +Prereqs the target app must already satisfy: + +- A Kubernetes `Service` in its own namespace, listening on a known port. +- A DNS record for the hostname pointing at the cluster ingress IP. +- No competing plain `Ingress` in another namespace for the same hostname. + If one exists (e.g. a legacy direct ingress), delete it first — Traefik + will otherwise route the pre-existing ingress and Pangolin's IngressRoute + never wins. + +Steps: + +1. **Log in** to (SSO via Pocket-ID, or + local admin from the `pangolin-secrets` sealed secret). +2. **Create the Resource.** + - Type: HTTP/HTTPS. + - Domain: the exact FQDN (e.g. `newapp.basicstack.de`). + - Base domain: `basicstack.de`. +3. **Target.** + - Target type: HTTP. + - Backend URL: `http://..svc.cluster.local:` + — the port here is written to the `targets.port` column in Pangolin's + Postgres and used by the controller when it generates the `TraefikService`. + If the Service port later changes, update the target port in the UI (or + via SQL: `UPDATE targets SET port= WHERE "targetId"=;`). + - Path match: `/`, strip prefix off. +4. **TLS.** Enable TLS, force HTTPS redirect. The Pangolin-generated IngressRoute + hardcodes `tls.certResolver: letsencrypt`, which is provisioned on our Traefik + install (HTTP-01, persistent `/data` PVC). Do not disable that resolver in the + Traefik `HelmChartConfig` without also removing every Pangolin-generated route + — Traefik will otherwise fail with `nonexistent certificate resolver`. +5. **Auth.** Enable authentication, method SSO/OIDC. In Community Edition (what we + run) there are no shared policies or group rules — add authorised users + individually by email. If you need group-based rules, that's an Enterprise + feature. +6. **Save** — controller applies CRDs within a poll interval (~15s). +7. **Verify.** + + ```bash + kubectl -n pangolin get ingressroute,middleware,traefikservice \ + | grep -i + kubectl -n pangolin logs -l app=pangolin-controller --tail=50 + # HTTP smoke test + curl -sSI https:/// # expect 302 to Pocket-ID + ``` + +## Troubleshooting + +- **badger middleware returns HTTP 500.** The middleware is trying to call + `apiBaseUrl` and either the URL is not an FQDN or Pangolin isn't reachable + from `kube-system`. Confirm `pangolin-config.yaml` sets + `server.internal_hostname` and `server.badger_override` to + `pangolin.pangolin.svc.cluster.local` — the short name `pangolin` does not + resolve from Traefik's namespace. +- **Controller logs `conflict with "controller" using traefik.io/v1alpha1: .spec…`.** + A pre-existing CRD was created with the `Update` verb and the current + controller uses server-side `Apply`. Delete the offending + `Middleware/TraefikService/IngressRoute` once — the controller recreates it + via SSA and the conflict is gone. +- **Traefik logs `service port not found: `.** The port stored in Pangolin's + `targets` table doesn't match the Kubernetes Service port. Fix in the UI, or: + `UPDATE targets SET port= WHERE "targetId"=;` (identifier case matters, + double-quote it). +- **Traefik logs `nonexistent certificate resolver: letsencrypt`.** The + `letsencrypt` resolver isn't loaded in Traefik. Every Pangolin IngressRoute + references it — either define the resolver in Traefik (see + `infrastructure/networking/traefik-helmchartconfig.yaml`) or override + `traefik.cert_resolver` in `pangolin-config.yaml`. Grep the Traefik startup + log for `unable to get ACME account`; that appears when the resolver was + skipped entirely. +- **Route doesn't resolve cross-namespace.** Traefik must run with + `--providers.kubernetescrd.allowCrossNamespace=true`. Pangolin's CRDs live in + `pangolin`; backend Services usually don't. +- **CRDs never appear after saving a Resource.** + ```bash + kubectl -n pangolin get pods -l app=pangolin-controller + kubectl -n pangolin logs -l app=pangolin-controller --tail=200 + kubectl -n pangolin get configmap pangolin-controller-config -o yaml + ``` + Common causes: controller CrashLoop, wrong `CONFIG_ENDPOINT`, missing auth + header sealed secret, RBAC missing after a controller image upgrade. +- **Login loop / redirects to Pocket-ID forever.** Clear cookies for both + `pangolin.basicstack.de` and `auth.basicstack.de`, confirm the OIDC callback + URL matches the Pocket-ID client, and check Pangolin logs for OIDC errors: + `kubectl -n pangolin logs -l app=pangolin`. +- **Pangolin pod stuck in `Init:0/2`.** The `wait-for-postgres` init container + is looping on `pg_isready`. Check `pangolin-postgres` health and the + `pangolin-postgres-secrets` connection details. + +## Critical notes — DO NOT DELETE + +Pangolin holds live user/account/config state. Losing a PVC here means +re-provisioning every protected resource, every target port, every user grant, +and re-doing the OIDC client trust from scratch — plus temporary outages on +every downstream site that Pangolin fronts. + +- **`pangolin-postgres-data` PVC (ns=pangolin, 10 Gi, `hcloud-volumes-encrypted`).** + Backs Postgres. Contains resources, targets, users, sessions, and everything + the controller reads. Annotated with + `argocd.argoproj.io/sync-options: Delete=false` — do not remove that + annotation, and do not delete the PVC as a "reset" step. Snapshot the volume + before any migration or destructive DB operation. +- **`pangolin-data` PVC (ns=pangolin, 10 Gi, `hcloud-volumes-encrypted`).** + Runtime files under `/app/config`. Also marked `Delete=false`. Do not delete. +- **Sealed secrets** (`pangolin-secrets`, `pangolin-postgres-secrets`, + `pangolin-controller-api-key`) — rotating them out of order will lock the + controller out of the internal API or take Pangolin off Postgres. Rotate one + secret at a time and verify the corresponding pod comes back before + proceeding. +- **`pangolin-config` ConfigMap.** Editing `server.internal_hostname` / + `server.badger_override` back to the default short hostname will break the + badger middleware cluster-wide (see Troubleshooting). +- **The Traefik `HelmChartConfig`** in `kube-system` + (`infrastructure/networking/traefik-helmchartconfig.yaml`) — Pangolin depends + on `allowCrossNamespace=true` and on the `letsencrypt` resolver being loaded + with a persistent `/data` PVC. Removing either silently breaks every + Pangolin-managed route. + +If access to Pangolin is broken, **restore from a snapshot first; do not +"reset" or re-provision** the admin/OIDC layer. That is the standing rule for +every stateful service in this cluster. + +## Related + +- History and gotchas: internal memory `pangolin-kube-integration` (from DEV-457). +- Configuration walkthrough for the `paperclip.basicstack.de` resource: + [`pangolin-paperclip-configuration-guide-community.md`](pangolin-paperclip-configuration-guide-community.md). +- Pangolin docs: +- pangolin-kube-controller: