docs(pangolin): add apps/pangolin/README.md (DEV-461)
Documents the Pangolin + pangolin-kube-controller deployment: architecture, per-file component map, the two SSO flows (dashboard login vs downstream resource protection), the ingress-protection request flow, a step-by-step runbook for adding a new protected ingress, troubleshooting rooted in the DEV-457 findings, and hard "do not delete" notes on the two PVCs, the sealed secrets, and the Traefik HelmChartConfig that Pangolin depends on. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
ee56681f86
commit
73430bf378
1 changed files with 240 additions and 0 deletions
240
apps/pangolin/README.md
Normal file
240
apps/pangolin/README.md
Normal file
|
|
@ -0,0 +1,240 @@
|
||||||
|
# Pangolin
|
||||||
|
|
||||||
|
Pangolin is BasicStack's self-hosted reverse proxy and SSO gateway. It sits in front of
|
||||||
|
selected cluster services (e.g. `paperclip.basicstack.de`) and forces every request through
|
||||||
|
Pocket-ID before it reaches the backend. Pangolin does not terminate TLS or handle the
|
||||||
|
raw HTTP request path itself — it publishes a Traefik dynamic config, and the
|
||||||
|
`pangolin-kube-controller` translates that config into Traefik CRDs that our existing
|
||||||
|
Traefik ingress controller enforces.
|
||||||
|
|
||||||
|
Public URL: <https://pangolin.basicstack.de> — App source: <https://github.com/fosrl/pangolin>
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Browser ──HTTPS──► Traefik (ns=kube-system)
|
||||||
|
│
|
||||||
|
│ IngressRoute + Middleware (badger) + TraefikService
|
||||||
|
│ ▲ managed by pangolin-kube-controller (ns=pangolin)
|
||||||
|
│ │
|
||||||
|
│ │ polls /api/v1/traefik-config every ~15s
|
||||||
|
│ │
|
||||||
|
▼ │
|
||||||
|
Pangolin API (ns=pangolin)
|
||||||
|
├─ port 3000 external dashboard API (session auth, behind Ingress)
|
||||||
|
├─ port 3001 internal traefik-config API (Bearer auth, cluster-internal only)
|
||||||
|
└─ port 3002 UI HTTP (behind Ingress)
|
||||||
|
│
|
||||||
|
├── PostgreSQL (ns=pangolin, pangolin-postgres)
|
||||||
|
└── OIDC provider: Pocket-ID at https://auth.basicstack.de
|
||||||
|
|
||||||
|
Auth flow for a protected resource (e.g. paperclip.basicstack.de):
|
||||||
|
Browser ──► Traefik ──► badger middleware ──callback──► Pangolin API (3001)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Pocket-ID / session
|
||||||
|
```
|
||||||
|
|
||||||
|
Two long-running workloads live in the `pangolin` namespace:
|
||||||
|
|
||||||
|
- **Pangolin** — the app itself: dashboard, resource/target CRUD, OIDC handshake,
|
||||||
|
session cookies, and the internal `/api/v1/traefik-config` endpoint that
|
||||||
|
publishes the current desired Traefik state.
|
||||||
|
- **pangolin-kube-controller** — a Kubernetes controller (leader-elected) that
|
||||||
|
polls the Pangolin API, then upserts Traefik `IngressRoute`, `Middleware`,
|
||||||
|
`TraefikService`, etc. via server-side apply into the `pangolin` namespace.
|
||||||
|
|
||||||
|
The controller only writes CRDs into `pangolin`; the backing Service typically
|
||||||
|
lives in the target app's own namespace. Traefik is started with
|
||||||
|
`--providers.kubernetescrd.allowCrossNamespace=true` so those cross-namespace
|
||||||
|
refs resolve.
|
||||||
|
|
||||||
|
## Components (files in this directory)
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| `namespace.yaml` | Creates the `pangolin` namespace. |
|
||||||
|
| `pangolin-config.yaml` | Pangolin `config.yml` template (ConfigMap). `__DATABASE_URL__` is rendered at startup by the `render-config` init container from the sealed secret. |
|
||||||
|
| `pangolin-deployment.yaml` | Pangolin app Deployment, Service (ports 3000/3001/3002), Ingress, and the `pangolin-data` PVC. |
|
||||||
|
| `pangolin-secrets-sealed.yaml` | Sealed OIDC client id/secret and local admin password. |
|
||||||
|
| `pangolin-postgres-secrets-sealed.yaml` | Sealed Postgres credentials plus the rendered `DATABASE_URL`. |
|
||||||
|
| `postgres.yaml` | Single-replica Postgres 17.5 + `pangolin-postgres-data` PVC. |
|
||||||
|
| `pangolin-controller-config.yaml` | `pangolin-kube-controller` env: `CONFIG_ENDPOINT`, target namespace, log level. |
|
||||||
|
| `pangolin-controller-deployment.yaml` | Controller Deployment (image `fosrl/pangolin-kube-controller:0.1.0-alpha.1`). |
|
||||||
|
| `pangolin-controller-rbac.yaml` | ServiceAccount + ClusterRole/Binding: CRUD on Traefik CRDs and leases. |
|
||||||
|
| `pangolin-controller-service.yaml` | ClusterIP for controller metrics. |
|
||||||
|
| `pangolin-controller-api-key-sealed.yaml` | Sealed `Authorization` header the controller sends to the internal API. |
|
||||||
|
|
||||||
|
ArgoCD Application: [`apps/app-pangolin.yaml`](../app-pangolin.yaml) — sync policy is
|
||||||
|
`automated: { prune: true, selfHeal: true }`. Any local edit must land in git.
|
||||||
|
|
||||||
|
## SSO integration (Pocket-ID)
|
||||||
|
|
||||||
|
Pangolin authenticates users against Pocket-ID via OIDC. Two independent SSO
|
||||||
|
flows exist and should not be confused:
|
||||||
|
|
||||||
|
1. **Logging into the Pangolin dashboard.** Configured in
|
||||||
|
`pangolin-config.yaml` (`oidc.*`) and via env in `pangolin-deployment.yaml`:
|
||||||
|
- Issuer: `https://auth.basicstack.de`
|
||||||
|
- Callback URL: `https://pangolin.basicstack.de/auth/callback`
|
||||||
|
- Scopes: `openid profile email groups`
|
||||||
|
- Client id/secret: `pangolin-secrets` sealed secret keys
|
||||||
|
`oidc-client-id` / `oidc-client-secret`.
|
||||||
|
2. **Protecting a downstream resource (e.g. paperclip.basicstack.de).**
|
||||||
|
Handled by the badger Traefik middleware that the controller emits.
|
||||||
|
The middleware calls back into Pangolin's internal API (port 3001) to
|
||||||
|
check the session cookie, and if unauthenticated redirects the browser
|
||||||
|
through Pangolin's own OIDC flow. There is nothing to configure per-resource
|
||||||
|
in Kubernetes — the resource's auth policy is set inside the Pangolin UI.
|
||||||
|
|
||||||
|
The Pocket-ID client is provisioned separately (see [pocket-id docs](https://pocket-id.org/docs/client-examples/pangolin)).
|
||||||
|
Do not reconfigure the OIDC client from the Pangolin side once it is working.
|
||||||
|
|
||||||
|
## Ingress protection workflow
|
||||||
|
|
||||||
|
The end-to-end flow for a protected hostname like `paperclip.basicstack.de`:
|
||||||
|
|
||||||
|
1. Operator creates a **Resource** in the Pangolin UI (hostname, backend
|
||||||
|
Service, port, auth policy).
|
||||||
|
2. Pangolin persists it in Postgres and exposes the new desired state at
|
||||||
|
`http://pangolin.pangolin.svc.cluster.local:3001/api/v1/traefik-config`.
|
||||||
|
3. `pangolin-kube-controller` polls that endpoint on a ~15s loop.
|
||||||
|
4. On change, the controller server-side-applies into `ns=pangolin`:
|
||||||
|
- `IngressRoute` for the hostname (with `tls.certResolver: letsencrypt`).
|
||||||
|
- `Middleware/badger` (session check + OIDC redirect).
|
||||||
|
- `TraefikService/1-<resource>-service` pointing at the backend
|
||||||
|
Service in the app's namespace, port from the DB.
|
||||||
|
5. Traefik picks the new CRDs up and starts serving the route.
|
||||||
|
6. First request → redirected to Pocket-ID → session cookie → subsequent
|
||||||
|
requests are proxied to the backend Service.
|
||||||
|
|
||||||
|
## Adding a new protected ingress
|
||||||
|
|
||||||
|
Prereqs the target app must already satisfy:
|
||||||
|
|
||||||
|
- A Kubernetes `Service` in its own namespace, listening on a known port.
|
||||||
|
- A DNS record for the hostname pointing at the cluster ingress IP.
|
||||||
|
- No competing plain `Ingress` in another namespace for the same hostname.
|
||||||
|
If one exists (e.g. a legacy direct ingress), delete it first — Traefik
|
||||||
|
will otherwise route the pre-existing ingress and Pangolin's IngressRoute
|
||||||
|
never wins.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
|
||||||
|
1. **Log in** to <https://pangolin.basicstack.de> (SSO via Pocket-ID, or
|
||||||
|
local admin from the `pangolin-secrets` sealed secret).
|
||||||
|
2. **Create the Resource.**
|
||||||
|
- Type: HTTP/HTTPS.
|
||||||
|
- Domain: the exact FQDN (e.g. `newapp.basicstack.de`).
|
||||||
|
- Base domain: `basicstack.de`.
|
||||||
|
3. **Target.**
|
||||||
|
- Target type: HTTP.
|
||||||
|
- Backend URL: `http://<service>.<namespace>.svc.cluster.local:<port>`
|
||||||
|
— the port here is written to the `targets.port` column in Pangolin's
|
||||||
|
Postgres and used by the controller when it generates the `TraefikService`.
|
||||||
|
If the Service port later changes, update the target port in the UI (or
|
||||||
|
via SQL: `UPDATE targets SET port=<N> WHERE "targetId"=<id>;`).
|
||||||
|
- Path match: `/`, strip prefix off.
|
||||||
|
4. **TLS.** Enable TLS, force HTTPS redirect. The Pangolin-generated IngressRoute
|
||||||
|
hardcodes `tls.certResolver: letsencrypt`, which is provisioned on our Traefik
|
||||||
|
install (HTTP-01, persistent `/data` PVC). Do not disable that resolver in the
|
||||||
|
Traefik `HelmChartConfig` without also removing every Pangolin-generated route
|
||||||
|
— Traefik will otherwise fail with `nonexistent certificate resolver`.
|
||||||
|
5. **Auth.** Enable authentication, method SSO/OIDC. In Community Edition (what we
|
||||||
|
run) there are no shared policies or group rules — add authorised users
|
||||||
|
individually by email. If you need group-based rules, that's an Enterprise
|
||||||
|
feature.
|
||||||
|
6. **Save** — controller applies CRDs within a poll interval (~15s).
|
||||||
|
7. **Verify.**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n pangolin get ingressroute,middleware,traefikservice \
|
||||||
|
| grep -i <resource-name>
|
||||||
|
kubectl -n pangolin logs -l app=pangolin-controller --tail=50
|
||||||
|
# HTTP smoke test
|
||||||
|
curl -sSI https://<host>/ # expect 302 to Pocket-ID
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- **badger middleware returns HTTP 500.** The middleware is trying to call
|
||||||
|
`apiBaseUrl` and either the URL is not an FQDN or Pangolin isn't reachable
|
||||||
|
from `kube-system`. Confirm `pangolin-config.yaml` sets
|
||||||
|
`server.internal_hostname` and `server.badger_override` to
|
||||||
|
`pangolin.pangolin.svc.cluster.local` — the short name `pangolin` does not
|
||||||
|
resolve from Traefik's namespace.
|
||||||
|
- **Controller logs `conflict with "controller" using traefik.io/v1alpha1: .spec…`.**
|
||||||
|
A pre-existing CRD was created with the `Update` verb and the current
|
||||||
|
controller uses server-side `Apply`. Delete the offending
|
||||||
|
`Middleware/TraefikService/IngressRoute` once — the controller recreates it
|
||||||
|
via SSA and the conflict is gone.
|
||||||
|
- **Traefik logs `service port not found: <N>`.** The port stored in Pangolin's
|
||||||
|
`targets` table doesn't match the Kubernetes Service port. Fix in the UI, or:
|
||||||
|
`UPDATE targets SET port=<N> WHERE "targetId"=<id>;` (identifier case matters,
|
||||||
|
double-quote it).
|
||||||
|
- **Traefik logs `nonexistent certificate resolver: letsencrypt`.** The
|
||||||
|
`letsencrypt` resolver isn't loaded in Traefik. Every Pangolin IngressRoute
|
||||||
|
references it — either define the resolver in Traefik (see
|
||||||
|
`infrastructure/networking/traefik-helmchartconfig.yaml`) or override
|
||||||
|
`traefik.cert_resolver` in `pangolin-config.yaml`. Grep the Traefik startup
|
||||||
|
log for `unable to get ACME account`; that appears when the resolver was
|
||||||
|
skipped entirely.
|
||||||
|
- **Route doesn't resolve cross-namespace.** Traefik must run with
|
||||||
|
`--providers.kubernetescrd.allowCrossNamespace=true`. Pangolin's CRDs live in
|
||||||
|
`pangolin`; backend Services usually don't.
|
||||||
|
- **CRDs never appear after saving a Resource.**
|
||||||
|
```bash
|
||||||
|
kubectl -n pangolin get pods -l app=pangolin-controller
|
||||||
|
kubectl -n pangolin logs -l app=pangolin-controller --tail=200
|
||||||
|
kubectl -n pangolin get configmap pangolin-controller-config -o yaml
|
||||||
|
```
|
||||||
|
Common causes: controller CrashLoop, wrong `CONFIG_ENDPOINT`, missing auth
|
||||||
|
header sealed secret, RBAC missing after a controller image upgrade.
|
||||||
|
- **Login loop / redirects to Pocket-ID forever.** Clear cookies for both
|
||||||
|
`pangolin.basicstack.de` and `auth.basicstack.de`, confirm the OIDC callback
|
||||||
|
URL matches the Pocket-ID client, and check Pangolin logs for OIDC errors:
|
||||||
|
`kubectl -n pangolin logs -l app=pangolin`.
|
||||||
|
- **Pangolin pod stuck in `Init:0/2`.** The `wait-for-postgres` init container
|
||||||
|
is looping on `pg_isready`. Check `pangolin-postgres` health and the
|
||||||
|
`pangolin-postgres-secrets` connection details.
|
||||||
|
|
||||||
|
## Critical notes — DO NOT DELETE
|
||||||
|
|
||||||
|
Pangolin holds live user/account/config state. Losing a PVC here means
|
||||||
|
re-provisioning every protected resource, every target port, every user grant,
|
||||||
|
and re-doing the OIDC client trust from scratch — plus temporary outages on
|
||||||
|
every downstream site that Pangolin fronts.
|
||||||
|
|
||||||
|
- **`pangolin-postgres-data` PVC (ns=pangolin, 10 Gi, `hcloud-volumes-encrypted`).**
|
||||||
|
Backs Postgres. Contains resources, targets, users, sessions, and everything
|
||||||
|
the controller reads. Annotated with
|
||||||
|
`argocd.argoproj.io/sync-options: Delete=false` — do not remove that
|
||||||
|
annotation, and do not delete the PVC as a "reset" step. Snapshot the volume
|
||||||
|
before any migration or destructive DB operation.
|
||||||
|
- **`pangolin-data` PVC (ns=pangolin, 10 Gi, `hcloud-volumes-encrypted`).**
|
||||||
|
Runtime files under `/app/config`. Also marked `Delete=false`. Do not delete.
|
||||||
|
- **Sealed secrets** (`pangolin-secrets`, `pangolin-postgres-secrets`,
|
||||||
|
`pangolin-controller-api-key`) — rotating them out of order will lock the
|
||||||
|
controller out of the internal API or take Pangolin off Postgres. Rotate one
|
||||||
|
secret at a time and verify the corresponding pod comes back before
|
||||||
|
proceeding.
|
||||||
|
- **`pangolin-config` ConfigMap.** Editing `server.internal_hostname` /
|
||||||
|
`server.badger_override` back to the default short hostname will break the
|
||||||
|
badger middleware cluster-wide (see Troubleshooting).
|
||||||
|
- **The Traefik `HelmChartConfig`** in `kube-system`
|
||||||
|
(`infrastructure/networking/traefik-helmchartconfig.yaml`) — Pangolin depends
|
||||||
|
on `allowCrossNamespace=true` and on the `letsencrypt` resolver being loaded
|
||||||
|
with a persistent `/data` PVC. Removing either silently breaks every
|
||||||
|
Pangolin-managed route.
|
||||||
|
|
||||||
|
If access to Pangolin is broken, **restore from a snapshot first; do not
|
||||||
|
"reset" or re-provision** the admin/OIDC layer. That is the standing rule for
|
||||||
|
every stateful service in this cluster.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- History and gotchas: internal memory `pangolin-kube-integration` (from DEV-457).
|
||||||
|
- Configuration walkthrough for the `paperclip.basicstack.de` resource:
|
||||||
|
[`pangolin-paperclip-configuration-guide-community.md`](pangolin-paperclip-configuration-guide-community.md).
|
||||||
|
- Pangolin docs: <https://docs.pangolin.net/>
|
||||||
|
- pangolin-kube-controller: <https://github.com/fosrl/pangolin-kube-controller>
|
||||||
Loading…
Add table
Reference in a new issue