docs(pangolin): add apps/pangolin/README.md (DEV-461)
Documents the Pangolin + pangolin-kube-controller deployment: architecture, per-file component map, the two SSO flows (dashboard login vs downstream resource protection), the ingress-protection request flow, a step-by-step runbook for adding a new protected ingress, troubleshooting rooted in the DEV-457 findings, and hard "do not delete" notes on the two PVCs, the sealed secrets, and the Traefik HelmChartConfig that Pangolin depends on. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent
parent
ee56681f86
commit
73430bf378
1 changed files with 240 additions and 0 deletions
240
apps/pangolin/README.md
Normal file
240
apps/pangolin/README.md
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
# Pangolin
|
||||
|
||||
Pangolin is BasicStack's self-hosted reverse proxy and SSO gateway. It sits in front of
|
||||
selected cluster services (e.g. `paperclip.basicstack.de`) and forces every request through
|
||||
Pocket-ID before it reaches the backend. Pangolin does not terminate TLS or handle the
|
||||
raw HTTP request path itself — it publishes a Traefik dynamic config, and the
|
||||
`pangolin-kube-controller` translates that config into Traefik CRDs that our existing
|
||||
Traefik ingress controller enforces.
|
||||
|
||||
Public URL: <https://pangolin.basicstack.de> — App source: <https://github.com/fosrl/pangolin>
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser ──HTTPS──► Traefik (ns=kube-system)
|
||||
│
|
||||
│ IngressRoute + Middleware (badger) + TraefikService
|
||||
│ ▲ managed by pangolin-kube-controller (ns=pangolin)
|
||||
│ │
|
||||
│ │ polls /api/v1/traefik-config every ~15s
|
||||
│ │
|
||||
▼ │
|
||||
Pangolin API (ns=pangolin)
|
||||
├─ port 3000 external dashboard API (session auth, behind Ingress)
|
||||
├─ port 3001 internal traefik-config API (Bearer auth, cluster-internal only)
|
||||
└─ port 3002 UI HTTP (behind Ingress)
|
||||
│
|
||||
├── PostgreSQL (ns=pangolin, pangolin-postgres)
|
||||
└── OIDC provider: Pocket-ID at https://auth.basicstack.de
|
||||
|
||||
Auth flow for a protected resource (e.g. paperclip.basicstack.de):
|
||||
Browser ──► Traefik ──► badger middleware ──callback──► Pangolin API (3001)
|
||||
│
|
||||
▼
|
||||
Pocket-ID / session
|
||||
```
|
||||
|
||||
Two long-running workloads live in the `pangolin` namespace:
|
||||
|
||||
- **Pangolin** — the app itself: dashboard, resource/target CRUD, OIDC handshake,
|
||||
session cookies, and the internal `/api/v1/traefik-config` endpoint that
|
||||
publishes the current desired Traefik state.
|
||||
- **pangolin-kube-controller** — a Kubernetes controller (leader-elected) that
|
||||
polls the Pangolin API, then upserts Traefik `IngressRoute`, `Middleware`,
|
||||
`TraefikService`, etc. via server-side apply into the `pangolin` namespace.
|
||||
|
||||
The controller only writes CRDs into `pangolin`; the backing Service typically
|
||||
lives in the target app's own namespace. Traefik is started with
|
||||
`--providers.kubernetescrd.allowCrossNamespace=true` so those cross-namespace
|
||||
refs resolve.
|
||||
|
||||
## Components (files in this directory)
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `namespace.yaml` | Creates the `pangolin` namespace. |
|
||||
| `pangolin-config.yaml` | Pangolin `config.yml` template (ConfigMap). `__DATABASE_URL__` is rendered at startup by the `render-config` init container from the sealed secret. |
|
||||
| `pangolin-deployment.yaml` | Pangolin app Deployment, Service (ports 3000/3001/3002), Ingress, and the `pangolin-data` PVC. |
|
||||
| `pangolin-secrets-sealed.yaml` | Sealed OIDC client id/secret and local admin password. |
|
||||
| `pangolin-postgres-secrets-sealed.yaml` | Sealed Postgres credentials plus the rendered `DATABASE_URL`. |
|
||||
| `postgres.yaml` | Single-replica Postgres 17.5 + `pangolin-postgres-data` PVC. |
|
||||
| `pangolin-controller-config.yaml` | `pangolin-kube-controller` env: `CONFIG_ENDPOINT`, target namespace, log level. |
|
||||
| `pangolin-controller-deployment.yaml` | Controller Deployment (image `fosrl/pangolin-kube-controller:0.1.0-alpha.1`). |
|
||||
| `pangolin-controller-rbac.yaml` | ServiceAccount + ClusterRole/Binding: CRUD on Traefik CRDs and leases. |
|
||||
| `pangolin-controller-service.yaml` | ClusterIP for controller metrics. |
|
||||
| `pangolin-controller-api-key-sealed.yaml` | Sealed `Authorization` header the controller sends to the internal API. |
|
||||
|
||||
ArgoCD Application: [`apps/app-pangolin.yaml`](../app-pangolin.yaml) — sync policy is
|
||||
`automated: { prune: true, selfHeal: true }`. Any local edit must land in git.
|
||||
|
||||
## SSO integration (Pocket-ID)
|
||||
|
||||
Pangolin authenticates users against Pocket-ID via OIDC. Two independent SSO
|
||||
flows exist and should not be confused:
|
||||
|
||||
1. **Logging into the Pangolin dashboard.** Configured in
|
||||
`pangolin-config.yaml` (`oidc.*`) and via env in `pangolin-deployment.yaml`:
|
||||
- Issuer: `https://auth.basicstack.de`
|
||||
- Callback URL: `https://pangolin.basicstack.de/auth/callback`
|
||||
- Scopes: `openid profile email groups`
|
||||
- Client id/secret: `pangolin-secrets` sealed secret keys
|
||||
`oidc-client-id` / `oidc-client-secret`.
|
||||
2. **Protecting a downstream resource (e.g. paperclip.basicstack.de).**
|
||||
Handled by the badger Traefik middleware that the controller emits.
|
||||
The middleware calls back into Pangolin's internal API (port 3001) to
|
||||
check the session cookie, and if unauthenticated redirects the browser
|
||||
through Pangolin's own OIDC flow. There is nothing to configure per-resource
|
||||
in Kubernetes — the resource's auth policy is set inside the Pangolin UI.
|
||||
|
||||
The Pocket-ID client is provisioned separately (see [pocket-id docs](https://pocket-id.org/docs/client-examples/pangolin)).
|
||||
Do not reconfigure the OIDC client from the Pangolin side once it is working.
|
||||
|
||||
## Ingress protection workflow
|
||||
|
||||
The end-to-end flow for a protected hostname like `paperclip.basicstack.de`:
|
||||
|
||||
1. Operator creates a **Resource** in the Pangolin UI (hostname, backend
|
||||
Service, port, auth policy).
|
||||
2. Pangolin persists it in Postgres and exposes the new desired state at
|
||||
`http://pangolin.pangolin.svc.cluster.local:3001/api/v1/traefik-config`.
|
||||
3. `pangolin-kube-controller` polls that endpoint on a ~15s loop.
|
||||
4. On change, the controller server-side-applies into `ns=pangolin`:
|
||||
- `IngressRoute` for the hostname (with `tls.certResolver: letsencrypt`).
|
||||
- `Middleware/badger` (session check + OIDC redirect).
|
||||
- `TraefikService/1-<resource>-service` pointing at the backend
|
||||
Service in the app's namespace, port from the DB.
|
||||
5. Traefik picks the new CRDs up and starts serving the route.
|
||||
6. First request → redirected to Pocket-ID → session cookie → subsequent
|
||||
requests are proxied to the backend Service.
|
||||
|
||||
## Adding a new protected ingress
|
||||
|
||||
Prereqs the target app must already satisfy:
|
||||
|
||||
- A Kubernetes `Service` in its own namespace, listening on a known port.
|
||||
- A DNS record for the hostname pointing at the cluster ingress IP.
|
||||
- No competing plain `Ingress` in another namespace for the same hostname.
|
||||
If one exists (e.g. a legacy direct ingress), delete it first — Traefik
|
||||
will otherwise route the pre-existing ingress and Pangolin's IngressRoute
|
||||
never wins.
|
||||
|
||||
Steps:
|
||||
|
||||
1. **Log in** to <https://pangolin.basicstack.de> (SSO via Pocket-ID, or
|
||||
local admin from the `pangolin-secrets` sealed secret).
|
||||
2. **Create the Resource.**
|
||||
- Type: HTTP/HTTPS.
|
||||
- Domain: the exact FQDN (e.g. `newapp.basicstack.de`).
|
||||
- Base domain: `basicstack.de`.
|
||||
3. **Target.**
|
||||
- Target type: HTTP.
|
||||
- Backend URL: `http://<service>.<namespace>.svc.cluster.local:<port>`
|
||||
— the port here is written to the `targets.port` column in Pangolin's
|
||||
Postgres and used by the controller when it generates the `TraefikService`.
|
||||
If the Service port later changes, update the target port in the UI (or
|
||||
via SQL: `UPDATE targets SET port=<N> WHERE "targetId"=<id>;`).
|
||||
- Path match: `/`, strip prefix off.
|
||||
4. **TLS.** Enable TLS, force HTTPS redirect. The Pangolin-generated IngressRoute
|
||||
hardcodes `tls.certResolver: letsencrypt`, which is provisioned on our Traefik
|
||||
install (HTTP-01, persistent `/data` PVC). Do not disable that resolver in the
|
||||
Traefik `HelmChartConfig` without also removing every Pangolin-generated route
|
||||
— Traefik will otherwise fail with `nonexistent certificate resolver`.
|
||||
5. **Auth.** Enable authentication, method SSO/OIDC. In Community Edition (what we
|
||||
run) there are no shared policies or group rules — add authorised users
|
||||
individually by email. If you need group-based rules, that's an Enterprise
|
||||
feature.
|
||||
6. **Save** — controller applies CRDs within a poll interval (~15s).
|
||||
7. **Verify.**
|
||||
|
||||
```bash
|
||||
kubectl -n pangolin get ingressroute,middleware,traefikservice \
|
||||
| grep -i <resource-name>
|
||||
kubectl -n pangolin logs -l app=pangolin-controller --tail=50
|
||||
# HTTP smoke test
|
||||
curl -sSI https://<host>/ # expect 302 to Pocket-ID
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **badger middleware returns HTTP 500.** The middleware is trying to call
|
||||
`apiBaseUrl` and either the URL is not an FQDN or Pangolin isn't reachable
|
||||
from `kube-system`. Confirm `pangolin-config.yaml` sets
|
||||
`server.internal_hostname` and `server.badger_override` to
|
||||
`pangolin.pangolin.svc.cluster.local` — the short name `pangolin` does not
|
||||
resolve from Traefik's namespace.
|
||||
- **Controller logs `conflict with "controller" using traefik.io/v1alpha1: .spec…`.**
|
||||
A pre-existing CRD was created with the `Update` verb and the current
|
||||
controller uses server-side `Apply`. Delete the offending
|
||||
`Middleware/TraefikService/IngressRoute` once — the controller recreates it
|
||||
via SSA and the conflict is gone.
|
||||
- **Traefik logs `service port not found: <N>`.** The port stored in Pangolin's
|
||||
`targets` table doesn't match the Kubernetes Service port. Fix in the UI, or:
|
||||
`UPDATE targets SET port=<N> WHERE "targetId"=<id>;` (identifier case matters,
|
||||
double-quote it).
|
||||
- **Traefik logs `nonexistent certificate resolver: letsencrypt`.** The
|
||||
`letsencrypt` resolver isn't loaded in Traefik. Every Pangolin IngressRoute
|
||||
references it — either define the resolver in Traefik (see
|
||||
`infrastructure/networking/traefik-helmchartconfig.yaml`) or override
|
||||
`traefik.cert_resolver` in `pangolin-config.yaml`. Grep the Traefik startup
|
||||
log for `unable to get ACME account`; that appears when the resolver was
|
||||
skipped entirely.
|
||||
- **Route doesn't resolve cross-namespace.** Traefik must run with
|
||||
`--providers.kubernetescrd.allowCrossNamespace=true`. Pangolin's CRDs live in
|
||||
`pangolin`; backend Services usually don't.
|
||||
- **CRDs never appear after saving a Resource.**
|
||||
```bash
|
||||
kubectl -n pangolin get pods -l app=pangolin-controller
|
||||
kubectl -n pangolin logs -l app=pangolin-controller --tail=200
|
||||
kubectl -n pangolin get configmap pangolin-controller-config -o yaml
|
||||
```
|
||||
Common causes: controller CrashLoop, wrong `CONFIG_ENDPOINT`, missing auth
|
||||
header sealed secret, RBAC missing after a controller image upgrade.
|
||||
- **Login loop / redirects to Pocket-ID forever.** Clear cookies for both
|
||||
`pangolin.basicstack.de` and `auth.basicstack.de`, confirm the OIDC callback
|
||||
URL matches the Pocket-ID client, and check Pangolin logs for OIDC errors:
|
||||
`kubectl -n pangolin logs -l app=pangolin`.
|
||||
- **Pangolin pod stuck in `Init:0/2`.** The `wait-for-postgres` init container
|
||||
is looping on `pg_isready`. Check `pangolin-postgres` health and the
|
||||
`pangolin-postgres-secrets` connection details.
|
||||
|
||||
## Critical notes — DO NOT DELETE
|
||||
|
||||
Pangolin holds live user/account/config state. Losing a PVC here means
|
||||
re-provisioning every protected resource, every target port, every user grant,
|
||||
and re-doing the OIDC client trust from scratch — plus temporary outages on
|
||||
every downstream site that Pangolin fronts.
|
||||
|
||||
- **`pangolin-postgres-data` PVC (ns=pangolin, 10 Gi, `hcloud-volumes-encrypted`).**
|
||||
Backs Postgres. Contains resources, targets, users, sessions, and everything
|
||||
the controller reads. Annotated with
|
||||
`argocd.argoproj.io/sync-options: Delete=false` — do not remove that
|
||||
annotation, and do not delete the PVC as a "reset" step. Snapshot the volume
|
||||
before any migration or destructive DB operation.
|
||||
- **`pangolin-data` PVC (ns=pangolin, 10 Gi, `hcloud-volumes-encrypted`).**
|
||||
Runtime files under `/app/config`. Also marked `Delete=false`. Do not delete.
|
||||
- **Sealed secrets** (`pangolin-secrets`, `pangolin-postgres-secrets`,
|
||||
`pangolin-controller-api-key`) — rotating them out of order will lock the
|
||||
controller out of the internal API or take Pangolin off Postgres. Rotate one
|
||||
secret at a time and verify the corresponding pod comes back before
|
||||
proceeding.
|
||||
- **`pangolin-config` ConfigMap.** Editing `server.internal_hostname` /
|
||||
`server.badger_override` back to the default short hostname will break the
|
||||
badger middleware cluster-wide (see Troubleshooting).
|
||||
- **The Traefik `HelmChartConfig`** in `kube-system`
|
||||
(`infrastructure/networking/traefik-helmchartconfig.yaml`) — Pangolin depends
|
||||
on `allowCrossNamespace=true` and on the `letsencrypt` resolver being loaded
|
||||
with a persistent `/data` PVC. Removing either silently breaks every
|
||||
Pangolin-managed route.
|
||||
|
||||
If access to Pangolin is broken, **restore from a snapshot first; do not
|
||||
"reset" or re-provision** the admin/OIDC layer. That is the standing rule for
|
||||
every stateful service in this cluster.
|
||||
|
||||
## Related
|
||||
|
||||
- History and gotchas: internal memory `pangolin-kube-integration` (from DEV-457).
|
||||
- Configuration walkthrough for the `paperclip.basicstack.de` resource:
|
||||
[`pangolin-paperclip-configuration-guide-community.md`](pangolin-paperclip-configuration-guide-community.md).
|
||||
- Pangolin docs: <https://docs.pangolin.net/>
|
||||
- pangolin-kube-controller: <https://github.com/fosrl/pangolin-kube-controller>
|
||||
Loading…
Add table
Reference in a new issue