ProofedMesh/services/transparency-log/README.md

93 lines
3.8 KiB
Markdown
Raw Normal View History

# transparency-log
Serverseitige Implementierung von **`POST /api/v1/log/entries`** laut
[`docs/adr/0005-signing-service-and-transparency-log-api.md`](../../docs/adr/0005-signing-service-and-transparency-log-api.md).
Die admin-api und spätere Agent-Dienste rufen diesen Service auf, um DSSE-
Envelopes in einen Append-Only-Log zu schreiben und einen Inclusion-Proof
zurückzubekommen. Der zurückgelieferte `content_hash` ist die stabile
Attestation-ID überall sonst im System.
## Vertrag
Siehe ADR-0005. Zusammengefasst:
- `POST /api/v1/log/entries` mit `{ envelope, content_hash }`.
- Server rechnet `content_hash` serverseitig nach — Mismatch → **409**.
- Idempotent: Wiederaufruf mit identischem `content_hash` liefert `200` +
dasselbe `log_index` + gültigen (aktuellen) `inclusion_proof`.
- Erstanlage antwortet mit `201`.
- Trillian-Backend nicht erreichbar / nicht deployed → **503**.
Authentisierung (Pocket-ID Bearer) wird außerhalb des Handlers über eine
Middleware eingehängt; der Core-Handler ist absichtlich auth-agnostisch,
damit Dev- und CI-Runs ohne IdP laufen.
## Backends
transparency-log: real Trillian backend wiring + agent-trust deploy (DEV-604) - services/transparency-log/pkg/backend/trillian.go: real gRPC client on github.com/google/trillian v1.6.1. Submit() does QueueLeaf with the canonical DSSE envelope as LeafValue and the raw content_hash bytes as LeafIdentityHash, then polls GetLatestSignedLogRoot + GetInclusionProofByHash until the leaf is sequenced. Idempotency uses Trillian's ALREADY_EXISTS signal on the inner QueuedLogLeaf.Status, so a repeated submit still returns 200 OK with the originally-assigned log_index + a valid proof against the current STH. - Unconfigured backend (missing endpoint or LogID) returns the historical ErrTrillianUnavailable so the httpapi layer keeps mapping it to 503 instead of silently 500. Existing stub-returns-503 test updated. - cmd/transparency-log: pointer-receiver Trillian wiring; flag docs mention the production SRV name. - services/transparency-log/Dockerfile: distroless multi-stage build used by deploy/trillian/40-transparency-log.yaml. - deploy/trillian/: full K8s stack for the agent-trust namespace — MySQL 8.4 StatefulSet (schema init ConfigMap vendored from upstream storage.sql v1.6.1), Trillian LogServer + LogSigner Deployments on gcr.io/trillian-opensource-ci:v1.6.1, createtree Job that writes the TreeID to a ConfigMap (re-apply safe), transparency-log Deployment with --backend=trillian pulling TRILLIAN_LOG_ID from the ConfigMap. - deploy/trillian/README.md: first-time apply + smoke-test + secret rotation + signer-key clarification (STH key is Trillian-internal; DSSE envelope signing stays with signing-service / OpenBao). - services/admin-api: go.mod / go.sum pick up the new transitive deps (trillian + grpc) via the local replace on transparency-log. - services/transparency-log/README.md: Backends table updated; next- steps list no longer carries Trillian-deployment as a follow-up. Deployed to the live cluster from this heartbeat: MySQL + LogServer + LogSigner running in agent-trust, tree id 8482455225499059043, service smoke-tested end-to-end (201 on first POST, 200 on idempotent repeat, 409 on content_hash mismatch, tree_size=2 with correct audit path). Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-25 18:39:44 +00:00
| Backend | Status | Wofür |
| ----------- | ----------------------------------------- | -------------------------------------------- |
| `memory` | implementiert | Dev + CI. Append-Log + RFC 6962 Merkle Tree. |
| `trillian` | implementiert (DEV-604); 503 wenn unkonfiguriert | Prod. gRPC-Client auf Trillian LogServer. Siehe [`deploy/trillian/`](../../deploy/trillian/). |
Der In-Memory-Backend ist bewusst klein gehalten: er speichert jeden
Envelope als Leaf-Hash (`SHA-256(0x00 || canonical envelope bytes)`),
rechnet Baumwurzel und Audit-Path pro Submit frisch aus und liefert
deterministische `signed_tree_head_ref`-Werte aus `(tree_size, root)`.
Das reicht für die admin-api-Interop-Tests und den lokalen Durchstich der
Vertrauenskette.
## Starten (lokal)
```bash
cd services/transparency-log
go run ./cmd/transparency-log --addr :8080 --backend memory
```
Smoke-Test (gegen ADR-0005-Beispielpayload):
```bash
curl -sS -X POST http://127.0.0.1:8080/api/v1/log/entries \
-H 'Content-Type: application/json' \
-d '{"envelope":{"payloadType":"application/vnd.in-toto+json","payload":"aGVsbG8","signatures":[{"keyid":"k","sig":"c2ln"}]},"content_hash":"<siehe EnvelopeContentHash>"}'
```
Der korrekte `content_hash` ergibt sich aus `backend.EnvelopeContentHash`
(bit-identisch zu `admin-api/internal/attest.EnvelopeContentHash`).
## Tests
```bash
cd services/transparency-log
go vet ./...
go test ./...
```
Die wichtigen Testpakete:
- `internal/backend` — Append-Log, Idempotenz, content_hash-Mismatch,
RFC-6962-Audit-Path-Verifikation.
- `internal/httpapi` — HTTP-Statuscodes (201/200/400/404/409/503).
- `internal/interop` — admin-api `tlog.HTTPSubmitter` gegen diesen Server,
Happy-Path + Idempotenz + Content-Hash-Mismatch.
Die `go.mod` enthält eine `replace`-Direktive auf die lokale `admin-api`,
damit der Interop-Test den tatsächlichen Produktions-Client importieren
kann.
## Nächste Schritte
transparency-log: real Trillian backend wiring + agent-trust deploy (DEV-604) - services/transparency-log/pkg/backend/trillian.go: real gRPC client on github.com/google/trillian v1.6.1. Submit() does QueueLeaf with the canonical DSSE envelope as LeafValue and the raw content_hash bytes as LeafIdentityHash, then polls GetLatestSignedLogRoot + GetInclusionProofByHash until the leaf is sequenced. Idempotency uses Trillian's ALREADY_EXISTS signal on the inner QueuedLogLeaf.Status, so a repeated submit still returns 200 OK with the originally-assigned log_index + a valid proof against the current STH. - Unconfigured backend (missing endpoint or LogID) returns the historical ErrTrillianUnavailable so the httpapi layer keeps mapping it to 503 instead of silently 500. Existing stub-returns-503 test updated. - cmd/transparency-log: pointer-receiver Trillian wiring; flag docs mention the production SRV name. - services/transparency-log/Dockerfile: distroless multi-stage build used by deploy/trillian/40-transparency-log.yaml. - deploy/trillian/: full K8s stack for the agent-trust namespace — MySQL 8.4 StatefulSet (schema init ConfigMap vendored from upstream storage.sql v1.6.1), Trillian LogServer + LogSigner Deployments on gcr.io/trillian-opensource-ci:v1.6.1, createtree Job that writes the TreeID to a ConfigMap (re-apply safe), transparency-log Deployment with --backend=trillian pulling TRILLIAN_LOG_ID from the ConfigMap. - deploy/trillian/README.md: first-time apply + smoke-test + secret rotation + signer-key clarification (STH key is Trillian-internal; DSSE envelope signing stays with signing-service / OpenBao). - services/admin-api: go.mod / go.sum pick up the new transitive deps (trillian + grpc) via the local replace on transparency-log. - services/transparency-log/README.md: Backends table updated; next- steps list no longer carries Trillian-deployment as a follow-up. Deployed to the live cluster from this heartbeat: MySQL + LogServer + LogSigner running in agent-trust, tree id 8482455225499059043, service smoke-tested end-to-end (201 on first POST, 200 on idempotent repeat, 409 on content_hash mismatch, tree_size=2 with correct audit path). Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-25 18:39:44 +00:00
1. Pocket-ID-Middleware vor den Handler hängen (gleiche OIDC-Discovery wie
signing-service, Audience/`client_id`-Whitelist über OPA-Policy).
transparency-log: real Trillian backend wiring + agent-trust deploy (DEV-604) - services/transparency-log/pkg/backend/trillian.go: real gRPC client on github.com/google/trillian v1.6.1. Submit() does QueueLeaf with the canonical DSSE envelope as LeafValue and the raw content_hash bytes as LeafIdentityHash, then polls GetLatestSignedLogRoot + GetInclusionProofByHash until the leaf is sequenced. Idempotency uses Trillian's ALREADY_EXISTS signal on the inner QueuedLogLeaf.Status, so a repeated submit still returns 200 OK with the originally-assigned log_index + a valid proof against the current STH. - Unconfigured backend (missing endpoint or LogID) returns the historical ErrTrillianUnavailable so the httpapi layer keeps mapping it to 503 instead of silently 500. Existing stub-returns-503 test updated. - cmd/transparency-log: pointer-receiver Trillian wiring; flag docs mention the production SRV name. - services/transparency-log/Dockerfile: distroless multi-stage build used by deploy/trillian/40-transparency-log.yaml. - deploy/trillian/: full K8s stack for the agent-trust namespace — MySQL 8.4 StatefulSet (schema init ConfigMap vendored from upstream storage.sql v1.6.1), Trillian LogServer + LogSigner Deployments on gcr.io/trillian-opensource-ci:v1.6.1, createtree Job that writes the TreeID to a ConfigMap (re-apply safe), transparency-log Deployment with --backend=trillian pulling TRILLIAN_LOG_ID from the ConfigMap. - deploy/trillian/README.md: first-time apply + smoke-test + secret rotation + signer-key clarification (STH key is Trillian-internal; DSSE envelope signing stays with signing-service / OpenBao). - services/admin-api: go.mod / go.sum pick up the new transitive deps (trillian + grpc) via the local replace on transparency-log. - services/transparency-log/README.md: Backends table updated; next- steps list no longer carries Trillian-deployment as a follow-up. Deployed to the live cluster from this heartbeat: MySQL + LogServer + LogSigner running in agent-trust, tree id 8482455225499059043, service smoke-tested end-to-end (201 on first POST, 200 on idempotent repeat, 409 on content_hash mismatch, tree_size=2 with correct audit path). Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-25 18:39:44 +00:00
2. STH-Verteilung: periodisch signierten Tree Head veröffentlichen (eigenes
Deployment, nutzt denselben Signing-Service).
transparency-log: real Trillian backend wiring + agent-trust deploy (DEV-604) - services/transparency-log/pkg/backend/trillian.go: real gRPC client on github.com/google/trillian v1.6.1. Submit() does QueueLeaf with the canonical DSSE envelope as LeafValue and the raw content_hash bytes as LeafIdentityHash, then polls GetLatestSignedLogRoot + GetInclusionProofByHash until the leaf is sequenced. Idempotency uses Trillian's ALREADY_EXISTS signal on the inner QueuedLogLeaf.Status, so a repeated submit still returns 200 OK with the originally-assigned log_index + a valid proof against the current STH. - Unconfigured backend (missing endpoint or LogID) returns the historical ErrTrillianUnavailable so the httpapi layer keeps mapping it to 503 instead of silently 500. Existing stub-returns-503 test updated. - cmd/transparency-log: pointer-receiver Trillian wiring; flag docs mention the production SRV name. - services/transparency-log/Dockerfile: distroless multi-stage build used by deploy/trillian/40-transparency-log.yaml. - deploy/trillian/: full K8s stack for the agent-trust namespace — MySQL 8.4 StatefulSet (schema init ConfigMap vendored from upstream storage.sql v1.6.1), Trillian LogServer + LogSigner Deployments on gcr.io/trillian-opensource-ci:v1.6.1, createtree Job that writes the TreeID to a ConfigMap (re-apply safe), transparency-log Deployment with --backend=trillian pulling TRILLIAN_LOG_ID from the ConfigMap. - deploy/trillian/README.md: first-time apply + smoke-test + secret rotation + signer-key clarification (STH key is Trillian-internal; DSSE envelope signing stays with signing-service / OpenBao). - services/admin-api: go.mod / go.sum pick up the new transitive deps (trillian + grpc) via the local replace on transparency-log. - services/transparency-log/README.md: Backends table updated; next- steps list no longer carries Trillian-deployment as a follow-up. Deployed to the live cluster from this heartbeat: MySQL + LogServer + LogSigner running in agent-trust, tree id 8482455225499059043, service smoke-tested end-to-end (201 on first POST, 200 on idempotent repeat, 409 on content_hash mismatch, tree_size=2 with correct audit path). Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-25 18:39:44 +00:00
3. Image-Build-Pipeline (CI/CD): Push von `services/transparency-log/Dockerfile`
nach `harbor.basicstack.de/proofedmesh/transparency-log:<tag>` automatisieren
und `deploy/trillian/40-transparency-log.yaml` auf diese Tags anpassen.
## Prod-Deployment
Siehe [`deploy/trillian/README.md`](../../deploy/trillian/README.md) —
Trillian (MySQL + LogServer + LogSigner + Tree) läuft im Namespace
`agent-trust` auf dem BasicStack-Cluster (DEV-604).