basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

Merge branch 'main' of https://forgejo.basicstack.de/basicstack/stack.basicstack.de

Browse source
This commit is contained in:
al66 2026-10-24 11:09:14 +02:00
commit 221bd6b473
215 changed files with 13493 additions and 34360 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

48
DEV-349-progress.md Normal file
View file

@ -0,0 +1,48 @@
## Dozzle Deployment Progress
### Completed
1. ✅ Created deployment manifests in `apps/dozzle/` directory:
- Namespace, ServiceAccount with cluster-wide RBAC for pod log access
- PVC (1Gi, hcloud-volumes-encrypted, ReadWriteOnce with Recreate strategy)
- Deployment with oauth2-proxy sidecar for Pocket ID OIDC authentication
- Service routing to oauth2-proxy (port 4180)
- Ingress with TLS for `dozzle.basicstack.de`
2. ✅ Created ArgoCD application (`app-dozzle.yaml`) with automated sync
3. ✅ Created Pocket ID OIDC client:
- Client ID: `179c13f2-d251-4e1e-b1a0-c070df350c4e`
- Callback URL: `https://dozzle.basicstack.de/oauth2/callback`
- Sealed secret with client credentials and oauth2-proxy cookie secret
4. ✅ Configured oauth2-proxy sidecar (running successfully)
5. ✅ All manifests committed and pushed to repository
### Current Blocker
Dozzle container is failing to start with error: **"Could not connect to any Docker Engine"**
**Root cause**: Dozzle v10.6.10 is not auto-detecting Kubernetes mode despite:
- Proper RBAC permissions (ClusterRole with pods, pods/log, namespaces access)
- Service account correctly mounted at `/var/run/secrets/kubernetes.io/serviceaccount/`
- ClusterRoleBinding connecting service account to ClusterRole
**Troubleshooting attempted**:
- Tried various command-line flags (`--agent`, `--no-analytics`)
- Verified RBAC permissions are correct
- Checked service account token is mounted
- oauth2-proxy sidecar is working correctly (1/2 containers ready)
**Next steps**:
1. Research Dozzle Kubernetes deployment examples/Helm charts for missing configuration
2. Consider alternative: Try older Dozzle version or different container log viewer
3. May need to explicitly configure Kubernetes API endpoint or use different deployment approach
**Current pod status**:
```
dozzle-f4d557466-qh28j 1/2 CrashLoopBackOff
```
All infrastructure is in place; just need to resolve Dozzle's Kubernetes detection issue.

19
apps/app-argocd.yaml
View file

@ -3,16 +3,27 @@ kind: Application
metadata: metadata:
name: argocd name: argocd
namespace: argocd namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec: spec:
project: default project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/argocd
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
namespace: argocd namespace: argocd
sources:
- repoURL: https://argoproj.github.io/argo-helm
chart: argo-cd
targetRevision: 10.4.0
helm:
releaseName: argocd
valueFiles:
- $values/apps/argocd/values.yaml
- repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/argocd
ref: values
syncPolicy: syncPolicy:
syncOptions: syncOptions:
- CreateNamespace=true - CreateNamespace=true
- ServerSideApply=true - ServerSideApply=true
- ApplyOutOfSyncOnly=true

4
apps/app-basicstack-org.yaml
View file

@ -6,9 +6,9 @@ metadata:
spec: spec:
project: default project: default
source: source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/basicstack.org.git repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main targetRevision: main
path: k8s path: apps/basicstack-org
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
namespace: basicstack-web namespace: basicstack-web

20
apps/app-dozzle.yaml Normal file
View file

@ -0,0 +1,20 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: dozzle
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/dozzle
destination:
server: https://kubernetes.default.svc
namespace: dozzle
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

18
apps/app-forgejo-runner.yaml Normal file
View file

@ -0,0 +1,18 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: forgejo-runner
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/forgejo-runner
destination:
server: https://kubernetes.default.svc
namespace: forgejo
syncPolicy:
automated:
prune: true
selfHeal: true

9
apps/app-backup.yaml → apps/app-grist.yaml
View file

@ -1,17 +1,20 @@
apiVersion: argoproj.io/v1alpha1 apiVersion: argoproj.io/v1alpha1
kind: Application kind: Application
metadata: metadata:
name: backup name: grist
namespace: argocd namespace: argocd
spec: spec:
project: default project: default
source: source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main targetRevision: main
path: apps/backup path: apps/grist
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
namespace: backup namespace: grist
syncPolicy: syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions: syncOptions:
- CreateNamespace=true - CreateNamespace=true

20
apps/app-harbor.yaml Normal file
View file

@ -0,0 +1,20 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: harbor
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/harbor
destination:
server: https://kubernetes.default.svc
namespace: harbor
syncPolicy:
automated:
prune: false
selfHeal: true
syncOptions:
- CreateNamespace=true

9
apps/app-platform-prod.yaml → apps/app-kaneo.yaml
View file

@ -1,17 +1,20 @@
apiVersion: argoproj.io/v1alpha1 apiVersion: argoproj.io/v1alpha1
kind: Application kind: Application
metadata: metadata:
name: platform-prod name: kaneo
namespace: argocd namespace: argocd
spec: spec:
project: default project: default
source: source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main targetRevision: main
path: apps/platform-prod path: apps/kaneo
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
namespace: platform-prod namespace: kaneo
syncPolicy: syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions: syncOptions:
- CreateNamespace=true - CreateNamespace=true

20
apps/app-openwebui.yaml Normal file
View file

@ -0,0 +1,20 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: openwebui
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/openwebui
destination:
server: https://kubernetes.default.svc
namespace: openwebui
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

20
apps/app-outline.yaml Normal file
View file

@ -0,0 +1,20 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: outline
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/outline
destination:
server: https://kubernetes.default.svc
namespace: outline
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

20
apps/app-pangolin.yaml Normal file
View file

@ -0,0 +1,20 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: pangolin
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/pangolin
destination:
server: https://kubernetes.default.svc
namespace: pangolin
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

23
apps/app-velero-monitoring.yaml Normal file
View file

@ -0,0 +1,23 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: velero-monitoring
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: velero
directory:
recurse: false
include: '{servicemonitor.yaml,prometheusrule.yaml}'
destination:
server: https://kubernetes.default.svc
namespace: velero
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true

23
apps/app-velero-restore-drill.yaml Normal file
View file

@ -0,0 +1,23 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: velero-restore-drill
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: velero/restore-drill
directory:
recurse: false
include: '*.yaml'
destination:
server: https://kubernetes.default.svc
namespace: velero
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true

23
apps/app-velero-schedules.yaml Normal file
View file

@ -0,0 +1,23 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: velero-schedules
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: velero/schedules
directory:
recurse: false
include: '*.yaml'
destination:
server: https://kubernetes.default.svc
namespace: velero
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true

21
apps/app-velero.yaml Normal file
View file

@ -0,0 +1,21 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: velero
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/velero
destination:
server: https://kubernetes.default.svc
namespace: velero
syncPolicy:
automated:
prune: false
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true

20
apps/app-vikunja.yaml Normal file
View file

@ -0,0 +1,20 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: vikunja
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/vikunja
destination:
server: https://kubernetes.default.svc
namespace: vikunja
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

99
apps/argocd/README.md
View file

@ -2,69 +2,83 @@
This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster. This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster.
Argo CD itself is installed from the community Helm chart (`argoproj/argo-helm`, chart `argo-cd`). This directory holds the chart values file plus a small kustomize wrapper for the ingress and sealed secrets that stay in git.
## Files ## Files
- `argocd-install.yaml` - Auto-generated Argo CD installation manifest (DO NOT EDIT DIRECTLY) - `values.yaml` - Helm values for the `argo-cd` chart (image tag, OIDC/RBAC config, resource limits, ingress disabled).
- `kustomization.yaml` - Kustomize overlay that adds resource limits and other customizations - `kustomization.yaml` - Kustomize wrapper for the ingress + sealed secrets (does NOT install Argo CD itself).
- `argocd-ingress.yaml` - Ingress configuration for Argo CD UI - `argocd-ingress.yaml` - Ingress configuration for the Argo CD UI (Traefik + cert-manager `letsencrypt-prod`).
- `argocd-oidc-secret-sealed.yaml` - Sealed secret for OIDC integration - `argocd-oidc-secret-sealed.yaml` - Sealed secret for Pocket ID OIDC integration.
- `repo-*.yaml` - Sealed secrets for Git repository access - `repo-*.yaml` - Sealed secrets for Git repository access.
## Resource Limits ## How the install is wired
**IMPORTANT**: Resource limits were added after DEV-281 (resource exhaustion incident on 2026-07-12). The root [`../app-argocd.yaml`](../app-argocd.yaml) is an Argo CD `Application` with two sources:
All Argo CD components now have memory limits to prevent OOM incidents: 1. The public Helm chart at `https://argoproj.github.io/argo-helm`, chart `argo-cd`, `targetRevision` pinned in git.
2. This repo (`ref: values`) providing the `values.yaml` used by source (1) AND the ingress/sealed secrets applied via `kustomize`.
Once bootstrapped, Argo CD manages its own install by syncing this Application.
## Resource limits
Memory limits were added after DEV-281 (resource exhaustion incident on 2026-07-12) and are now driven by `values.yaml`:
| Component | Memory Limit | Memory Request | | Component | Memory Limit | Memory Request |
|-----------|--------------|----------------| |-----------|--------------|----------------|
| application-controller | 512Mi | 256Mi | | application-controller | 1Gi | 512Mi |
| repo-server | 512Mi | 256Mi | | repo-server | 512Mi | 256Mi |
| redis | 256Mi | 128Mi | | redis | 256Mi | 128Mi |
| server | 256Mi | 128Mi | | server | 256Mi | 128Mi |
| notifications-controller | 128Mi | 64Mi | | notifications-controller | 128Mi | 64Mi |
| applicationset-controller | 256Mi | 128Mi | | applicationset-controller | 256Mi | 128Mi |
These limits are based on observed usage patterns and provide headroom while preventing unlimited memory consumption.
## Deployment ## Deployment
### Option 1: Apply with kustomize (RECOMMENDED) ### Steady state (managed by Argo CD)
Once the cluster is bootstrapped, changes to this directory are picked up by the root `argocd` Application on the next sync. No manual `kubectl apply` is required.
### First-time / disaster-recovery bootstrap
Argo CD cannot install itself while it is gone. Bootstrap with helm, then hand ownership back:
```bash ```bash
kubectl apply -k apps/argocd/ helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd \
--version 10.4.0 \
--namespace argocd --create-namespace \
--values apps/argocd/values.yaml \
--wait --timeout 10m
kubectl apply -k apps/argocd/ # ingress + sealed secrets
kubectl apply -f apps/app-argocd.yaml # hand ownership back to GitOps
``` ```
This will apply the base manifests plus all patches defined in `kustomization.yaml`.
### Option 2: Direct apply (not recommended)
```bash
kubectl apply -f apps/argocd/argocd-install.yaml
kubectl apply -f apps/argocd/argocd-ingress.yaml
# etc.
```
**Note**: This skips the resource limit patches and is NOT recommended.
## Updating Argo CD ## Updating Argo CD
When updating to a new Argo CD version: Bump the chart and the image tag in a single PR:
1. Download the new install manifest: 1. Refresh the local helm repo cache and check what's available:
```bash ```bash
curl -sSL https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml > argocd-install.yaml helm repo update
helm search repo argo/argo-cd --versions | head
``` ```
2. Apply with kustomize (resource limits will be automatically applied): 2. Bump both fields together, keeping them in sync with the chart's `appVersion`:
- `apps/app-argocd.yaml` -> `spec.sources[0].targetRevision` (chart version, e.g. `10.4.0`)
- `apps/argocd/values.yaml` -> `global.image.tag` (app version, e.g. `v3.5.1`)
3. Optionally render locally to sanity-check the output before opening the PR:
```bash ```bash
kubectl apply -k apps/argocd/ helm template argocd argo/argo-cd \
--version <new-chart-version> \
-f apps/argocd/values.yaml -n argocd | less
``` ```
3. Verify resource limits are in place: 4. Open the PR. After merge, Argo CD syncs itself onto the new version.
```bash
kubectl get statefulset,deployment -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'
```
## Troubleshooting ## Troubleshooting
@ -74,22 +88,21 @@ When updating to a new Argo CD version:
kubectl top pods -n argocd kubectl top pods -n argocd
``` ```
### Check if resource limits are applied ### Check applied resource limits
```bash ```bash
kubectl get deployment,statefulset -n argocd -o json | jq '.items[] | {name: .metadata.name, limits: .spec.template.spec.containers[0].resources.limits}' kubectl get deployment,statefulset -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'
``` ```
### Rollback if needed ### Rollback if a chart upgrade misbehaves
If there are issues after applying resource limits:
```bash ```bash
# Remove limits from a specific component helm -n argocd history argocd
kubectl patch deployment -n argocd argocd-server --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/resources"}]' helm -n argocd rollback argocd <previous-revision>
``` ```
## History ## History
- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281) - **2026-08-23**: Switched to the community Helm chart, bumped to `v3.5.1` / chart `10.4.0` (DEV-519).
- **2026-07-11**: Initial deployment - **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281).
- **2026-07-11**: Initial deployment (vendored `install.yaml`).

33364
apps/argocd/argocd-install.yaml
View file

File diff suppressed because it is too large Load diff

2
apps/argocd/argocd-oidc-secret-sealed.yaml
View file

@ -16,4 +16,6 @@ spec:
creationTimestamp: null creationTimestamp: null
name: argocd-oidc-secret name: argocd-oidc-secret
namespace: argocd namespace: argocd
labels:
app.kubernetes.io/part-of: argocd
type: Opaque type: Opaque

75
apps/argocd/kustomization.yaml
View file

@ -4,82 +4,7 @@ kind: Kustomization
namespace: argocd namespace: argocd
resources: resources:
- argocd-install.yaml
- argocd-ingress.yaml - argocd-ingress.yaml
- argocd-oidc-secret-sealed.yaml - argocd-oidc-secret-sealed.yaml
- repo-basicstack-org-secret-sealed.yaml - repo-basicstack-org-secret-sealed.yaml
- repo-stack-basicstack-de-secret-sealed.yaml - repo-stack-basicstack-de-secret-sealed.yaml
patches:
# Add memory limits to prevent OOM incidents (DEV-281)
- target:
kind: StatefulSet
name: argocd-application-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 512Mi
requests:
memory: 256Mi
- target:
kind: Deployment
name: argocd-repo-server
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 512Mi
requests:
memory: 256Mi
- target:
kind: Deployment
name: argocd-redis
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 256Mi
requests:
memory: 128Mi
- target:
kind: Deployment
name: argocd-server
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 256Mi
requests:
memory: 128Mi
- target:
kind: Deployment
name: argocd-notifications-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 128Mi
requests:
memory: 64Mi
- target:
kind: Deployment
name: argocd-applicationset-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 256Mi
requests:
memory: 128Mi

87
apps/argocd/values.yaml Normal file
View file

@ -0,0 +1,87 @@
global:
image:
tag: v3.5.1
configs:
cm:
url: https://argo.basicstack.de
application.instanceLabelKey: argocd.argoproj.io/instance
resource.exclusions: |
- apiGroups: [cilium.io]
kinds: [CiliumIdentity, CiliumEndpoint, CiliumEndpointSlice]
- apiGroups: [kyverno.io, reports.kyverno.io, wgpolicyk8s.io]
kinds: [PolicyReport, ClusterPolicyReport, EphemeralReport,
ClusterEphemeralReport, AdmissionReport, ClusterAdmissionReport,
BackgroundScanReport, ClusterBackgroundScanReport, UpdateRequest]
oidc.config: |
name: Pocket ID
issuer: https://auth.basicstack.de
clientID: $argocd-oidc-secret:oidc.pocketid.clientId
clientSecret: $argocd-oidc-secret:oidc.pocketid.clientSecret
requestedScopes: [openid, profile, email, groups]
requestedIDTokenClaims:
groups: {essential: true}
rbac:
policy.default: role:readonly
policy.csv: |
g, argo_admins, role:admin
p, role:admin, applications, *, */*, allow
p, role:admin, clusters, *, *, allow
p, role:admin, repositories, *, *, allow
p, role:admin, projects, *, *, allow
p, role:admin, accounts, *, *, allow
p, role:admin, gpgkeys, *, *, allow
p, role:admin, certificates, *, *, allow
p, role:admin, exec, *, *, allow
params:
server.insecure: "true"
ssh:
extraHosts: |
forgejo.forgejo.svc.cluster.local ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCnFlDTGVmri7geuOfZ1UvHzCRi1U5BqCq+9MwkrbGkZCmBmUt3ek95JBZzurLvR/0rE624B9tB1AvSJIEIQ1W1YXD4ydiDaHX6J0sip6lEfqhREx0y91C15zHgi9jN0UKidse1g0xlLHpwePUmz8/2BJcLHJuwbZSdUu7+uhDIYtIJ5+3trX3IABNeluRA5TkspPAC0ViEaz4saWgWQWjKL8dsb3jIR94DiWAVpQnhaCBEILkIStJrmWl0O5B10Jr3KWy18szr9UVd8edCkoEXriCR8gx69jHdmuem5WlZPGvsK5Adf0mXE8S3rdyHqWOkDRs6Wlwd9p/1nDY8c8wtE3vqhefCpt1BwpTys9PMmgUfG0at/W+NdvalRqgQz26Bso8Tf7hcfyA/B69U2pvjA0tbgdIGJ5kLeCzpz9kpCYn8wSIuNIJ86BpyfnjRYFCYJVc6Ls86i8j3fEZAjEX7bmbeDBHQnyyH+rjq+Llo8aUd2Uf7HoSA93EjeuWq/Ta+YbWEWrp9Mrd48jnypxHXsiwDzqkm+YGQbXHfasiarxji/eQ5UMMG8hCxpLp1lJLhGN2th4eCLkpwchFr5jZGWgyZCth1WzQGj7NHvDTxKRU6n4MfsEX1B6GF0D60qtM5vZynpmO902mkn6wtxo+pCkDMroj668a62zw3rSS5Bw==
controller:
resources:
requests:
memory: 512Mi
limits:
memory: 1Gi
repoServer:
resources:
requests:
memory: 256Mi
limits:
memory: 512Mi
server:
resources:
requests:
memory: 128Mi
limits:
memory: 256Mi
ingress:
enabled: false
redis:
resources:
requests:
memory: 128Mi
limits:
memory: 256Mi
notifications:
resources:
requests:
memory: 64Mi
limits:
memory: 128Mi
applicationSet:
resources:
requests:
memory: 128Mi
limits:
memory: 256Mi

80
apps/backup/forgejo-backup-cronjob.yaml
View file

@ -1,80 +0,0 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: forgejo-backup
namespace: backup
labels:
app: forgejo-backup
spec:
schedule: "0 3 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 7
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: forgejo-backup
image: postgres:16-alpine
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -lc
- |
set -euo pipefail
ts=$(date -u +%Y%m%dT%H%M%SZ)
BACKUP_DIR="/data/forgejo-${ts}"
mkdir -p "${BACKUP_DIR}"
echo "[${ts}] Starting Forgejo backup..."
# Backup PostgreSQL database
echo "[$(date -u +%H:%M:%S)] Dumping PostgreSQL database..."
PGPASSWORD="${POSTGRES_PASSWORD}" pg_dump \
-h forgejo-postgres.forgejo.svc.cluster.local \
-U "${POSTGRES_USER}" \
-d "${POSTGRES_DB}" \
-F c \
-f "${BACKUP_DIR}/forgejo-db.dump"
echo "[$(date -u +%H:%M:%S)] Database backup complete: $(ls -lh ${BACKUP_DIR}/forgejo-db.dump | awk '{print $5}')"
# Create backup manifest
echo "backup_timestamp=${ts}" > "${BACKUP_DIR}/manifest.txt"
echo "type=postgresql_custom_dump" >> "${BACKUP_DIR}/manifest.txt"
echo "database=forgejo" >> "${BACKUP_DIR}/manifest.txt"
echo "restore_cmd=PGPASSWORD=<pass> pg_restore -h <host> -U forgejo -d forgejo -F c forgejo-db.dump" >> "${BACKUP_DIR}/manifest.txt"
# Cleanup old backups (keep 14 days)
find /data -maxdepth 1 -type d -name "forgejo-*" -mtime +14 -exec rm -rf {} + 2>/dev/null || true
echo "[$(date -u +%H:%M:%S)] Backup complete. Files:"
ls -lh "${BACKUP_DIR}/"
echo "[$(date -u +%H:%M:%S)] All backups in storage:"
ls -1 /data | grep "forgejo-"
env:
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: forgejo-backup-secret
key: POSTGRES_USER
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: forgejo-backup-secret
key: POSTGRES_PASSWORD
- name: POSTGRES_DB
valueFrom:
secretKeyRef:
name: forgejo-backup-secret
key: POSTGRES_DB
volumeMounts:
- name: backup-data
mountPath: /data
volumes:
- name: backup-data
persistentVolumeClaim:
claimName: platform-backup-data

18
apps/backup/forgejo-backup-secret-sealed.yaml
View file

@ -1,18 +0,0 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: forgejo-backup-secret
namespace: backup
spec:
encryptedData:
POSTGRES_DB: AgBAaN4x6Izo1/jx7eeNHiDefYNqm+Mh482poUIxw+TgBlLloMDJ7biKazDiNUR+Ym7DniMjroVRjMc3Fi4hiqhcDBt06tuiKXe3qX2lEDwkZbe4mZ1VeqMgSqKbM0wY98y9XWN+G3JDHs5GYQsfJDXwQ9+QzOtNGByQw2Pi2jlov3xAo0vqZLucXPmzhhCDzR49CnaouR200U95vhXZesHF4un3qSFzmi5Ty2JGMd8gKo57OOkWs2GysAoMwDuAv5337aBkNicZK4M0ZtlyMS1YHKnHyxCeghAZPvrWTwLgy1mjP9pcZ+cFQB5qN8wIvXDh9URa6xhUB2QQB6+/1v7kFNisLHs1lTGjYnHPvdyXvFF+IIPfS0eqdz2X2aVRJCkBbiJgwTj1fBiBxdEZ/UP7qJUlemGWMn/lWiThubx3pMuqGHEKnEaCo6sC4GKneFkj6isIrmW6oVc9Xwj9QwL0ELglntteFp6/RSKkCgfHwt2BJQVMMRO7KSD3p1aFUy9bhOcecOrlVGWrEyKPZTTfCySLNi+Eg+3Pz3A2heiYid4X4Y4fQOOWYNz2nO5w2/aVvUsLXA7LbPp/7hg9BPDKmsgr48i2nJXzK9Gjt1cQuSiLy3kwQpry3nwPYclTdwVJTjprdRf0vAODq1ohaXfKoYOu/bvfKzWImqjWo7kB33qkQhT6xVocwadDrgmQlecxcM5eIInh
POSTGRES_PASSWORD: AgB9KzB//Le/OMAum6LXDjEw8bH4jaa149ROYUhe6aG2zxVsFJDK55xAxGCCCOt1dpxV/EtzfzQjbf/CCs37D6jnwqEjlE/yhIKTwRN43ns5MJ6UXoGZVRWbw/yWSR1Cb6Z7P1hPg/wBp9mImX2lU8fDqNLYEN5Mze56UBH12peFBGBWRdtFPUqXmVfxZ2nmgBP9sb8KcxP6iMMFcGf5eoBz0Yta2iha4DyMu2cD4lGUfc6ZN5VREAwyOcqgUfTlHmpz6BWBXIQR0hxE1wHDrjyWouxZT9uvjuKalTtE9cEwkZV8qKz8S9Ba7Ms7uMqOnJ4Cm3ODDvmEAlcgP3mYlaoCjhBA4LmVM72cv6wA6xg4qCunbJDM+5rB8RetXPDfhV4LcYZe6QDGj1As7ztoOdNhdvZQwICoH4660xnxLUYPRN1UGKyoMNHrxZuzMWxDgULRyrl7TDy+bKJHv74L9VOSjJnKw8ov7cuomrqnUdrq6eawQquxnJgY0jv4lDs1Xn/5Gq+72TBSHOS5k6E6p7aIfG0QeilOVQK+OetGOyCCEaTH7Zo7URpjHnY+571RSdV86hX+qXPZBwjtHnwjgi5HiGn41k7/R3OrtdTj8lnftn246YGaYlIZ1gPY7jUQKYzMxhthXSow6uBA8IBSBmBL3tS8D9qLfr9yXoJ5CwIlDx2P5XyzwQU0NaPWb0Um+XnxteJRSH4bOSM+O84T9n3oXNqAPWpUlJMkIW5+zaCxHxOiWw3ZJWX1UZRrzw==
POSTGRES_USER: AgBtQxQlKNvZbArIy4GTQd/J88Z6ShZoamMMw7sDSXsm0UwnToY8WKJflG+ydG3zzNCIfT1S2bcb0XVZbiDEgWySwct/CS6KtafLJqElJ9e65i4EOsdklBiUssjsceWC8yRoUIqrblwP6vqwrwhkdI46QaIcHJkdNmq/giT3yQqx35Mv/slJvoLwom76bZgs51jm22VjEAzcPA0d9rbe9flKXVGf6OmQpViMiV8OSDUgQ2Yjx6rQ5zFvCUdvoX6UIFxt9xmA75YiFgltxElOAWLRMa/CvdA6ZIY9jy1oD/gDIPjCL9pm4u5wM24mkQWMCfZiHAsDfkrxkMOsDlI8fRvFO4tMZgDnFNhi1EFOR1ga1WiK4NRKprKQ5WN0NjSuh59DxGSuKrg0WQjg88oXUdfYg7d9oqZiL7WUWVMWwtyLyW1wKuVR8qIoPxYcY88BaEOtw7tQ8odNszWBigsXCeHP9zVpVQjdPhDyGiN38MqftqQS1ywj/oLHWBC2UbH09I1VuAokiUl92CBAt7EiuvyVyYBv0yZlFT6ReatPeUfCWfAWmXkTg6iOiW+Y4+9HPuhBVjhyUw20WifEajNAbva8azSyc4S1LIIOpIlvoQp9joG7a65yRFjt1KzFoh4KDg+WJC1MNtBgqvMBS/4jOiBcrYh2lKIgGO/kBznPqVlZCDn8tdJyxGth56Td/QGfLdCPOxea1g8q
template:
metadata:
creationTimestamp: null
name: forgejo-backup-secret
namespace: backup
type: Opaque

34
apps/basicstack-org/README.md Normal file
View file

@ -0,0 +1,34 @@
# BasicStack.org Web Application
This directory contains the Kubernetes deployment manifests for the BasicStack.org frontend website.
## Architecture
- **Application**: Nuxt 3 + Vue 3 frontend
- **Container Registry**: Harbor at `10.106.73.119:5000`
- **Image Name**: `basicstack-web`
- **Namespace**: `basicstack-web`
## Components
- `configmap.yaml` - Environment configuration
- `deployment.yaml` - Application deployment and service
- `ingress.yaml` - Traefik ingress rules for basicstack.org and www.basicstack.org
## CI/CD
The application is built and pushed to Harbor by the Forgejo Actions workflow in the [basicstack.org repository](https://forgejo.basicstack.de/basicstack/basicstack.org).
The workflow:
1. Builds the Docker image from the Dockerfile
2. Pushes to Harbor registry at `10.106.73.119:5000/basicstack-web:latest`
3. Argo CD automatically syncs the deployment when new images are available
## Image Pull Policy
The deployment uses `imagePullPolicy: Always` to ensure the latest image is pulled on each pod restart, enabling continuous deployment of new builds.
## Domains
- Primary: https://basicstack.org
- Redirect: https://www.basicstack.org → https://basicstack.org

9
apps/basicstack-org/configmap.yaml Normal file
View file

@ -0,0 +1,9 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: basicstack-web-config
namespace: basicstack-web
data:
DIRECTUS_URL: "https://directus.basicstack.de"
SITE_URL: "https://basicstack.org"
NODE_ENV: "production"

64
apps/basicstack-org/deployment.yaml Normal file
View file

@ -0,0 +1,64 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: basicstack-web
namespace: basicstack-web
labels:
app: basicstack-web
spec:
replicas: 2
selector:
matchLabels:
app: basicstack-web
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
metadata:
labels:
app: basicstack-web
spec:
containers:
- name: basicstack-web
image: harbor.basicstack.de/basicstack/basicstack-web:latest
imagePullPolicy: Always
ports:
- containerPort: 3000
name: http
envFrom:
- configMapRef:
name: basicstack-web-config
resources:
requests:
memory: "128Mi"
cpu: "50m"
limits:
memory: "256Mi"
cpu: "200m"
livenessProbe:
httpGet:
path: /
port: 3000
initialDelaySeconds: 15
periodSeconds: 30
readinessProbe:
httpGet:
path: /
port: 3000
initialDelaySeconds: 10
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: basicstack-web
namespace: basicstack-web
spec:
selector:
app: basicstack-web
ports:
- port: 80
targetPort: 3000
name: http

50
apps/basicstack-org/ingress.yaml Normal file
View file

@ -0,0 +1,50 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: basicstack-web
namespace: basicstack-web
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
cert-manager.io/cluster-issuer: letsencrypt-prod
traefik.ingress.kubernetes.io/router.middlewares: basicstack-web-redirect@kubernetescrd
spec:
ingressClassName: traefik
tls:
- hosts:
- basicstack.org
- www.basicstack.org
secretName: basicstack-org-tls
rules:
- host: basicstack.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: basicstack-web
port:
number: 80
- host: www.basicstack.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: basicstack-web
port:
number: 80
---
# Redirect www to non-www
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: redirect
namespace: basicstack-web
spec:
redirectRegex:
regex: "^https://www\\.basicstack\\.org/(.*)"
replacement: "https://basicstack.org/${1}"
permanent: true

6
apps/bookstack/bookstack-deployment.yaml
View file

@ -100,6 +100,8 @@ metadata:
namespace: bookstack namespace: bookstack
spec: spec:
replicas: 1 replicas: 1
strategy:
type: Recreate
selector: selector:
matchLabels: matchLabels:
app: bookstack app: bookstack
@ -201,7 +203,7 @@ spec:
cpu: "1000m" cpu: "1000m"
livenessProbe: livenessProbe:
httpGet: httpGet:
path: / path: /status
port: 80 port: 80
initialDelaySeconds: 60 initialDelaySeconds: 60
periodSeconds: 10 periodSeconds: 10
@ -209,7 +211,7 @@ spec:
failureThreshold: 6 failureThreshold: 6
readinessProbe: readinessProbe:
httpGet: httpGet:
path: / path: /status
port: 80 port: 80
initialDelaySeconds: 30 initialDelaySeconds: 30
periodSeconds: 5 periodSeconds: 5

2
apps/directus/directus-admin-credentials-sealed.yaml
View file

@ -14,4 +14,6 @@ spec:
creationTimestamp: null creationTimestamp: null
name: directus-admin-credentials name: directus-admin-credentials
namespace: directus namespace: directus
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
type: Opaque type: Opaque

2
apps/directus/directus-agent-token-sealed.yaml
View file

@ -13,4 +13,6 @@ spec:
creationTimestamp: null creationTimestamp: null
name: directus-agent-token name: directus-agent-token
namespace: directus namespace: directus
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
type: Opaque type: Opaque

2
apps/directus/directus-app-secrets-sealed.yaml
View file

@ -14,4 +14,6 @@ spec:
creationTimestamp: null creationTimestamp: null
name: directus-app-secrets name: directus-app-secrets
namespace: directus namespace: directus
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
type: Opaque type: Opaque

87
apps/directus/directus-backup-cronjob.yaml Normal file
View file

@ -0,0 +1,87 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: directus-backup
namespace: directus
labels:
app: directus
component: backup
spec:
schedule: "0 2 * * *"
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
metadata:
labels:
app: directus
component: backup
spec:
restartPolicy: OnFailure
containers:
- name: backup
image: postgres:16-alpine
command:
- /bin/sh
- -c
- |
set -e
BACKUP_DIR="/backups/directus"
BACKUP_FILE="${BACKUP_DIR}/directus-backup-$(date +%Y%m%d-%H%M%S).sql.gz"
echo "Starting Directus database backup to $BACKUP_FILE"
# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"
# Perform PostgreSQL dump and compress
PGPASSWORD="$POSTGRES_PASSWORD" pg_dump \
-h directus-postgres \
-U directus \
-d directus \
--clean \
--if-exists \
--no-owner \
--no-privileges \
| gzip > "$BACKUP_FILE"
# Verify backup was created
if [ -f "$BACKUP_FILE" ]; then
SIZE=$(du -h "$BACKUP_FILE" | cut -f1)
echo "Backup completed successfully: $BACKUP_FILE ($SIZE)"
ls -lh "$BACKUP_FILE"
else
echo "ERROR: Backup file was not created"
exit 1
fi
# Clean up backups older than 30 days
echo "Cleaning up backups older than 30 days..."
find "$BACKUP_DIR" -name "directus-backup-*.sql.gz" -type f -mtime +30 -delete
# Show remaining backups
echo "Current backups:"
ls -lh "$BACKUP_DIR" | grep "directus-backup-" || echo "No backups found"
echo "Backup job completed"
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: directus-db-credentials
key: postgres-password
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
volumeMounts:
- name: backup-storage
mountPath: /backups
volumes:
- name: backup-storage
hostPath:
path: /var/backups
type: DirectoryOrCreate

2
apps/directus/directus-db-credentials-sealed.yaml
View file

@ -14,4 +14,6 @@ spec:
creationTimestamp: null creationTimestamp: null
name: directus-db-credentials name: directus-db-credentials
namespace: directus namespace: directus
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
type: Opaque type: Opaque

15
apps/directus/directus-db-pvc.yaml Normal file
View file

@ -0,0 +1,15 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: directus-db
namespace: directus
labels:
app: directus
component: database
spec:
accessModes:
- ReadWriteOnce
storageClassName: hcloud-volumes-encrypted
resources:
requests:
storage: 20Gi

154
apps/directus/directus-deployment.yaml Normal file
View file

@ -0,0 +1,154 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: directus
namespace: directus
labels:
app: directus
component: app
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: directus
component: app
template:
metadata:
labels:
app: directus
component: app
spec:
initContainers:
- name: wait-for-postgres
image: postgres:16-alpine
command:
- sh
- -c
- |
until pg_isready -h directus-postgres -U directus; do
echo "Waiting for PostgreSQL..."
sleep 2
done
securityContext:
fsGroup: 1000
containers:
- name: directus
image: directus/directus:12.3.0
ports:
- name: http
containerPort: 8055
env:
- name: DB_CLIENT
value: pg
- name: DB_HOST
value: directus-postgres
- name: DB_PORT
value: "5432"
- name: DB_DATABASE
value: directus
- name: DB_USER
value: directus
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: directus-db-credentials
key: postgres-password
- name: KEY
valueFrom:
secretKeyRef:
name: directus-app-secrets
key: key
- name: SECRET
valueFrom:
secretKeyRef:
name: directus-app-secrets
key: secret
- name: ADMIN_EMAIL
valueFrom:
secretKeyRef:
name: directus-admin-credentials
key: admin-email
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: directus-admin-credentials
key: admin-password
- name: LICENSE_KEY
valueFrom:
secretKeyRef:
name: directus-license
key: license-key
- name: PUBLIC_URL
value: https://directus.basicstack.de
- name: AUTH_PROVIDERS
value: openid
- name: AUTH_OPENID_DRIVER
value: openid
- name: AUTH_OPENID_CLIENT_ID
valueFrom:
secretKeyRef:
name: directus-oidc-secret
key: client-id
- name: AUTH_OPENID_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: directus-oidc-secret
key: client-secret
- name: AUTH_OPENID_ISSUER_URL
valueFrom:
secretKeyRef:
name: directus-oidc-secret
key: issuer-url
- name: AUTH_OPENID_IDENTIFIER_KEY
value: email
- name: AUTH_OPENID_ALLOW_PUBLIC_REGISTRATION
value: "true"
- name: AUTH_OPENID_DEFAULT_ROLE_ID
value: ""
- name: AUTH_OPENID_ICON
value: account_circle
- name: AUTH_OPENID_LABEL
value: SSO Login
- name: AUTH_OPENID_SCOPE
value: openid email profile
- name: CORS_ENABLED
value: "true"
- name: CORS_ORIGIN
value: "true"
- name: CACHE_ENABLED
value: "false"
- name: RATE_LIMITER_ENABLED
value: "true"
- name: RATE_LIMITER_POINTS
value: "50"
- name: RATE_LIMITER_DURATION
value: "1"
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "1"
memory: 2Gi
livenessProbe:
tcpSocket:
port: 8055
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: 8055
initialDelaySeconds: 30
periodSeconds: 5
timeoutSeconds: 3
volumeMounts:
- name: uploads
mountPath: /directus/uploads
volumes:
- name: uploads
persistentVolumeClaim:
claimName: directus-uploads

26
apps/directus/directus-ingress.yaml Normal file
View file

@ -0,0 +1,26 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: directus
namespace: directus
labels:
app: directus
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: traefik
tls:
- hosts:
- directus.basicstack.de
secretName: directus-tls
rules:
- host: directus.basicstack.de
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: directus
port:
number: 80

17
apps/directus/directus-license-sealed.yaml Normal file
View file

@ -0,0 +1,17 @@
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: directus-license
namespace: directus
spec:
encryptedData:
license-key: AgBnmjrrBIYuOegTmt2AEUSTPXXUcciYx5ubGVyICDKAgLfnCFZgDzjOfnMCEu+aA6zQ3AwW92wWcftHbDzZjxrS2VfW/AprRLgxlSi/OX7sYncaPwo9v2/iFOzC5E+1OuJiZXIfZp1nboXf16Ujck23k4HRHSFk/y3+XM7WQWGg4h6pl7ZacN/in+fu3mYlB8QSkE0eZcsVS+A7WDHGZQzkJfGwVasEBxVfMPuooCvrv0EoenNEofYCz6LZ4dbsI7EnsbGwX3d5KN9pHzEkacCEnhzAhicDMk7nNMY8JvbTgT8gdlfWyFZoI2kBHfVDAO16+BbblTHJ9SYlYcmDHg3v7MMzyyvZ1XLQ8gQAH8imD54S/thLx8EX7X80c0wbtqJ2cDrOy2diKZjdGXsGNgy/+ErDVV7QqOTY/vH4L2hKE1xI1nHIdHaCjA+4zN9SMroCGioXistgcBnr5TkU42POPzrBCUm5eFYptsV1OIgOpMa7mvmN2uq86Pi8o0hNkLzZ2kW8QUhd3+Edesw9+64iACtPM2hdS6+jDZtyv7LdGnShYDu5tciMFs/dElE0UQW6HqaZ7z/PnJlhlx88mIijD2YsFo+M/8JFojf6iLFyWP6TVPK51yf/lqGwK/4S4c9w4MVfhiTrSmlMk9hOkZy1i91p4xnzaWuBAX8U9XEzN4+qJeHlcQRHiSyi+V45ZD5cZTPqOEfLdOyh/sTXXOQaIicWqHzuCzUv5HzeSQ==
template:
metadata:
creationTimestamp: null
name: directus-license
namespace: directus
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
type: Opaque

2
apps/directus/directus-oidc-secret-sealed.yaml
View file

@ -15,4 +15,6 @@ spec:
creationTimestamp: null creationTimestamp: null
name: directus-oidc-secret name: directus-oidc-secret
namespace: directus namespace: directus
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
type: Opaque type: Opaque

70
apps/directus/directus-postgres-deployment.yaml Normal file
View file

@ -0,0 +1,70 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: directus-postgres
namespace: directus
labels:
app: directus
component: database
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: directus
component: database
template:
metadata:
labels:
app: directus
component: database
spec:
containers:
- name: postgres
image: postgres:16-alpine
ports:
- name: postgres
containerPort: 5432
env:
- name: POSTGRES_DB
value: directus
- name: POSTGRES_USER
value: directus
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: directus-db-credentials
key: postgres-password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "1"
memory: 1Gi
livenessProbe:
exec:
command:
- pg_isready
- -U
- directus
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
exec:
command:
- pg_isready
- -U
- directus
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: directus-db

17
apps/directus/directus-postgres-service.yaml Normal file
View file

@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: directus-postgres
namespace: directus
labels:
app: directus
component: database
spec:
type: ClusterIP
selector:
app: directus
component: database
ports:
- name: postgres
port: 5432
targetPort: 5432

17
apps/directus/directus-service.yaml Normal file
View file

@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: directus
namespace: directus
labels:
app: directus
component: app
spec:
type: ClusterIP
selector:
app: directus
component: app
ports:
- name: http
port: 80
targetPort: 8055

15
apps/directus/directus-uploads-pvc.yaml Normal file
View file

@ -0,0 +1,15 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: directus-uploads
namespace: directus
labels:
app: directus
component: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: hcloud-volumes-encrypted
resources:
requests:
storage: 10Gi

44
apps/dozzle/README.md Normal file
View file

@ -0,0 +1,44 @@
# Dozzle Deployment
Dozzle is a real-time log viewer for Docker containers running in the Kubernetes cluster.
## Components
- **Namespace**: `dozzle`
- **Domain**: `dozzle.basicstack.de`
- **Storage**: 1Gi PVC using `hcloud-volumes-encrypted` storage class
- **Authentication**: Pocket ID OIDC via oauth2-proxy sidecar
## Architecture
Dozzle doesn't support native OIDC authentication, so we use oauth2-proxy as a sidecar container:
1. **oauth2-proxy** (port 4180): Handles OIDC authentication with Pocket ID
2. **Dozzle** (port 8080): Receives authenticated requests from oauth2-proxy with user headers
The oauth2-proxy authenticates users via Pocket ID OIDC and forwards authenticated requests to Dozzle with `X-Forwarded-User`, `X-Forwarded-Email`, and `X-Forwarded-Preferred-Username` headers. Dozzle is configured with `forward-proxy` authentication to trust these headers.
## Files
- `namespace.yaml`: Dozzle namespace
- `service-account.yaml`: Service account with RBAC for accessing pod logs cluster-wide
- `pvc.yaml`: Persistent volume claim for Dozzle settings (1Gi, ReadWriteOnce with Recreate strategy)
- `deployment.yaml`: Dozzle deployment with oauth2-proxy sidecar
- `service.yaml`: Kubernetes service (routes to oauth2-proxy port 4180)
- `ingress.yaml`: Traefik ingress with TLS (routes to oauth2-proxy)
- `dozzle-oidc-sealed.yaml`: Sealed secret with OIDC client credentials and oauth2-proxy cookie secret
## Pocket ID OIDC Client
- **Client ID**: `179c13f2-d251-4e1e-b1a0-c070df350c4e`
- **Client Name**: Dozzle
- **Callback URL**: `https://dozzle.basicstack.de/oauth2/callback`
- **Scopes**: `openid profile email`
## Access
After deployment, access Dozzle at https://dozzle.basicstack.de and authenticate with Pocket ID credentials.
## ArgoCD
The application is managed by ArgoCD via `app-dozzle.yaml` in the parent apps directory.

127
apps/dozzle/deployment.yaml Normal file
View file

@ -0,0 +1,127 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dozzle
namespace: dozzle
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: dozzle
template:
metadata:
labels:
app: dozzle
spec:
serviceAccountName: dozzle
containers:
- name: dozzle
image: amir20/dozzle:v10.7.3
ports:
- containerPort: 8080
name: dozzle-http
protocol: TCP
env:
- name: DOZZLE_MODE
value: "k8s"
- name: DOZZLE_LEVEL
value: "info"
- name: DOZZLE_AUTH_PROVIDER
value: "forward-proxy"
- name: DOZZLE_AUTH_HEADER_USER
value: "X-Forwarded-User"
- name: DOZZLE_AUTH_HEADER_EMAIL
value: "X-Forwarded-Email"
- name: DOZZLE_AUTH_HEADER_NAME
value: "X-Forwarded-Preferred-Username"
- name: DOZZLE_NO_ANALYTICS
value: "true"
startupProbe:
httpGet:
path: /healthcheck
port: 8080
initialDelaySeconds: 10
periodSeconds: 3
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 20
readinessProbe:
httpGet:
path: /healthcheck
port: 8080
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
livenessProbe:
httpGet:
path: /healthcheck
port: 8080
periodSeconds: 30
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
volumeMounts:
- name: dozzle-data
mountPath: /data
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 250m
memory: 256Mi
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:latest
ports:
- containerPort: 4180
name: http
protocol: TCP
args:
- --http-address=0.0.0.0:4180
- --upstream=http://127.0.0.1:8080
- --provider=oidc
- --oidc-issuer-url=https://auth.basicstack.de
- --redirect-url=https://dozzle.basicstack.de/oauth2/callback
- --email-domain=*
- --pass-user-headers=true
- --set-xauthrequest=true
- --pass-access-token=false
- --cookie-secure=true
- --cookie-httponly=true
- --cookie-name=__Host-oauth2-proxy
- --cookie-expire=12h
- --code-challenge-method=S256
- --proxy-websockets=true
- --pass-host-header=true
- --flush-interval=1s
- --upstream-timeout=3600s
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: dozzle-oidc
key: client-id
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: dozzle-oidc
key: client-secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: dozzle-oidc
key: cookie-secret
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 50m
memory: 64Mi
volumes:
- name: dozzle-data
persistentVolumeClaim:
claimName: dozzle-data

20
apps/dozzle/dozzle-oidc-sealed.yaml Normal file
View file

@ -0,0 +1,20 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: dozzle-oidc
namespace: dozzle
spec:
encryptedData:
client-id: AgBNals2yE6cxlM9ZRQBj4iWka8F7ldPlmypB9MphG3ywoDkWpPzqlcKmmg3C98s4FwxGuYEr3CcSH46E5DhBsS5Xm5IZ+pJ27VVSdYaJe/3tkSWGTnB56Oq30Gr1pPLToVSp24rSHQVd2VcNdy/lizFGB757b6zKsR+0yVKNETWdXefhbCfZbKmaznGTYd5weGlH9Gz4x79jP2ZaFAgNLL+QYzpmoFHdQTWycyKYj3Hwy4guKxyN1eqNf/70Vk7RbbpiNX/7nF0s3+FaPi/Lqnc3AzrnrCunlGFvwjiKOX4ORAPGJRe8tTybutVAx9I5eAoNyHSuE0nJvobi6PmT9EhFtldRg60Nuw3amIhCty77eyu3tJobw9fJ9c+7Nt7mpS/+pJyKYfYXmyTUzfSX4zRKd/U0+A0VuVTaYjadbyV7s4A2Z0y3trJ+DZlWC2DqgA38FmY59CYt6ZDgFzXooWrapCSy/dW+yzG9XYz2WZQHlYyGSg3mQA3TZETAbg+lunKwZXl+Byc8Dyz2pwIISsFHHrBsr+B7tbx0pBQTvL9cNNAkvJvJ6F8YG+I+6nw5W5WFgB2CN1md/AEITiNB7tqVnC+/BJXXdNLzeUkcXEwLkH9MDQgZBlJrP6kjUlnm5bWZOckv8xy6ZFkbfqtPCdxCv+AhkNa+gCPmRAt1VeR1NsCP+jfzhm8Cu9j3j1CQaPJvmDDiO/r3bWD1ACBwXULa/98BAsEYlENdnl5vpIswKbE1ag=
client-secret: AgBaOMjTwXmqocYK4Xe/i45ByFCVnUiums3hVVxK7xMk8oV+voIsWQRdW8HF6wPfG7T54Ga5AxYj3wousZ/8Z1TIigQTBTD9DIGZfyTndY38cztyNNB0WPgpA2TQ0VHa0OwPC7+V2Pc+IY/eFa9GghkZA5DBjnT7DnyclJcuQwjvdvgOwYP4xtOZ1+nvUq3FSeN6gAyqUH+aVvkhhUAJ44WxqOdL86QlUiSUJaf+Qr9ABQp2Or+76mjt5FW3mxFrHfzDnmbxVq1WOJub8fHlKFw83D5PVnle4ThbVATn2yO7cxuZc/qJc1EB6pPySEIczv62gvbNzZjodXVTk8F+2WanKgBbETJxVo43G+BUJy3IqO4G+2wAhDVR48wugpsT3MQtPvHmDJLiHov5tBzaFN6d8jQZY8ngU9VuZS+grsByVKyEQU7v1lgdIyNTaaWTEn2ppGWmIO+yB6/m5DRa+K6TZHgpQp7jJQqCHSSIevZ0Q0hor8TmdZurJ+20FQX4/QdS2EaNdLPi+wc49hAmCtwBbj0itaGE4DsnP586G9coqD8YLzyPWmiu2HunqFWdenhb+mnw2pFEF+GYQcN+ZGn5KwDJcQAfkE/4pY1SeHpKQVsLiaFah8iydZvK8sW7Ep5+r9tjTUbu9JU8HlwuWwytAEtXXysP9lLMfHLOrHr0EA2J51vfgZT53vMwctE96fsBOR4AVd8Ns7bj/F+snBq7M8+JqPvD3/++5oRWBQtLGA==
cookie-secret: AgArAJOvo7vgt1Gy0mG+5AajPkwJ/vfyV2U193A8NK4NukB2I59ZkqRgWhmEpedbwAxonxtkLlXb8Vu5bdKgeS5No9JhaGYJt0KDtiYhnEx7ov5MCIB4zxAUa+GJixvCAglw0b6awkk7lGT68w09rwQ1DNWdG66nKFNep/RaZFbjWW61MlaIEJ7xjJ8zg797082IyqrcBM6FeHsgOH1Cl/pGAUqtCIX1Oo9NKrp0Qojj0ETc1kNFcixn1hNDLTBDFn3lmceUCz3MADY+0hJEa9AR0kkH5ulGW5hXRCM402W9CZdux/IOweGvWqfg9Jn7f8lhjEx6aUmcUFlumMZRSvAJIvG3Hncjot0Zk5CPm2RCdJ7nr2bibqXKl/LOh9g1qhSRCSM1i7DggcI0qAuCmhwq1zWqatYOVslk5yBYPW2khTfEe4nqf63AjbcORWuZQ4U6fOc1RFD/rb+ISvKDfcIT58cAsT80+NC+2bXQY2wKLrXVV5sv+EksDzUKCP4g1GPLFiTi8paI5R/o6OWe90AX7DD/7pJvXLiw56OhC7GQapTOYk5l6+/Cw+B7U9CzcTAP8E8+b35lWRHYl3iClSUDVlzGvjDju5r6wUVfA1JaaCayN/IXqpf0Xk5p6xeVSU+Tn8hzthh6Ooip6FY2KSBXqNZ+0wCEpBzXlehxzIUXymOgE1ZNpH3MRso+3yMpKG70zGfcziE7llmWb4QhNq7WzZih1CMZuS7mCvsuUnK7+1wiMBYhesnPp9QVrQ==
template:
metadata:
creationTimestamp: null
name: dozzle-oidc
namespace: dozzle
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
type: Opaque

45
apps/dozzle/ingress.yaml Normal file
View file

@ -0,0 +1,45 @@
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
name: dozzle
namespace: dozzle
spec:
forwardingTimeouts:
dialTimeout: 30s
idleConnTimeout: 3600s
responseHeaderTimeout: 0s
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dozzle
namespace: dozzle
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
# Websocket timeout settings - critical for connection stability
traefik.ingress.kubernetes.io/service.serverstransport: dozzle@kubernetescrd
# Websocket support annotations
traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
traefik.ingress.kubernetes.io/service.sticky.cookie.name: "dozzle_session"
traefik.ingress.kubernetes.io/service.sticky.cookie.httpOnly: "true"
traefik.ingress.kubernetes.io/service.sticky.cookie.secure: "true"
traefik.ingress.kubernetes.io/service.passhostheader: "true"
spec:
ingressClassName: traefik
rules:
- host: dozzle.basicstack.de
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: dozzle
port:
number: 4180
tls:
- hosts:
- dozzle.basicstack.de
secretName: dozzle-tls

4
apps/dozzle/namespace.yaml Normal file
View file

@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: dozzle

12
apps/dozzle/pvc.yaml Normal file
View file

@ -0,0 +1,12 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dozzle-data
namespace: dozzle
spec:
accessModes:
- ReadWriteOnce
storageClassName: hcloud-volumes-encrypted
resources:
requests:
storage: 1Gi

55
apps/dozzle/service-account.yaml Normal file
View file

@ -0,0 +1,55 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: dozzle
namespace: dozzle
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: dozzle
rules:
- apiGroups:
- ""
resources:
- pods
- pods/log
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- metrics.k8s.io
resources:
- pods
verbs:
- get
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dozzle
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dozzle
subjects:
- kind: ServiceAccount
name: dozzle
namespace: dozzle

13
apps/dozzle/service.yaml Normal file
View file

@ -0,0 +1,13 @@
apiVersion: v1
kind: Service
metadata:
name: dozzle
namespace: dozzle
spec:
selector:
app: dozzle
ports:
- name: http
port: 4180
targetPort: 4180
protocol: TCP

103
apps/forgejo-runner/README.md Normal file
View file

@ -0,0 +1,103 @@
# Forgejo Actions Runner
This directory contains the deployment configuration for the Forgejo Actions runner.
## Prerequisites
Before deploying the runner, you need to obtain a registration token from Forgejo.
### Getting the Registration Token
**Option 1: Via Forgejo Admin UI**
1. Log in to https://forgejo.basicstack.de as admin
2. Navigate to Site Administration → Actions → Runners
3. Click "Create new Runner"
4. Copy the registration token
**Option 2: Via API**
```bash
export FORGEJO_TOKEN="your-api-token"
curl -X POST \
-H "Authorization: token $FORGEJO_TOKEN" \
https://forgejo.basicstack.de/api/v1/admin/runners/registration-token
```
### Creating the Secret
Once you have the registration token, create a sealed secret:
```bash
# Create a temporary secret file
kubectl create secret generic forgejo-runner-token \
--from-literal=token='YOUR_REGISTRATION_TOKEN' \
--namespace=forgejo \
--dry-run=client -o yaml > /tmp/runner-token-secret.yaml
# Seal it with kubeseal
kubeseal --format=yaml < /tmp/runner-token-secret.yaml > apps/forgejo-runner/forgejo-runner-token-sealed.yaml
# Clean up
rm /tmp/runner-token-secret.yaml
```
## Deployment
The runner is deployed via Argo CD. After creating the sealed secret, apply the Argo CD application:
```bash
kubectl apply -f apps/app-forgejo-runner.yaml
```
## Runner Configuration
The runner is deployed as a Deployment (single replica) that bind-mounts the
host's `/var/run/docker.sock` to execute job containers. This requires the
Docker Engine (package `docker.io`) to be installed and running on the target
node — see **Node prerequisites** below.
Configuration:
- **Deployment type**: Deployment (replicas=1)
- **Docker execution**: Host Docker socket (`/var/run/docker.sock`)
- **Concurrent jobs**: 2 (configurable via config.yaml)
- **Labels**: ubuntu-latest:docker://node:24-bookworm, ubuntu-22.04:docker://node:24-bookworm
- **Forgejo URL**: https://forgejo.basicstack.de (external URL for proper webhook/API access)
- **Node selector**: `basicstack.de/docker=true` — schedules only on nodes with the
Docker Engine installed. See **Node prerequisites** below.
## Node prerequisites (required)
The runner uses the host's Docker daemon. Every worker node that should be
eligible to run the runner MUST have `docker.io` installed, the `docker`
systemd unit enabled, and be labeled `basicstack.de/docker=true`. Bootstrap a
worker with:
```bash
ssh root@<node>
DEBIAN_FRONTEND=noninteractive apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y docker.io
systemctl enable --now docker
# from the control plane:
kubectl label node <node-name> basicstack.de/docker=true --overwrite
```
Rolling OS updates that reimage a node will remove Docker. Re-run the bootstrap
after any node reimage. Without Docker, the runner pod fails to mount
`/var/run/docker.sock` (hostPath type check for `Socket` fails); the
nodeSelector prevents that scheduling mistake by pinning to labeled nodes.
## Troubleshooting
Check runner logs:
```bash
kubectl logs -n forgejo -l app=forgejo-runner -f
```
Check if runner is registered:
```bash
kubectl exec -n forgejo deployment/forgejo-runner -- forgejo-runner list
```
## Kubernetes native runners
Kubernetes native runner still in discussion / under construction:\\
https://codeberg.org/forgejo/discussions/issues/66

23
apps/forgejo-runner/forgejo-runner-config.yaml Normal file
View file

@ -0,0 +1,23 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: forgejo-runner-config
namespace: forgejo
data:
config.yaml: |
log:
level: debug
runner:
name: k3s-runner-1
capacity: 2
labels:
- "ubuntu-latest:docker://node:24-bookworm"
- "ubuntu-22.04:docker://node:24-bookworm"
timeout: 3h
cache:
enabled: true
dir: /data/cache
container:
network: host
privileged: false
options: "-v /usr/bin/docker:/usr/bin/docker:ro"

85
apps/forgejo-runner/forgejo-runner-deployment.yaml Normal file
View file

@ -0,0 +1,85 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: forgejo-runner
namespace: forgejo
spec:
replicas: 1
selector:
matchLabels:
app: forgejo-runner
template:
metadata:
labels:
app: forgejo-runner
spec:
serviceAccountName: forgejo-runner
nodeSelector:
basicstack.de/docker: "true"
containers:
- name: runner
image: code.forgejo.org/forgejo/runner:4.0.1
imagePullPolicy: IfNotPresent
securityContext:
runAsUser: 0
runAsGroup: 0
command:
- sh
- -c
- |
# Wait for Forgejo to be ready (external URL)
while ! wget -q -O- https://forgejo.basicstack.de/api/healthz > /dev/null 2>&1; do
echo "Waiting for Forgejo to be ready..."
sleep 5
done
# Re-register if the stored address is the internal cluster URL
if [ -f /data/.runner ] && grep -q "svc.cluster.local" /data/.runner; then
echo "Detected internal cluster URL in .runner — forcing re-registration with external URL"
rm /data/.runner
fi
# Register runner if not already registered
if [ ! -f /data/.runner ]; then
forgejo-runner register \
--no-interactive \
--instance https://forgejo.basicstack.de \
--token "$RUNNER_TOKEN" \
--name k3s-runner-1 \
--labels ubuntu-latest:docker://node:24-bookworm,ubuntu-22.04:docker://node:24-bookworm
fi
# Start runner
forgejo-runner daemon --config /etc/forgejo-runner/config.yaml
env:
- name: RUNNER_TOKEN
valueFrom:
secretKeyRef:
name: forgejo-runner-token
key: token
- name: DOCKER_HOST
value: unix:///var/run/docker.sock
volumeMounts:
- name: docker-sock
mountPath: /var/run/docker.sock
- name: runner-data
mountPath: /data
- name: config
mountPath: /etc/forgejo-runner
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "2"
memory: 2Gi
volumes:
- name: docker-sock
hostPath:
path: /var/run/docker.sock
type: Socket
- name: runner-data
emptyDir: {}
- name: config
configMap:
name: forgejo-runner-config

29
apps/forgejo-runner/forgejo-runner-serviceaccount.yaml Normal file
View file

@ -0,0 +1,29 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: forgejo-runner
namespace: forgejo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: forgejo-runner
namespace: forgejo
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: forgejo-runner
namespace: forgejo
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: forgejo-runner
subjects:
- kind: ServiceAccount
name: forgejo-runner
namespace: forgejo

14
apps/forgejo-runner/forgejo-runner-token-sealed.yaml Normal file
View file

@ -0,0 +1,14 @@
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: forgejo-runner-token
namespace: forgejo
spec:
encryptedData:
token: AgAGd2S0U2AzKqKApPmXMECBG9fNkmdJK6YDRCtQ0mxPCk2OFc5rpIh77FMmf+WcnEtTSlMIGkWQXuOvKIvn78ZNEdYKYuIMTYjzsS+LA9X4gUnhPzvaPj9RmO9NbjFC2Uf3rD7M90SEm1jUY+2F0D2mV2qgYUxfuuG6ksBDFswQkcyDUc5JB+ZZnCYBaNiyY+9/3Y1weQVrf2ckG0fgURiw3AgBw6xcR7i1PEw22z6N+bukf/ykPlUKtEn/elWy7yn6qxXP+QVn4BXPBKIgrVkd1Xq3313U+N2pIGHZXSajjs0oF7mdL5g0dX8lRMS5jLABu3oMPxxRkv4H6D4SKHkloYseQRIQCbOcsgHd6YJjXReGB/9iEvja8Btyg2T6cHQ0m6JgLL6il02tH7NEPEjCAa+ztSBMs/LwqnXkXTzKC+39K6LuQZ/sNBwat7jy6nFn2gvwZ7c0b+GmuxbzukggquXg40toRSBDbAms9J21Owh3g9Hym8oCxgd0Kfa9By3xt+ZXRAMe+RlCnXrF0zWnRLjuL5t+4VVxS4a9IKmB0ggXhfWcF2VtAIxA1r8m/Z2qEqEiBarXspjcSPbOEqzyZdq/iy//KJay1w89D0M/AKweVmhaulJ0iZZrDQm3AR/B11GOgz5bB8IvWDrlZTq0XBQM4g9f8qQPx57OC224tFvNHVORom0tocX9fwZyJbWTLM4ELRX+InRcBh1yHUL46P3GH9TQVmwTI4ZRD5SDpC0kDTKz5J9W
template:
metadata:
creationTimestamp: null
name: forgejo-runner-token
namespace: forgejo

204
apps/forgejo/forgejo-backup-cronjob.yaml Normal file
View file

@ -0,0 +1,204 @@
---
# Forgejo pg_dump → restic → Hetzner Object Storage (DEV-514).
#
# Replaces the legacy volume-based backup that wrote pg_dump files to
# an RWO PVC (`platform-backup-data`). The hcloud volume backing that
# PVC was deleted externally on 2026-08-17; rather than re-provision
# the same legacy pattern, this migrates to the restic→S3 pipeline
# already used for loki/grafana/prometheus (DEV-485…DEV-492).
#
# initContainer runs pg_dump -F c into an emptyDir; main container
# runs `restic backup --stdin-from-command` isn't used here because
# the dump has to complete before restic starts (need a proper exit
# code, and the custom-format dump is not resumable). Instead we
# stage the dump on emptyDir and let restic dedup it into
# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/forgejo`.
apiVersion: batch/v1
kind: CronJob
metadata:
name: forgejo-backup
namespace: forgejo
labels:
app: forgejo-backup
backend: restic
spec:
schedule: "0 3 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
metadata:
labels:
app: forgejo-backup
backend: restic
spec:
backoffLimit: 2
activeDeadlineSeconds: 3600
template:
metadata:
labels:
app: forgejo-backup
backend: restic
spec:
restartPolicy: OnFailure
initContainers:
- name: pgdump
image: postgres:16-alpine
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- |
set -eu
echo "=== forgejo pg_dump started at $(date -u +%FT%TZ) ==="
mkdir -p /dump
PGPASSWORD="${POSTGRES_PASSWORD}" pg_dump \
-h forgejo-postgres.forgejo.svc.cluster.local \
-U "${POSTGRES_USER}" \
-d "${POSTGRES_DB}" \
-F c \
-f /dump/forgejo-db.dump
echo "pg_dump size: $(wc -c < /dump/forgejo-db.dump) bytes"
# Manifest lets a restic-restore consumer identify the
# dump without touching the binary.
{
echo "backup_timestamp=$(date -u +%Y%m%dT%H%M%SZ)"
echo "type=postgresql_custom_dump"
echo "database=${POSTGRES_DB}"
echo "restore_cmd=PGPASSWORD=<pass> pg_restore -h <host> -U ${POSTGRES_USER} -d ${POSTGRES_DB} -F c forgejo-db.dump"
} > /dump/manifest.txt
echo "=== forgejo pg_dump finished at $(date -u +%FT%TZ) ==="
env:
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: forgejo-postgres-secret
key: POSTGRES_USER
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: forgejo-postgres-secret
key: POSTGRES_PASSWORD
- name: POSTGRES_DB
valueFrom:
secretKeyRef:
name: forgejo-postgres-secret
key: POSTGRES_DB
volumeMounts:
- name: dump
mountPath: /dump
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 1000m
memory: 512Mi
containers:
- name: restic
image: harbor.basicstack.de/library/restic:0.19.1
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: forgejo-s3-backup
key: access-key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: forgejo-s3-backup
key: secret-key
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef:
name: forgejo-s3-backup
key: restic-password
- name: S3_ENDPOINT
valueFrom:
secretKeyRef:
name: forgejo-s3-backup
key: endpoint
- name: S3_BUCKET
valueFrom:
secretKeyRef:
name: forgejo-s3-backup
key: bucket
- name: RESTIC_REPOSITORY
value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/forgejo"
command:
- /bin/sh
- -c
- |
set -eu
echo "=== backup-forgejo-restic started at $(date -u +%FT%TZ) ==="
echo "Repository: ${RESTIC_REPOSITORY}"
if restic snapshots >/dev/null 2>&1; then
echo "Repo exists, skipping init."
else
echo "Repo missing, initialising..."
restic init
fi
echo "--- restic backup /source ---"
restic backup /source \
--tag forgejo \
--host k3s
echo "--- restic forget/prune ---"
restic forget --tag forgejo \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
--prune
echo "--- restic check --read-data-subset=5% ---"
CHECK_STATUS=0
restic check --read-data-subset=5% || CHECK_STATUS=$?
echo "restic check exit: ${CHECK_STATUS}"
echo "--- restic stats (repo size) ---"
REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
| grep -oE '"total_size":[0-9]+' \
| head -1 \
| cut -d: -f2)
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics, same wiring as
# loki/grafana siblings (DEV-494). Atomic write.
{
echo "backup_forgejo_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
echo "backup_forgejo_timestamp_seconds $(date +%s)"
echo "backup_forgejo_check_status ${CHECK_STATUS}"
echo "restic_repo_size_bytes{repo=\"forgejo\"} ${REPO_SIZE_BYTES}"
} > /metrics/backup_forgejo.prom.tmp
mv /metrics/backup_forgejo.prom.tmp /metrics/backup_forgejo.prom
echo "=== backup-forgejo-restic finished at $(date -u +%FT%TZ) ==="
exit ${CHECK_STATUS}
volumeMounts:
- name: dump
mountPath: /source
readOnly: true
- name: metrics
mountPath: /metrics
- name: cache
mountPath: /root/.cache/restic
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 1500m
memory: 1Gi
volumes:
- name: dump
emptyDir:
sizeLimit: 5Gi
- name: metrics
hostPath:
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
- name: cache
emptyDir: {}

6
apps/forgejo/forgejo-deployment.yaml
View file

@ -5,6 +5,8 @@ metadata:
namespace: forgejo namespace: forgejo
spec: spec:
replicas: 1 replicas: 1
strategy:
type: Recreate
selector: selector:
matchLabels: matchLabels:
app: forgejo app: forgejo
@ -66,6 +68,10 @@ spec:
value: forgejo.basicstack.de value: forgejo.basicstack.de
- name: FORGEJO__service__DISABLE_REGISTRATION - name: FORGEJO__service__DISABLE_REGISTRATION
value: "true" value: "true"
- name: FORGEJO__actions__ENABLED
value: "true"
- name: FORGEJO__actions__DEFAULT_ACTIONS_URL
value: "https://code.forgejo.org"
resources: resources:
requests: requests:
cpu: 250m cpu: 250m

19
apps/forgejo/forgejo-s3-backup-sealed.yaml Normal file
View file

@ -0,0 +1,19 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: forgejo-s3-backup
namespace: forgejo
spec:
encryptedData:
access-key: AgAu/XawJy0uhi3l8vwtd+loShP2ITnbOEZ/9eu6dkypz7l1MmFJhnEGq8Sy1lsg7BKj6epYRI40kDJIAG6KMBD5i7FvLYCvbFuPJaS3yehA8b1ECreHXvx/VYRUZL4K3WtiPO0fXmUNkxAqIcfHOcHtOZ7YcgSSyTw4DaXzngtek4xcTQlisyIVeBgc1dLPY+vHGn10BGkU5R/LnsUQqgGdp6u24Y+m7OrQGqMH8xwtZR7siEEkasu4IWG0QcY+okcMeiCi8mgAF64rdeZnIeEWqRnKHXxwEurtm20NmmCeRR8OgZfWDesgasThsnNSlRySd4EKlP3DMz8Yyluz/LlSkPt24brnP4qKuEoRpCHHOWgOizX8hkkS+Al7Rnpmxv2pt5CGlRw9s/5uPY81PAN9WLIgjs+XIylvaJBSW06nw3v+MjCUr5jfimW9riTLN32cBSurdpNMzlgb5+DJzEvaGNEG+jSjjFj2eSLci23UWySh7oFdmXeBp3kDWCrrNlJOYqJEwAAunTwvtM/1XknW2qJyyJrD7hbWN7SX4Ymb5Q5q18fMGH+IzB/Si1jdBli8pi8B2Gd/jo6RSmhXUheO3xsB2sXYc09GuSBBBBNzr9MsRL3mvUVr22pay+YPcCbwbwtdR0qmrdhEHAychYSJpNGuwGvs0AthRsNGN6hr1/IJqnnSRkGdJPr6k+T3v7XJzouYy+8Dh8r6oPW3hOhHo4HpJw==
bucket: AgAyQVCK+wE0wWxwVjx5DVnI2falTGNzsUrolv1YyoG2Bx6msYe2G00ovxCMJ6/dWws1Q/Sf6yoiFT0IByYflQFhY/ABzXRi7/gFxCWyYU/7yTo7w1Ue0+eEnBKFtkP/ZhIvtfubo6GDDtbmshpp4UtzPq7kB6qvE6oCFyf6awQD2pe/GKX8RZaDx5TCbxPxhedLkYdc7srR/8rFZBnQiXm/2ohtDOwvLA+RVOpnCW/bsVLvLYQmXyhszMv2M3zqXwM9ODYzQ4Y7lIw6T3Hsmel/1hHWJtN8/mlq+Q3Z/NbzhVeEWs8/xbK/ShC9RwZUgBWxAiW04HvwcxVYs60KaJ4oOMLtO4Iko4+iPTNonjra2wmziEpUtsotvtReRjwHdvk1bBn3q6uqzNaJFv9wLc/RVCvKSpqE5KOxmdbke9EDUs4wQHVQCAfFfwnzzpcF/Rf6ML3TUWc+U0W6UjCfVTWwROznpi0E18tA07apx+xm1y+MyuVYnSpZv++vXetnCjA+KFvqZSon/WnKQrmSO2RFR07FcMs6gQ+EOSaRXBPCnEcnF2IJUbJLdFsgpQbzYG6zo3m2KmxN7gaGrTUdhNg/N7LklnNbJML9KRa7e2Rv8t4NGoigOpZG6+soIijZOSfEJF6owN6MP3oecgUgNsnQ5ftKK/RznNaDtIeZ5KByHlQ7CsZ0MIkHBmoEkDt64xWNEoW4NrWPSxas6pOQGZMPMA==
endpoint: AgBhXqp3TdUU6GxsexRIoeHNwjvxnd2v4Any9WGs7gtyZvfdGIuWHIQ+JpsuRDrEq+BvTaX49D0QYoqnoQ6ZgmmGUI7kJxc+Ombcqo/8PAsOmzfdUaz1NcLAeR/UTGu8BDxAQcySSFax2dyK/zkY3jcsmuUCJNJLB57EZwtwx4fyz+ota+zNBE7pY+sDpWkXKHHtWrvNkD7tSLYHQuS3uzATidRixngL+iaH8rQX3ZK6KaK4H4ZdFG3/1Fw0wjLt8+yVBntBnh/bCL9/JLWQWIiHtooQnKXDdEwYQHWhrktti1KDzy/S++o4XVcRG1alzsQ0vV7uzSZAh4xCmnzximJxeJKawzFXRgFUcoEPrOI/nVedYrAU0DzkuwPdqtQDi/fGoqzZPx3PYhB9wv2PJsLg20ZlFcfHYUCad3O6OmO+VlrpYYcUe4oStkOygrmK5bv89fQ4Tov7BB14+RQIo9ogKmLzZu3xStqIVK+/Y8OUC64zyo7FpPMoYKtRW5oawu8pxo7Gtb+27BMMXQBiMEPNf38C4aj3GGJOvm4CFxVHV510/7o2hSW2ITt5X2RzNOEzULJOHD3DvO8wdWzvpBykRFnZ8JTYv77Kbx7bd9Ylx62VXm2a+Bd0K1Nc/sNJE63pglVhdl05oLYsUTrcSsNoJ6Dz3pMKeue3pYrn9KNurG0s+BFfO6I2Aa4/L8iAggXBzf9G1ALSkn7A30C9rMeyV+/1n+q8GpdPqq4ltqLOHVwIZw==
restic-password: AgB1JGFnSDe3N5ZaiT9xShfzUQiyDtGwH2wotYO5Ekk9nqAbS4xTNBXu/1Z1+cB8zVUP80hrPHHbVXjNu47AcLI+Bn8Ozn9fKIZFsnVHXqicG0Y+WHRMpQxuMijVEMl48oUNE+vw+xRhYTzibVWoxur+VEzyPCn8rLUX07tcNhY+jYptNVBYlVVXDrHcOs0s9wpvIlSWmc9RsINMu+6FaedXAs3nKY4gUpWxmNLEpLG8V7D2mNJZHk1adghTN9p8RMGcLtNylDY9W1Sdnziga/NG0WulZf+4IWV6e7wc2oBCSekTofCZuCva8HazJ1EIeqBvt2GuXoG1CjjbPNb2UwZeH21Iol1W0C5So1VFguxHZq5WL44mqD79NLlnSAMmQMSvxf7gXQVXEdtT5UlF9nYCkuQ+K+mT+8iRMy1YSCQSQ9kkD36unkrlYcLXKoPLYxHcoo7fGZ0MPIW8J+P77Hp9Z8jNPCyCaeSJYApCxybO8ujbUGdIr2FCP1z/JDP4/zZat9MkTA6G3H9kDNso7AJBHL/NfkWImvQnveLGj0CuhPpf2zwOylCqTeXrgoN8zztO2/qVDa3VRQbznap0njr0qzVrvOfTxYoiRmkGh3PXZ2mb/NqbeIvCFhj2VyVgavdMchlIbM3fCzXGEdCa2RN+ZYVtcFe/b6wR/qf7lrMaPHsTZiMfYuMJoC9F9wdbkOttT9c36Sob4x+aQYLs7/+XGCJdbw672qot6x6P7GHjY8zVKlMYLSZjKcoD48c=
secret-key: AgAHz7lfqTLG0Kldt/IZGW/o8KKazICHGbdVklXs3Gzs6Mc2KaG3nggkZDRWwY6noochg4Wc9mDDMGY3XIQnEy3PXs/rjQgwym0Dbvc54CvjWUcfmxCJyIqY1uCUxoXp/WAEV18lpWFSkVx+5U9wSccYVmkOJXNJ/MgH0vzJPaNW6xpYDg1XRC4nj72vwqyvZ/gM/LEX41NOpPjBROLGql3uBk5UfzNZG4dBun0aaPiYx5KKAbz8JAbK0umvOOOctTPSv1hIhyv9xJRRB67MW/J8T6mB2YfezsisXSSOAVNuE2JWrafHFxKq/hSNQHSrQRKjt18k6vfxmfkICbgQUZl/7AnoKi/f+76Feu+3wuMT+XPsaJN+iWb3jNzcb1T+Aiv4rGyIA7uCB6FNOc9Ngf5ngDQJw0nDNHDSnUoowgcE96kuPefU1rmn2V+JJcwoYdzYE3Ds48C0rNT+WSIit4551SiyWuj7iCRLhyEf5vhojxkuoyCN+EC1a50XwIkt223EBmy6FFPeBxql6HyJbOE313BPsQU6enPrTKNFd7iZYqJXq1k+SRuGwJhMeo1BBNhmVqxFGOpyxkSVs3VPybeXd82Zk/xF0SWKyBp3XQnLB39ZmT83Z5jwrYfkZmWkkdRvCME6NmlQtftRDDk3efGbgw7bz4Eq0d08KPCezsaDSAGNewonYy3UOiv/CNQg5hQ5G6R55y7+kEMV4nNkjGj0qF1hrZURS0mV4I/7H0OwXpDYiVetfaYI
template:
metadata:
creationTimestamp: null
name: forgejo-s3-backup
namespace: forgejo

94
apps/grist/README.md Normal file
View file

@ -0,0 +1,94 @@
# Grist Deployment
This directory contains the Kubernetes manifests for the Grist deployment at https://grist.basicstack.de
Grist is an open-source, self-hostable spreadsheet / database hybrid ([grist-core](https://github.com/gristlabs/grist-core)).
## Components
- **grist-deployment.yaml**: Main Grist application deployment
- Image: `gristlabs/grist:1.7.7`
- Port: 8484
- OIDC login via Pocket ID (`https://auth.basicstack.de`)
- Data persisted at `/persist`
- **grist-service.yaml**: ClusterIP service exposing port 80 -> 8484
- **grist-data-pvc.yaml**: PVC for Grist app data (`/persist`), 20 GiB on `hcloud-volumes-encrypted`
- **grist-postgres-deployment.yaml** / **grist-postgres-service.yaml**: In-namespace PostgreSQL 16 for the Grist home database (TypeORM connection)
- **grist-db-pvc.yaml**: PVC for PostgreSQL data, 20 GiB on `hcloud-volumes-encrypted`
- **grist-ingress.yaml**: Traefik ingress with TLS
- Host: `grist.basicstack.de`
- TLS via cert-manager (`letsencrypt-prod`, DNS-01 via Hetzner webhook)
- **grist-db-credentials-sealed.yaml**: SealedSecret for PostgreSQL user password
- **grist-app-secrets-sealed.yaml**: SealedSecret with `GRIST_SESSION_SECRET`
- **grist-oidc-secret-sealed.yaml**: SealedSecret with Pocket ID OIDC client credentials
## OIDC Configuration (Pocket ID)
- Pocket ID group: `grist`
- Client name in Pocket ID: `Grist`
- Callback URL: `https://grist.basicstack.de/oauth2/callback`
- Client is restricted to members of the `grist` group
Grist environment variables driving OIDC (from https://pocket-id.org/docs/client-examples/grist):
```
GRIST_OIDC_IDP_ISSUER=https://auth.basicstack.de
GRIST_OIDC_IDP_CLIENT_ID=<from secret>
GRIST_OIDC_IDP_CLIENT_SECRET=<from secret>
GRIST_OIDC_SP_HOST=https://grist.basicstack.de
GRIST_OIDC_IDP_SCOPES="openid email profile"
GRIST_OIDC_IDP_END_SESSION_ENDPOINT=https://auth.basicstack.de/api/oidc/end-session
GRIST_OIDC_SP_IGNORE_EMAIL_VERIFIED=true
GRIST_DEFAULT_EMAIL=andreas.leinen@basicstack.de
GRIST_FORCE_LOGIN=true
```
`GRIST_DEFAULT_EMAIL` is used by Grist as the primary admin identity - the value must be a valid Pocket ID user.
## Storage
Both PVCs (`grist-data`, `grist-db`) use `hcloud-volumes-encrypted` (Retain reclaim policy) as required for
sensitive workload data on Hetzner.
## GitOps
Managed by Argo CD via `apps/app-grist.yaml`:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: grist
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/grist
destination:
server: https://kubernetes.default.svc
namespace: grist
syncPolicy:
syncOptions:
- CreateNamespace=true
```
## Verification
```
kubectl -n grist get pods
kubectl -n grist get pvc
kubectl -n grist get ingress
curl -I https://grist.basicstack.de
```
## Backup
The PostgreSQL PVC uses reclaim policy `Retain`. For scheduled logical backups add a `CronJob` like the one in `apps/directus/directus-backup-cronjob.yaml`.

15
apps/grist/grist-app-secrets-sealed.yaml Normal file
View file

@ -0,0 +1,15 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: grist-app-secrets
namespace: grist
spec:
encryptedData:
session-secret: AgBN2KDg6QQgvhD2WvTG/r4+y6FrY7EXc8kU8N4hUDIh/VJDoyK+xiEHFvuOXMH3ZpNw4qTEpL+a+6LVWM1/c6mdcTIW5J52+hOOoXEtSXJhoBCjljgddlTTxHy9sIdE0694dMgeBJwmnvT9vm1TNQWMbiBaD05Xh4VCX3D908L4l/CiM11M9CLkoMGK7cTz9OCuGtd36k4OUxmw3/mQnafCDyzKf07NRW70Fr/bMGCdKE4Jt1rqWBfIaTb2mq96FCHosFq37keTCh1gu2nKf3iWesfv40rsVNN3wEhATaG2ghmjP2ul21zMOIO5GuCnMcawZv8VR1gyb5f98WOinL1zf5NMPCPRdTl/3G2gpqPc9VRNxOGguup3NLCaV2J+vbH5dgEj/h6zeTXw5bRARRQHAaGhUgFS3PALzubTzaUyNyFOyaMoXbAnE79HU+7QEd3Yjpdujup4Pe6OioK2k7jGWJj/jxZokffCjUsG9+hvNlg2E4ETtXxmHXIkwgTx7nHON6UApnprCtAUatRuMT3H+4VzfgkjV6y2aIHcp21EbH+qDBLo83+sBeUeWYyRtA/W2BGoAhx1CXeBysmEjD/Cls2h2PVhIvUdfNBY9fDgLemuK7MxdKbR8hRb3zHeedLXM4wevg2zCTyyGtizeC3JmpnqO1UHsKBZMGyUh43r+7sh3SyBKVUJxZt0q2JOWibaZNbIsrqcAyb8Te/QZlayZ2s2BxvMvvNrNpcOTYs/KzwwaHva8OUl
template:
metadata:
creationTimestamp: null
name: grist-app-secrets
namespace: grist

15
apps/grist/grist-data-pvc.yaml Normal file
View file

@ -0,0 +1,15 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grist-data
namespace: grist
labels:
app: grist
component: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: hcloud-volumes-encrypted
resources:
requests:
storage: 20Gi

15
apps/grist/grist-db-credentials-sealed.yaml Normal file
View file

@ -0,0 +1,15 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: grist-db-credentials
namespace: grist
spec:
encryptedData:
postgres-password: AgA7MFNJ8//EOg4CdfHlSe2yGW6zcJyGsoiWVYlbVkb2GgMZmGQDH/ImYOTfeZXkEQhAWhA0braRB7Prx8fwpON2+6PsOczg+S1N1ykuDMJ+KDuXHhJtm5LNqbgibddX9y6YYInorgxM5mh/ORt2D1IdTwf1AXACkEkp8Ws4IGIfuUCym7kNM77mVD9jW4xyf/c7RE/J3va7L/6SRdx2rdoil8loMzMTHH+lvc738UdFA67hoLkKuseiYgA6NZap5Ppb48GTQul47ji3wH24F/u8DTKiHWbMBxwWO48gQJvAISgXSCW4eiKhkAcm28JACCOd1jT7Asen/yd5qEISB8Fr8q2qwxQwM3kXFIJRf9VVN7BWv6hEMnDO3AhBmcRr+gCPhKial17Varakh0ruBXB4eQDlxGQZ09trwtRPZkOsQnNf38iMQBpDH5U12tASmi1t5ddGKzQQh3EWx1vJyap8xNFf5zK6MuoDEMekHrjcclqgoMWppMH26Daghp2XZl740bV0UhLQBO1TOFEORHBs2DA2A3msmN25selVxjFBHvwLZHQJOHoVHbcqlbF7e2bFSLnij1TBMlPre/llBzoiVYrjHQoHw3fg7xVMrPBT+U1wn4ItDFZTP+nCxnWejtQ/dxzBfMrvBcHVhACsw7VhVkM5nhNFJBda65czyud0BWCSMYJYZEBZjDs220/mhQWIh2/RLANN6pcwJ2ONGyNJs3uo7Q245DlmDTjH5Qppmg==
template:
metadata:
creationTimestamp: null
name: grist-db-credentials
namespace: grist

15
apps/grist/grist-db-pvc.yaml Normal file
View file

@ -0,0 +1,15 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grist-db
namespace: grist
labels:
app: grist
component: database
spec:
accessModes:
- ReadWriteOnce
storageClassName: hcloud-volumes-encrypted
resources:
requests:
storage: 20Gi

127
apps/grist/grist-deployment.yaml Normal file
View file

@ -0,0 +1,127 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: grist
namespace: grist
labels:
app: grist
component: app
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: grist
component: app
template:
metadata:
labels:
app: grist
component: app
spec:
initContainers:
- name: wait-for-postgres
image: postgres:16-alpine
command:
- sh
- -c
- |
until pg_isready -h grist-postgres -U grist; do
echo "Waiting for PostgreSQL..."
sleep 2
done
securityContext:
fsGroup: 1000
containers:
- name: grist
image: gristlabs/grist:1.7.19
ports:
- name: http
containerPort: 8484
env:
- name: APP_HOME_URL
value: https://grist.basicstack.de
- name: GRIST_DOMAIN
value: grist.basicstack.de
- name: GRIST_SINGLE_ORG
value: docs
- name: GRIST_ORG_IN_PATH
value: "false"
- name: GRIST_DEFAULT_PRODUCT
value: Free
- name: GRIST_HIDE_UI_ELEMENTS
value: billing
- name: PORT
value: "8484"
- name: TYPEORM_TYPE
value: postgres
- name: TYPEORM_HOST
value: grist-postgres
- name: TYPEORM_PORT
value: "5432"
- name: TYPEORM_DATABASE
value: grist
- name: TYPEORM_USERNAME
value: grist
- name: TYPEORM_PASSWORD
valueFrom:
secretKeyRef:
name: grist-db-credentials
key: postgres-password
- name: GRIST_SESSION_SECRET
valueFrom:
secretKeyRef:
name: grist-app-secrets
key: session-secret
- name: GRIST_OIDC_IDP_ISSUER
value: https://auth.basicstack.de
- name: GRIST_OIDC_IDP_CLIENT_ID
valueFrom:
secretKeyRef:
name: grist-oidc-secret
key: client-id
- name: GRIST_OIDC_IDP_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: grist-oidc-secret
key: client-secret
- name: GRIST_OIDC_SP_HOST
value: https://grist.basicstack.de
- name: GRIST_OIDC_IDP_SCOPES
value: openid email profile
- name: GRIST_OIDC_IDP_END_SESSION_ENDPOINT
value: https://auth.basicstack.de/api/oidc/end-session
- name: GRIST_OIDC_SP_IGNORE_EMAIL_VERIFIED
value: "true"
- name: GRIST_DEFAULT_EMAIL
value: andreas.leinen@basicstack.de
- name: GRIST_FORCE_LOGIN
value: "true"
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "1"
memory: 2Gi
livenessProbe:
tcpSocket:
port: 8484
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /status
port: 8484
initialDelaySeconds: 30
periodSeconds: 5
timeoutSeconds: 3
volumeMounts:
- name: data
mountPath: /persist
volumes:
- name: data
persistentVolumeClaim:
claimName: grist-data

26
apps/grist/grist-ingress.yaml Normal file
View file

@ -0,0 +1,26 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grist
namespace: grist
labels:
app: grist
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: traefik
tls:
- hosts:
- grist.basicstack.de
secretName: grist-tls
rules:
- host: grist.basicstack.de
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grist
port:
number: 80

16
apps/grist/grist-oidc-secret-sealed.yaml Normal file
View file

@ -0,0 +1,16 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: grist-oidc-secret
namespace: grist
spec:
encryptedData:
client-id: AgAAQEIKFFtfc3pm1yp3HKu7uIiTsYtcer6CYJbrUAY6bwNBE+Cj2NF3wvDVeW1M2padhXqyuuhuOIaYbbQ8NOgR3kVg6UIKgn5oNimpXgkwef4xCuGNbNN7kfRfVTJfwrJm8jON0nzmmvxnmsNsz7hJgRszQ2Tp1qJlCI/KIPsIZ0vcXehy+VWV5xBOQOnP+b/wxR+tbt9BnDX/acRi95JAJLXYabqkD7FDc0rLGiWmEsyl4prhhRcT2Ux7O6CJuyegAOOodPDsylkV2iyaI6L3EEY9IrrV5w4ta2tDDPEcnuLkUo/1LipfJUSGzO/tqYJMAbd6klTtfI3cIjimmQXUP/0uiy4YXT7LJud/yEwjSajRIMAdIMQNhAcXPM3EnW85nck4NYSO8qUgu04+lpPiQKiE1eSjCa/LWBIqOMG69nE/hZCCceNBa2krvJIhjrPewBEMxBvg4aCUUS9GO3YanoDN3Y5T9HcpjvbhtLvLMGu20uAKxw63m8DTyNnzruYkL4Qr/5KxyhquCGv9YuwJMtaiUlWc36mh6YKEyyiYSe56cMegUnIZKBWWjsJwJm+ApPKZSI3u9/AGiHodF9tKmqt8vCzuqyzhsAskQXpewdn3iHe2IY2KwuWPPoObUaaSYWcKdoEZYVgaY9IofXf/x4GbJ5GVqbrwCGFpXJGk3GFFtvlyFICINU+CCp5PXgoMiqvpNvUVEHVclCmcTMi8lWufWQi/6Xu4gH+08QHVMkzf55o=
client-secret: AgBrF2e/M48zFC0PdXm08IavK+qa3Sl3a0qmO0+eH/z5WYnRpgos5dFdCLxcrdDcJUgKy//pU+I23qoTXqgybxRnfVzsOdYb4UksGgWh9640BQ/NRuxqhdnfXjQN7PrJj+D+fumwRdTQU1PqyzEePM/6CVf6Ro9vdP6iTMDzbAnPnHoA8L8Hd+f3YKo43OKMR2GMwjNKAhpD20wMCYt1vTiEQX4drR9JW4mjcKfSCdaIrEFwIYn4EBkr1SwuC4Ek/KTn4CNQO82BQH+jjWO5CHRvLP6b4v2bVkgdQEqLR4+HpNjMm0JBKOjHvhfKv952T5mwROamWcpNlN4KW02YzIZMBU+dh7WhouZkDkQCalP4G7AGoLrjhWmLIgS56JqTuoiobFf90Uvz6ewcDmHAzPuYZMfn2mKndPYvoi/KSGzM3ceeNrFxpHCPhsG+UHbOfSCBmg7jqIeE9ial6fMsRRLeEqc3ay4lAN7PBXwJ/o3XMrezUy8RQsXLex9oS0IS9rktimwQNWEpY1adwWRERdBKqGAIEbVofVwai/SuC/dcgQsoX2r9kJOiA8ahM+eDxUg1534hb21v3tFWykp3y/EbFBxbPPtCsUY4ghZZ8O/lLtXsZdHxaq68RKbVqDbRGa/eLqCqStWS8Bk1XzwkJFBBHFcN9hDvNwoT+33QSSsuuiw8gLiztlSrewo33rVKEc9Enqb+f/uUX6HY9j3Iz7hvKZ2aYX/U8ayNN7hQaJryYQ==
template:
metadata:
creationTimestamp: null
name: grist-oidc-secret
namespace: grist

70
apps/grist/grist-postgres-deployment.yaml Normal file
View file

@ -0,0 +1,70 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: grist-postgres
namespace: grist
labels:
app: grist
component: database
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: grist
component: database
template:
metadata:
labels:
app: grist
component: database
spec:
containers:
- name: postgres
image: postgres:16-alpine
ports:
- name: postgres
containerPort: 5432
env:
- name: POSTGRES_DB
value: grist
- name: POSTGRES_USER
value: grist
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: grist-db-credentials
key: postgres-password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
livenessProbe:
exec:
command:
- pg_isready
- -U
- grist
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
exec:
command:
- pg_isready
- -U
- grist
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: grist-db

17
apps/grist/grist-postgres-service.yaml Normal file
View file

@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: grist-postgres
namespace: grist
labels:
app: grist
component: database
spec:
type: ClusterIP
selector:
app: grist
component: database
ports:
- name: postgres
port: 5432
targetPort: 5432

17
apps/grist/grist-service.yaml Normal file
View file

@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: grist
namespace: grist
labels:
app: grist
component: app
spec:
type: ClusterIP
selector:
app: grist
component: app
ports:
- name: http
port: 80
targetPort: 8484

11
apps/harbor/Chart.yaml Normal file
View file

@ -0,0 +1,11 @@
apiVersion: v2
name: harbor
description: Harbor container registry deployment for BasicStack
type: application
version: 1.0.0
appVersion: "2.12.1"
dependencies:
- name: harbor
version: 1.16.1
repository: https://helm.goharbor.io

71
apps/harbor/README.md Normal file
View file

@ -0,0 +1,71 @@
# Harbor Container Registry
Harbor is deployed at https://harbor.basicstack.de
## Initial Access
The initial admin credentials are stored in the `harbor-secrets` sealed secret:
- Username: `admin`
- Password: Retrieved from secret key `harborAdminPassword`
## OIDC Authentication Setup
Harbor requires OIDC to be configured via the web UI or API after initial deployment. The Helm chart does not support OIDC configuration at deployment time.
### Steps to Configure Pocket ID OIDC
1. **Create OIDC Client in Pocket ID**
- Navigate to https://auth.basicstack.de
- Create a new client with these settings:
- Client ID: `harbor`
- Redirect URIs: `https://harbor.basicstack.de/c/oidc/callback`
- Scopes: `openid`, `profile`, `email`, `groups`
- Save the client secret
2. **Configure OIDC in Harbor**
- Log in to Harbor as admin: https://harbor.basicstack.de
- Navigate to: **Administration** → **Configuration** → **Authentication**
- Select **OIDC** as the authentication mode
- Fill in the following:
- **OIDC Provider Name**: `PocketID`
- **OIDC Endpoint**: `https://auth.basicstack.de`
- **OIDC Client ID**: `harbor`
- **OIDC Client Secret**: (paste the secret from Pocket ID)
- **Group Claim Name**: `groups`
- **OIDC Admin Group**: `admins`
- **OIDC Scope**: `openid,profile,email,groups`
- **Verify Certificate**: ✓ (enabled)
- **Automatic onboarding**: ✓ (enabled)
- **Username Claim**: `email`
- Click **Test OIDC Server** to verify connectivity
- Click **Save** to apply the configuration
3. **Test OIDC Login**
- Log out of Harbor
- Return to the Harbor login page
- You should now see a "Login via OIDC Provider" button
- Click it to authenticate via Pocket ID
### Reference Documentation
- Harbor OIDC Configuration: https://goharbor.io/docs/2.12.0/administration/configure-authentication/oidc-auth/
- Pocket ID Harbor Example: https://pocket-id.org/docs/client-examples/harbor
## Storage
Harbor uses encrypted Hetzner Cloud volumes for persistence:
- Registry data: 50Gi
- PostgreSQL database: 10Gi
- Redis cache: 5Gi
- Trivy vulnerability database: 5Gi
- Job service logs: 5Gi
All PVCs are configured with `resourcePolicy: keep` to prevent data loss during upgrades.
## Architecture
- **Ingress**: Traefik with Let's Encrypt TLS certificates
- **Database**: Internal PostgreSQL
- **Cache**: Internal Redis
- **Vulnerability Scanning**: Trivy enabled
- **Authentication**: OIDC via Pocket ID (after manual configuration)

20
apps/harbor/harbor-secrets-sealed.yaml Normal file
View file

@ -0,0 +1,20 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: harbor-secrets
namespace: harbor
spec:
encryptedData:
database.password: AgCS/v9ajG58l1fF7cegx/pQLn4jBgJD3ixVDbQow2f0o6FvKXJK2na7IBdvaoE3rP5xO8BygQkU3z/Dq6TKKp5AuU1mcevkM5gIE2YRP4ZkXaChFhdGnZa6fAp1wU+K2rHKbGM2vNHUFXznHEogemD4IvSOJuiKJfx9ezUNCz+n3v1i3U/5R1j8CmJX/nXxzZRocMl0yJrSbBFsObU87MlO0I0L5pwQwoXRiW8WL3Dl9AJtd6EGvR8AvBQVYXzS3ke7F9mgC/fm4cTkfliHeGUUIylx48/Jr5zkdj6qNDr6q46+XzhKIzDkQrkY9fJVrmWeZoLhws5F9SpNWBbfwYSM266mlzWafEFIDjYIlgl4bGovnKceT6Ezh2ltPvQ6l/ppO95TH/A3HjzrSRH6Oo+rw4d+OE9skFz3KUgtOZHTFbEzAUiMSSL8g5T2AAyN1nG5JnJm4fx45RWR+JCbddtrBA9R+UwRfMYbYXYxCbqYBINgWzgeal0O2Qd7jnJROOZlLOZGPl4PwN8NNqfWfVTkw4we2/vDXzCG70/zS/pY0Q47lNAjdQOn8WakfuumT3PkhQM6739qeTM3no/Ga7xG9OmjvroDEokdRnhI4HZf4KcbdvtAZcsV3KD6WoHhTULtK8C/4+cb/Bj1QIz/UQ0/ZF2oahLpCwQyDRxBOUfZB/G1PMPZbwg8LkgCKit1RDGOVuMD25TOKhm9W7ONsJrJ55h+pcNBc7/TcNv4mtNMlWh8dYfMFeKsyCw8vg==
harborAdminPassword: AgBTLxpA5ygiVNj5ADtI1/2w2fSdbYPwBOVmCkj7FodimXl5vCoNKnd0rO7t8SPW+1SDDvQsOjekhWaqkMEIm4rLtRpRU1TWWuVG6UkWID3aBwWBqOafMD11Wvq/uQrlYuQLmj2pKPRZxfODCWUxNEC9vZXAucCL8EzXlgYXy+P0ZX3vbj4/nnnPDYC4vfxl0/dLdVubNTfoBhZ8GA2waPVHuQRZKJCqFaHt2RAdXFzuGmuu0jIa7bf3Izr2CzAKkhz515E4TOtDitFNQOJExUCf/14NqKp142s1/4KtB9iuUSYodswKyJGtAtVGRPIGjkYB1a37RFbqVvP3RN5Bx99cH8Wdj816l4y+Qq0mvg/arClLggOtDZZKMsd5QXaPkTVlyBKItNhfDjDgk+G8rzzNwspN73BS6CRS6tWu1BGufsWmnfm/vSCdkw/EjuoPj5LOyWkPdhrJxK+dgvr5zYfw+/mISXucMy4SQDFuSbp9reBqE329nJolyUuRQFxx2hGKm7KIB0vFRi62cH2mBmfG//d79eq/4hSCPMNola8SJ0neWvvaJkR491LLN5Pus9zFcOHNSy/i+lpXs19ZMacLeL/X4yuq7neiFTaMtKIcJ/rQSwGxf9lDcC2jadi3AlSmRP6Va3XwNm7FEP83bRUV5pScLlnwY+EqgB7GdAC+2Bd+Ls3Hb5mc2JoNZnpZi9A1Y+108NWtvnjBMbFwp/seHXZqRUwrlhNq2R/cXdPRJLVnY2U+NRtEYTfivg==
oidc.clientSecret: AgCNjio841Vsf+UysN38gKTD+E1QknD5Wmsfp3IKoqvmChI3brEfTSjb4pROAcSscVnC/WHQbl73J1PJIj8YPs7173nn8gJDBnoIIjlVM3p33RdzWwljgb8MRCFXZ2fjimYE2DwcdIy3Izcyf/6m2w01xIcNY4GpHcKpcwA9LoZ/oGxzkPrZx7FOmDKpi3mN27eBfKbhvpVuIL+KhT40z1G8F3C6khA5cg2hiyPbQgWhcbDLhOCUZCTcqKeUJ7VWwYS1D9kNE05V+rWJJqfV/jvjHJ12ajwUSSkpHG5fjPq1eVuikkCzhXZyxSAocmtMt5Jspve1dz5AqEiw5EkH6Tb4EVfLYqCY6IJH9LAs1+8+8ETrszMRdCEZLSQhcAWXNVbURPcoJRUeSI9qQMBECdG7QVP0S6VQ8hAMadtJ1VnDHcF1lLFKKLDPLiNvnqz0FlO4gCYH193gNbfOqoSQkidwpzVX0aRWZUyBwMbS9umbrFrvZQ/hzj7U+7SIxVHoYvLAigw8gzy3ZyA8qkttddayZlO9pDvSBlNW6QbdTVn9Iag4rjgpkBvXv5MTHUR4S4ICO5LDzAyPmkxmMpcl+gYd/xEuI59c7q+GMQCHyTMi8UpIX2zbPKa4WROFBb684dgOh2dZyYO4+Ywbaj99chXX6AtWLIBVHfVcVXYIZVgduqjNkCfmGUuV+gTOBDbu97RLOJ6p6woWZ9j9yDWOTBmA1npisiTr30da0fJWQRgFag==
template:
metadata:
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
creationTimestamp: null
name: harbor-secrets
namespace: harbor
type: Opaque

121
apps/harbor/values.yaml Normal file
View file

@ -0,0 +1,121 @@
harbor:
expose:
type: ingress
tls:
enabled: true
certSource: secret
secret:
secretName: harbor-tls
ingress:
hosts:
core: harbor.basicstack.de
className: traefik
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
externalURL: https://harbor.basicstack.de
# Hetzner Cloud Volumes are ReadWriteOnce; RollingUpdate deadlocks on
# jobservice/registry rollouts because the new pod cannot attach the PVC
# while the old pod still holds it on another node (DEV-458).
updateStrategy:
type: Recreate
persistence:
enabled: true
resourcePolicy: keep
persistentVolumeClaim:
registry:
storageClass: hcloud-volumes-encrypted
size: 50Gi
database:
storageClass: hcloud-volumes-encrypted
size: 10Gi
redis:
storageClass: hcloud-volumes-encrypted
size: 5Gi
trivy:
storageClass: hcloud-volumes-encrypted
size: 5Gi
jobservice:
jobLog:
storageClass: hcloud-volumes-encrypted
size: 5Gi
# Use internal PostgreSQL and Redis
database:
type: internal
internal:
existingSecret: harbor-secrets
# Fix PostgreSQL data directory permissions
# The init container needs to set correct ownership for PostgreSQL
initContainer:
migrator: {}
permissions:
image:
registry: docker.io
repository: library/busybox
tag: latest
# Set proper security context for PostgreSQL
securityContext:
runAsUser: 999
fsGroup: 999
fsGroupChangePolicy: "OnRootMismatch"
redis:
type: internal
# Harbor admin credentials
existingSecretAdminPassword: harbor-secrets
existingSecretAdminPasswordKey: harborAdminPassword
# OIDC authentication via Pocket ID
# Note: Harbor requires database_auth to be configured first, then OIDC can be added via UI
# See: https://goharbor.io/docs/2.12.0/administration/configure-authentication/oidc-auth/
# Resource limits
core:
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 1Gi
cpu: 1000m
portal:
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 256Mi
cpu: 500m
registry:
# Skip Kopia fs-backup of the registry blob PVC (DEV-573 safety net).
podAnnotations:
backup.velero.io/backup-volumes-excludes: registry-data
registry:
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 1Gi
cpu: 1000m
trivy:
enabled: true
# Skip Kopia fs-backup of the trivy scan cache PVC (DEV-573 safety net).
podAnnotations:
backup.velero.io/backup-volumes-excludes: data
resources:
requests:
memory: 512Mi
cpu: 200m
limits:
memory: 2Gi
cpu: 1000m

2
apps/headlamp/deployment.yaml
View file

@ -16,7 +16,7 @@ spec:
serviceAccountName: headlamp-admin serviceAccountName: headlamp-admin
containers: containers:
- name: headlamp - name: headlamp
image: ghcr.io/headlamp-k8s/headlamp:v0.43.0 image: ghcr.io/headlamp-k8s/headlamp:v0.45.0
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
args: args:
- "-in-cluster" - "-in-cluster"

61
apps/kaneo/README.md Normal file
View file

@ -0,0 +1,61 @@
# Kaneo
Self-hosted [Kaneo](https://github.com/usekaneo/kaneo) — open-source project
management — deployed on the BasicStack Kubernetes cluster.
- **URL:** https://kaneo.basicstack.de
- **Namespace:** `kaneo`
- **Distribution:** hand-rolled manifests, GitOps-managed by Argo CD
(see `apps/app-kaneo.yaml`).
- **Auth:** OIDC via Pocket ID (`https://auth.basicstack.de`) with
`CUSTOM_OAUTH_*` env vars. Client id + secret live in
`kaneo-oidc-secret` (SealedSecret). Access is restricted to
Pocket ID groups `kaneo-admins` / `kaneo-members`.
- **Database:** dedicated PostgreSQL in-namespace on
`hcloud-volumes-encrypted` (20Gi). Both the Postgres and Kaneo
pods pin to the `fsn1` datacenter via
`csi.hetzner.cloud/location=fsn1` node affinity — the Hetzner
volume is DC-local.
- **TLS:** Traefik ingress, cert issued by `letsencrypt-prod`
(DNS-01 via the Hetzner cert-manager webhook).
- **Image:** `ghcr.io/usekaneo/kaneo:2.22.0` (single container
bundling API + web on port 5173, health at `/api/health`).
## Files
| File | Purpose |
| --- | --- |
| `kaneo-postgres-pvc.yaml` | 20 Gi encrypted Hetzner volume for Postgres data. |
| `kaneo-postgres-deployment.yaml` | Postgres 16-alpine, pinned to `fsn1`. |
| `kaneo-postgres-service.yaml` | ClusterIP service `kaneo-postgres:5432`. |
| `kaneo-db-credentials-sealed.yaml` | Postgres user password (SealedSecret). |
| `kaneo-app-secrets-sealed.yaml` | `AUTH_SECRET` (Better Auth JWT secret, ≥32 chars). |
| `kaneo-oidc-secret-sealed.yaml` | OIDC client id + secret from Pocket ID. |
| `kaneo-smtp-sealed.yaml` | SMTP submission credentials for `kaneo@basicstack.de` (SealedSecret). |
| `kaneo-deployment.yaml` | Kaneo app deployment (pinned to `fsn1`). |
| `kaneo-service.yaml` | ClusterIP service on port 80 → container 5173. |
| `kaneo-ingress.yaml` | Traefik ingress + `letsencrypt-prod` cert. |
## Post-deploy setup
1. Point browser at https://kaneo.basicstack.de and sign in via the
"Sign in with Pocket ID" (custom OAuth) button.
2. The first admin (`andreas.leinen@basicstack.de`, member of
`kaneo-admins`) must be promoted inside Kaneo — the current
upstream release does not map OIDC groups to roles.
## SMTP
Outbound SMTP is wired to the Stalwart submission relay at
`mail.basicstack.de:587` (STARTTLS) using the `kaneo@basicstack.de`
mailbox. Credentials live in the `kaneo-smtp` SealedSecret
(`kaneo-smtp-sealed.yaml`) with keys `smtp-user` and `smtp-password`.
SMTP is used only for workspace invites and notifications;
`DISABLE_EMAIL_OTP_SIGN_IN=true` remains set so users still sign in via
Pocket ID SSO, not email codes.
To rotate the mailbox password: update it in Stalwart, re-seal
`kaneo-smtp` with the new value, commit, and let Argo apply. The
existing pod picks up the new secret on next restart
(`kubectl -n kaneo rollout restart deploy/kaneo`).

15
apps/kaneo/kaneo-app-secrets-sealed.yaml Normal file
View file

@ -0,0 +1,15 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: kaneo-app-secrets
namespace: kaneo
spec:
encryptedData:
auth-secret: AgArCARyFt7NinaeQOfDDEpTKKWVba8IlKh7NN/vflgdOkE5bjz/P/ZzGUVhUOTyqCs+3dU+hX6sqc7IYbYckXdprwKJSXy47HLEsZ/cuJ0FRIBShRyXm+I62Yu8WWLHNJLTnlWyfirBC6Pg65kXckoKiYPwcVLSD5I+YHzo3Dt3Hl6MeW/iVocM9xgjBD/5yRt1ag+S4JYv0t1y39Cz80XxbFvo55RzYWr3OlYNI325b9W/d5uylqlDemln8NymL5iCdfnLDZoc+1r+eHAf1HDP6Uch1dG+c/xsrlvoWpzyMeSEbEY1tGeYFZ1oKm4B1pyUH9dfdxWEaNThheL5Xw2QX/klpZdTq5Jl6Lfxme5aLyqNtP9J6ovd5b1oVYW1DWc+RAqBPhFlHnkfc+8ZeP6vrmnUULq0ssSVqpXOz4zLgDHU61UzSllNdpZWXokigranj3dq5CM7WkYLEj9kfdDIGS3gDUD+EmIkIj/N96URNpOINSwFVAcnE73k3ij9mZWECLl++lzh1fqHrdYQiTI6A7UpxPudc7BoujBQFqGhUDXGdzBWsrqr11tldD+kb/kcKQYxs6sFHyw0pDgCLVZukPv7TWRNfVRYUv68h/1ZKAwmj8KkuGlNtAQ+NyvUWyct3SA+pQYWabr33pIPekWPIXMIty6/zfKu92hLFglwkauewxqLtEBZ6FFukKMyyfowV+iwDIMyY3//MvxiRzxD7sc5sLwcfvPCBzd9XGaJO1ZLX6r43gfOu6HFHiq4FqdjYJ9fqG3zUKeo5bnav7E6
template:
metadata:
creationTimestamp: null
name: kaneo-app-secrets
namespace: kaneo

16
apps/kaneo/kaneo-db-credentials-sealed.yaml Normal file
View file

@ -0,0 +1,16 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: kaneo-db-credentials
namespace: kaneo
spec:
encryptedData:
database-url: AgDEiqPdexD58tc4Sv63mk1erIHAeweJzGpKgoOcro6ioW4eX9pWnfVnVfmtsqQK8WXkRuEP5pAnj1LZAhuo6hfwk/lOpoYItqZKl6KGK0EQZ3u/hcmUt0yeEkTJuDPGBbgk1RuT1G2u2uDDYYJYHmAQtIIFbAoeJS2iDKnKkl6UHZdEJhO10w6IHGiNCT6Z9NovLJau5cw9tTv7xnbGO8F1QqwLh/dL0kI2oYNTYg1RxpegXRFeKGz+QSJ3rbxlH2P/E01Fevma2/ZhOBjALS1bpW9R6pgQm4n/FDu/fu72T62yw2oNvj6nIeRtIsm9ze02jQqYwF7mgct6IJOdqY7RoYnTZJQ+/Xl/iiBIGpzXan9bU/JZw6sEeyxS+tyeaolz4oiTeNfRXp8QNdX3qbXM9OE5Q+bjqx1q+cZcnFcBIvGK8vGqlWwDjwNicucuNBZuv4JWEOp2N8axoFVRUsUn/Frv7McqKT1Q2mRXf0aiGYh9m7EIdnFAx3JeiBJv392jBWcwvJpQ+J0LIsh/bJEBtdC4c16Y+38nbqfjYEUJWVIoj9SvJEo3QZPitUz99cwdtDn+cOVVYp+lv5BiunM7OgvAgBbX7F4Ri/QlL8rt3/8lO0Augwp933lSTuGGnMCrXPz2QCagI6B0l8F5q5OTnljgR7JXslyGFaemtj9jZEnCiSsw9KAiDdOUyk36CJSfHmRedJJ6o2nalb3zdFDF7iMXxz6fpTQK0QQGAccO518zUHOpb3Jb6XbOBKDgTBOWPEGlroiavrOtoRDJ+B5nR/N4c3Indpn7d0TTDU/4rz4V1uNkRrHnA26ya9Q=
postgres-password: AgBo5f/NzqZ/9jsOzXmoivPVVeLA3xk9GWocQQlLv34Rc1K5fzhSoCYdy8o7k06R83VeNsJylxfAsOI7HNIo0xWUPqC8HnbW482ajtREh93wOFCaZzjKu8ClbFIkvTb19GRCPLSmzBMFsTCJzjSa5rWGqK2Ezj7LFeTTiehsqS/PbPUYi6w4X6ZnChS3VuMXJVJAvbCF8fFS3x/YuuJVjjd+TmVsPmGOa6J+r/6RG7l4DsR+SmV3HQ7M6bLqTsUPjzg5Foz11yc8ER8xQQTgbbfrBPwgJQoTX622yHf0Q313ns7vyiKGMqge9DSQvDmGmMDmAxqtYInycVBH1CwmfKVbw3CyFGt0D5LJXiBaTE89uLErsNCQCK6NKAODqffKlOWpZ1KVF+ZIQ6nqJETtj49uqdKGowauCs3tDl/Hi82+hxycEVMWUwgu81CJ6GT7+vSZfz4+9FRbLoXlbdfR/duG14f+AkPirikeDcy3bFy5XiBgPVQXL0EbAY8TaPkYu+TS1Roo2F6Gqj3D1+i4GapVodrBntl1gMPvMnd3dcGGl/VzREGZvRsZgdRprFi+nGPmotRhIO3eGcPautlZ5m0WMqKCpA4jW0sG1sgkVwRpn2Zdq4Uw2+IKNHTkJKclIXCaw2/b6WNCmiwbe1ssFoR/QgikzshshoB5XpIYFe7Jr5Sj9mmNH6L4PpCIZ/sCe8MF5eMD8IuyWSVmflID07sfBD6GL3XB9TFLzSyviHE/VJdGY7l+ZeR64UZZ1ti4bwI=
template:
metadata:
creationTimestamp: null
name: kaneo-db-credentials
namespace: kaneo

157
apps/kaneo/kaneo-deployment.yaml Normal file
View file

@ -0,0 +1,157 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: kaneo
namespace: kaneo
labels:
app: kaneo
component: app
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: kaneo
component: app
template:
metadata:
labels:
app: kaneo
component: app
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: csi.hetzner.cloud/location
operator: In
values:
- fsn1
initContainers:
- name: wait-for-postgres
image: postgres:16-alpine
command:
- sh
- -c
- |
until pg_isready -h kaneo-postgres -U kaneo -d kaneo; do
echo "Waiting for PostgreSQL..."
sleep 2
done
containers:
- name: kaneo
image: ghcr.io/usekaneo/kaneo:2.22.0
ports:
- name: http
containerPort: 5173
env:
# Public URL of this Kaneo instance.
- name: KANEO_CLIENT_URL
value: https://kaneo.basicstack.de
- name: CORS_ORIGINS
value: https://kaneo.basicstack.de
# Better Auth JWT secret (>=32 chars).
- name: AUTH_SECRET
valueFrom:
secretKeyRef:
name: kaneo-app-secrets
key: auth-secret
# Database credentials. DATABASE_URL is preferred; POSTGRES_* are
# accepted as a fallback and useful for the built-in migrations.
- name: POSTGRES_DB
value: kaneo
- name: POSTGRES_USER
value: kaneo
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: kaneo-db-credentials
key: postgres-password
- name: POSTGRES_HOST
value: kaneo-postgres
- name: POSTGRES_PORT
value: "5432"
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: kaneo-db-credentials
key: database-url
# Self-hosted: keep public registration off, log in via OIDC only.
- name: DISABLE_REGISTRATION
value: "true"
- name: DISABLE_PASSWORD_REGISTRATION
value: "true"
- name: DISABLE_EMAIL_OTP_SIGN_IN
value: "true"
- name: KANEO_CLOUD
value: "false"
# Custom OAuth / OIDC via Pocket ID.
- name: CUSTOM_OAUTH_CLIENT_ID
valueFrom:
secretKeyRef:
name: kaneo-oidc-secret
key: client-id
- name: CUSTOM_OAUTH_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: kaneo-oidc-secret
key: client-secret
- name: CUSTOM_OAUTH_DISCOVERY_URL
value: https://auth.basicstack.de/.well-known/openid-configuration
- name: CUSTOM_OAUTH_AUTHORIZATION_URL
value: https://auth.basicstack.de/authorize
- name: CUSTOM_OAUTH_TOKEN_URL
value: https://auth.basicstack.de/api/oidc/token
- name: CUSTOM_OAUTH_USER_INFO_URL
value: https://auth.basicstack.de/api/oidc/userinfo
- name: CUSTOM_OAUTH_LOGOUT_URL
value: https://auth.basicstack.de/api/oidc/end-session
- name: CUSTOM_OAUTH_SCOPES
value: openid,profile,email,groups
- name: CUSTOM_AUTH_PKCE
value: "true"
# SMTP: outbound only, used for workspace invites / notifications.
# Sign-in still requires OIDC (DISABLE_EMAIL_OTP_SIGN_IN=true above).
- name: SMTP_HOST
value: mail.basicstack.de
- name: SMTP_PORT
value: "587"
- name: SMTP_SECURE
value: "false"
- name: SMTP_REQUIRE_TLS
value: "true"
- name: SMTP_USER
valueFrom:
secretKeyRef:
name: kaneo-smtp
key: smtp-user
- name: SMTP_PASSWORD
valueFrom:
secretKeyRef:
name: kaneo-smtp
key: smtp-password
- name: SMTP_FROM
value: "Kaneo <kaneo@basicstack.de>"
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
livenessProbe:
httpGet:
path: /api/health
port: http
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /api/health
port: http
initialDelaySeconds: 15
periodSeconds: 5
timeoutSeconds: 3

26
apps/kaneo/kaneo-ingress.yaml Normal file
View file

@ -0,0 +1,26 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kaneo
namespace: kaneo
labels:
app: kaneo
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: traefik
tls:
- hosts:
- kaneo.basicstack.de
secretName: kaneo-tls
rules:
- host: kaneo.basicstack.de
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kaneo
port:
number: 80

16
apps/kaneo/kaneo-oidc-secret-sealed.yaml Normal file
View file

@ -0,0 +1,16 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: kaneo-oidc-secret
namespace: kaneo
spec:
encryptedData:
client-id: AgBGmlu+mGYlHcWPCRHZ3IbevGB5kGntqPasFBnny6ZydTeCmN8NWpy0bmoorecJSbvdeP38UDgfEjtmusfkipnrlfsZd7BugZLiijq5+T+9al5JTl/UChMl6u/gReaVPgLi9iketDN6oB2XC/56mV+MdMb/bO4d1u3CRFUJ/Pq1znWeQpVsXO1yOLoOge4+PYDYjgd4O6mJsbAQcmGxQ7pwcByw//+50Mvuw0gJBqI5ZSGKd0jSNnaVcECtvjY6v3xN1NCeVF1FoWKl4s7o/Nrq5mDltW/y6g0h3N1Fzan/D47UVQjGLYu6cTmM7ReqToYk65cznttGkyXfqrlVL/oEiYENtaqp/BZvwPBMKEAjJsufwGvbNgsxa3eGs+p3L+HwzdgTQbVGu2wVp+TXW8zwZvkDyprUU91Fu6QFsspV/GRYgqVmp4RxrJ0PzZInp4ZhLt/zOyWzbEkILMPV1A5fzPjrPmwpmhln/K0VTdpRN62XQRgH0P7EP6Ao0sP2/ae39eTkGlhb5fwzhSqK3LSmBYmoXNcffA4nBLFohSndmwgoxzZXM1FJ1kXl4wuevU8Eh8BD3dVIZkuYacKiIUFoNn74NbPtJpl3/MWfXef9Eb7IBR8KyurbdwSmKSAZDqjmeGTa271+KUiGfST+FYQgSDfFfQQEmh+K49WqqbOiod4Mh5RTGWJx3WNwEONsTRKVjiGgLw4A+ENNDB39PoRChoRoRRogg+ZJZE27EjxbmrKIjrw=
client-secret: AgBJQi6fIlTkOIZa1zr9jjyNOFAyZUjHWy0ZBjfcNh+iY2V3qIpG8kNPBntmvtVtemYI+UxEvvlalq0iEue+AtVmUoJeeO/lYuEDv+BrJGUWeZITbItuyoG+xGZM4x41X6FPgfRcnLCfii3dmwOeDZh1asR2Ahy2szdXrd2QaDeZMCCXNR+IeuP9U+IqJncfm+KjCqPABVbsZuGfDrT7r4XRv4mHQGnILD4IyA9N0/XyfY/cZRJryWeF5xbrOAP/ZWJduqYGIm2ahJfRZWIzk1Ujknq35iJ4bOZiw5eWb+aYhD9cxJj2UMBwWdqnXnpH+p55lrBqWlCzI0yDOfk4UQCI2RzQBJc/RvVS0z16k7El4dl9lOFTcgwBg13C/JC8Y0wo4M1jEap2ty/2ifwVZBZdflT8Uy5sk8dKBOv9wCupA/N60ACwlsavCUlgtQetVZUHIlI1U1rKCzArpGZ6PuREZokjdHw01u/j9ANEt1AaMe3G5jLcGohyvpcgL+zqI2S0rCqbSdRqWzZvr2kM8uFoeZSJLG3y4wdwZMgvRGMfoAKD1YlUGuhWeeZTda77ScM6ii8iEZEjvIgJXrfSCUf1vJ4QIAdMnNi05d3pTTDVHNTRjWBrklVb6oTpe/rFF1BBP6zhwGylWtb71xszbQyX8lzD+7v1O4iXr7ZWo86L9mGKz6FMnj0bO4LLVoqeVMy5DHo2fCuplix03P3WQCyjGKranAWdLalEV59QdnEhaQ==
template:
metadata:
creationTimestamp: null
name: kaneo-oidc-secret
namespace: kaneo

83
apps/kaneo/kaneo-postgres-deployment.yaml Normal file
View file

@ -0,0 +1,83 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: kaneo-postgres
namespace: kaneo
labels:
app: kaneo
component: database
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: kaneo
component: database
template:
metadata:
labels:
app: kaneo
component: database
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: csi.hetzner.cloud/location
operator: In
values:
- fsn1
containers:
- name: postgres
image: postgres:16-alpine
ports:
- name: postgres
containerPort: 5432
env:
- name: POSTGRES_DB
value: kaneo
- name: POSTGRES_USER
value: kaneo
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: kaneo-db-credentials
key: postgres-password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
livenessProbe:
exec:
command:
- pg_isready
- -U
- kaneo
- -d
- kaneo
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
exec:
command:
- pg_isready
- -U
- kaneo
- -d
- kaneo
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: kaneo-postgres-data

15
apps/kaneo/kaneo-postgres-pvc.yaml Normal file
View file

@ -0,0 +1,15 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: kaneo-postgres-data
namespace: kaneo
labels:
app: kaneo
component: database
spec:
accessModes:
- ReadWriteOnce
storageClassName: hcloud-volumes-encrypted
resources:
requests:
storage: 20Gi

17
apps/kaneo/kaneo-postgres-service.yaml Normal file
View file

@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: kaneo-postgres
namespace: kaneo
labels:
app: kaneo
component: database
spec:
type: ClusterIP
selector:
app: kaneo
component: database
ports:
- name: postgres
port: 5432
targetPort: 5432

17
apps/kaneo/kaneo-service.yaml Normal file
View file

@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: kaneo
namespace: kaneo
labels:
app: kaneo
component: app
spec:
type: ClusterIP
selector:
app: kaneo
component: app
ports:
- name: http
port: 80
targetPort: 5173

16
apps/kaneo/kaneo-smtp-sealed.yaml Normal file
View file

@ -0,0 +1,16 @@
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: kaneo-smtp
namespace: kaneo
spec:
encryptedData:
smtp-password: AgByMIhrSeQbPJezoqPf4jtTCx/b/R+TGZPHjjvUcdZJimgZY8XO72BA2nI/419UWqY3rBmWyquYMnEFJT0/48Rn0Nk4aFxBwBIxVuiG9y1+f5yXIhT2JuOqGoBnImYezCCtYOcgn4Byo6Twldce1ElRCqwH6grsVOLKtogZvLJC0sz6EWbqPfLrDuGyOFz2wg5P3Cjmy8W9ZgcTDTpVI4JxZh2JKbIJpq4kGDQD0miW8Av0BOmSi75uvT2PvaV/uinJ72VJFLP/GTHWjf/9kpOiIC2TL4B+Kk0HtMeWSzNF79itsD53ow7bya/bq//zUiwdaIeSaAcSJL8XqL9q38LIEL0Vr9ccXN9xsPZttLw836NDeLsryE+RhF3pIK2vEs76YIeO03AEnXNjEYOcwy7ePGYh6Y/Uf/B75QNzbzrq8H4B5lKFVmQgTBLJnvxTzOc6Sd/Yssgl1wJw90jgw2HYcLHXt/eQrB4cGKWSwrh2x6fnU79UpSGatdTqIyP2Q8vbgKmxqqmg6NDzjgeatT6ADKOqJKAjnXfrr8Bm3NjvAVWOGWuIfWdS0wHiEOhAoUNNAQ+bD7f/SbJE24q8W6juhzTmRLuU1BM2K51YPHA3BqyEpzygaboSi/bgHVOuoIAEAt7jAM6mbW5RQBwiNglF/C2Oem8J3LOdDlndJoSMaWKm5p1lKc3M59jr47jaDXm9HKM4ypC58cqty3varqxBwwdZHirZ6No0Ar4Sd78I4jo4kXk9zu33UjCcR/X2NX1abIYZ8eNoUga+I9AOkpM=
smtp-user: AgAq1mT9fzt9/VHmcbn0Yzr/8HWadEBe1XjJOTonao7Tdc3EO1PvCgEE9nBkrqOK5x2uDP1jZuP98XxJXr6mHuCnEOn8SHMN93TqK5rLhsKir1ImdNWKC+ruVI3TPQfTkeEz5m3u0DjHWSymva7BjmTIIW5gE4SDt/7WOFrsWbX1fLqQKfuB/k1qIBd+rFL7rVSe3XnRzxxwwrIbYfczSCknMbOot65FRcR+pC6ox7hOYC7pMIU76WMrVYx0cmDfAZAidcOQP9+K/H4LY0vkwPnaNk7+Is5gJifWR37lTnCkjRknMWB+yyDZF3BkmmdHNvXE6OlMcBF8l/jrO7S0rTtkCDaXllpRt1tWoSY2Kyr2DTaLt9QQvd1v8Rz2mw1qeDZg0mO/BswPF10SEKwjD3RlGFkA6mK9hZE5G+eDpgMH9u5HEKftI6jD/wdzElY+GdEFhlmj/ATZuNk+oC+eVnHGHUnY39lQYX+7PEtOujsKAjsuCPBaMDMkTP+f+LOUguuaRaanpLctmRAnT5BK19dmNuJtZPZAPgzCLu44884BH1qpKpVXkaUZSUbYA1RpKflcf1+uWJaqp7oBOZLSVoVSLiVG8ypGv7IPeTiILQcWQK27z6H0VxiAZ0S8yzNcCmeTWg8/wUGVzk0VJ9imPIShpiq515X6GuU1v1LsMiuJtFfL3EpNdDln04dLFWVb0ghIdt4Hn4U3IpJRIcP+XEEpv4+6
template:
metadata:
creationTimestamp: null
name: kaneo-smtp
namespace: kaneo
---

57
apps/monitoring/README.md Normal file
View file

@ -0,0 +1,57 @@
# monitoring — backup CronJobs
Manifests recording the cluster-side monitoring backup CronJobs that were previously applied out-of-band. These files are the authoritative source (`kubectl apply -f apps/monitoring/`). See [DEV-464](/DEV/issues/DEV-464) for the repair context.
- `backup-loki-restic-cronjob.yaml` — daily restic backup of `loki-storage-encrypted` to `hetzner-s3:${BUCKET}/restic/loki`. Co-schedules with the Loki pod via `podAffinity` (RWO permits additional read-only mounts on the same node). Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-485](/DEV/issues/DEV-485)).
- `backup-grafana-restic-cronjob.yaml` — daily restic backup of `grafana-storage` to `hetzner-s3:${BUCKET}/restic/grafana`. Pinned to `k3s-worker-2` via `nodeSelector` (the local-path PV anchors the grafana pod there already, no `podAffinity` needed). Schedule `15 3 * * *` — offset from the loki run at `03:00`. Deployed per the [DEV-482](/DEV/issues/DEV-482) Option 4 rollout ([DEV-486](/DEV/issues/DEV-486)).
- `prometheus-backup-cronjob.yaml` + `prometheus-backup-sealed.yaml` — daily restic backup of `prometheus-data-encrypted` to `hetzner-s3:${BUCKET}/restic/prometheus`. Co-schedules with the Prometheus pod via `podAffinity` so the RWO PVC attaches on the same node. Schedule `30 3 * * *` — offset from the loki (03:00) and grafana (03:15) runs. Migrated from the DEV-465 `rclone sync` job to restic client-side encryption per [DEV-492](/DEV/issues/DEV-492) / [DEV-482](/DEV/issues/DEV-482) Option 4. **Compaction-race mitigation:** `--exclude wal/*` + `--exclude chunks_head/*` + accept `restic backup` exit code 3 (source file vanished mid-walk) as a warning, not a failure; restore drill re-runs `promtool tsdb analyze` per block.
- `backup-restic-alerts.yaml` + `backup-restic-alerts.test.yaml` — `PrometheusRule` with freshness, integrity, and repo-size alerts covering the three restic repos (loki/grafana/prometheus), plus a `promtool test rules` unit test proving each alert fires against synthetic samples ([DEV-490](/DEV/issues/DEV-490), extended in [DEV-492](/DEV/issues/DEV-492)). The `k8s-resources` restic repo was superseded by Velero cluster-reconstruction backups and retired in [DEV-576](/DEV/issues/DEV-576).
## Shared SealedSecret
All four restic CronJobs read Hetzner S3 credentials + the restic repository password from **SealedSecret `monitoring-s3-backup`** (namespace `monitoring`). Keys:
| Key | Purpose |
|------------------|------------------------------------------------------------------------------------------------------------------|
| `access-key` | Hetzner Object Storage access key ID |
| `secret-key` | Hetzner Object Storage secret access key |
| `endpoint` | S3 endpoint hostname (e.g. `fsn1.your-objectstorage.com`) |
| `bucket` | Bucket name (single bucket, per-prefix repos) |
| `restic-password`| 32-byte random string sealed at [DEV-484](/DEV/issues/DEV-484); plaintext copy in Passbolt entry `restic / monitoring backups`. Rotation: `restic key add` → seal new value → `restic key remove` old id. Same key protects all three remaining repos (loki/grafana/prometheus) — rotating rewrites the key file on every repo. |
## Restore
Restore procedure for all four restic repos: [`docs/monitoring/restic-restore.md`](../../docs/monitoring/restic-restore.md). First drill (2026-08-16, loki/k8s-resources) passed — see [DEV-488](/DEV/issues/DEV-488). Prometheus repo added and drilled same day — see [DEV-492](/DEV/issues/DEV-492).
## Emitted metrics (textfile-collector format)
Each restic CronJob writes `/metrics/backup_<kind>.prom` (atomic — `.prom.tmp` + `mv`) into a `hostPath` volume mounted at the node-exporter textfile-collector directory (`/var/lib/node_exporter/textfile_collector`). The kube-prometheus-stack node-exporter DaemonSet has `--collector.textfile.directory=/host/textfile_collector` enabled ([DEV-494](/DEV/issues/DEV-494), applied via [`apps/observability/patches/node-exporter-textfile-collector.yaml`](../observability/patches/node-exporter-textfile-collector.yaml)) and surfaces those samples in Prometheus.
| Metric | Emitted by |
|--------------------------------------------|----------------------------------------------------------------------------------|
| `backup_<kind>_success` (0/1) | `backup-{loki,grafana}-*-cronjob.yaml`, `prometheus-backup-cronjob.yaml` |
| `backup_<kind>_timestamp_seconds` | same |
| `backup_<kind>_check_status` (exit code) | same — from `restic check --read-data-subset=5%` |
| `backup_prometheus_backup_status` | `prometheus-backup-cronjob.yaml` — `restic backup` exit code (3 = accepted compaction race) |
| `restic_repo_size_bytes{repo="<kind>"}` | same — from `restic stats --json --mode raw-data` (added in DEV-490) |
**Cross-node staleness note.** Because a backup CronJob may run on a different worker across days (loki/prometheus follow their app pods), a `.prom` file can linger on a node the job has since left and node-exporter keeps exposing it. The freshness alerts collapse the per-node samples with `max()` so the freshest sample wins; check/size alerts fire when *any* node reports a bad value, which is intentional — a recent failure is still a signal until the file is manually cleaned or the job returns to that node.
## Alerts
`backup-restic-alerts.yaml` defines seven alerts (all `severity: warning`):
- `BackupLokiStale` / `BackupGrafanaStale` / `BackupPrometheusStale` — `time() - max(backup_<kind>_timestamp_seconds) > 28h`. Daily schedule + 4 h grace. `max()` collapses per-node samples so a stale `.prom` on a node the job has left does not fire.
- `BackupLokiCheckFailed` / `BackupGrafanaCheckFailed` / `BackupPrometheusCheckFailed` — `backup_<kind>_check_status != 0`.
- `ResticRepoOversize` — `restic_repo_size_bytes > 20 GiB`. Baseline expected < 5 GiB; catches retention/prune regressions.
All alerts carry a `Runbook: docs/monitoring/restic-restore.md` annotation. To iterate on the rule file locally:
```bash
awk '/^spec:/{f=1;next} f{sub(/^ /,"");print}' \
apps/monitoring/backup-restic-alerts.yaml > /tmp/backup-restic-rules.yaml
promtool check rules /tmp/backup-restic-rules.yaml
promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
```
The legacy `backup-volumes` CronJob and its 100 Gi local-path `backup-storage` PVC were retired in [DEV-489](/DEV/issues/DEV-489) once the restic pipeline was proven end-to-end. With the shared destination PVC gone, the DEV-483 bridge `nodeSelector` pinning Loki to `k3s-worker-2` was also removed — `backup-loki-restic` follows the Loki pod via `podAffinity` regardless of which node the RWO CSI volume lands on. `backup-grafana-restic` still nodeSelects `k3s-worker-2` because its source PV (`grafana-storage`, local-path) is anchored there. `prometheus-backup` uses `podAffinity` on `app=prometheus` (RWO PVC on Hetzner CSI, single-node attach) and follows the Prometheus pod between workers. The `backup-k8s-resources` CronJob and its restic repo were retired in [DEV-576](/DEV/issues/DEV-576) after Velero `cluster-all` took over cluster-manifest coverage.

160
apps/monitoring/backup-grafana-restic-cronjob.yaml Normal file
View file

@ -0,0 +1,160 @@
---
# Grafana data backup via restic to Hetzner Object Storage (DEV-486,
# DEV-482 Option 4). Step 3 of the Option 4 rollout.
#
# Streams the RWO PVC `grafana-storage` (mounted read-only) into
# `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/grafana`, a client-side
# encrypted restic repository.
#
# `grafana-storage` is a local-path PV anchored on k3s-worker-2, so
# the grafana pod is already pinned there; a plain nodeSelector on
# the same host is enough (no podAffinity like the loki job needed).
apiVersion: batch/v1
kind: CronJob
metadata:
name: backup-grafana-restic
namespace: monitoring
labels:
app: backup
type: grafana
backend: restic
spec:
schedule: "15 3 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
metadata:
labels:
app: backup
type: grafana
backend: restic
spec:
backoffLimit: 2
activeDeadlineSeconds: 3600
template:
metadata:
labels:
app: backup
type: grafana
backend: restic
spec:
restartPolicy: OnFailure
nodeSelector:
kubernetes.io/hostname: k3s-worker-2
containers:
- name: restic
image: harbor.basicstack.de/library/restic:0.19.1
# Mirrored from docker.io/restic/restic:0.19.1 (DEV-493) —
# deterministic ingress via Harbor. Retag procedure in
# docs/monitoring/restic-restore.md § "Tag-bump procedure".
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: access-key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: secret-key
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: restic-password
- name: S3_ENDPOINT
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: endpoint
- name: S3_BUCKET
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: bucket
- name: RESTIC_REPOSITORY
value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/grafana"
command:
- /bin/sh
- -c
- |
set -eu
echo "=== backup-grafana-restic started at $(date -u +%FT%TZ) ==="
echo "Repository: ${RESTIC_REPOSITORY}"
# First-run tolerance: init if the repo isn't there yet.
if restic snapshots >/dev/null 2>&1; then
echo "Repo exists, skipping init."
else
echo "Repo missing, initialising..."
restic init
fi
echo "--- restic backup /source ---"
restic backup /source \
--tag grafana \
--host k3s \
--exclude '*.tmp'
echo "--- restic forget/prune ---"
restic forget --tag grafana \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
--prune
echo "--- restic check --read-data-subset=5% ---"
CHECK_STATUS=0
restic check --read-data-subset=5% || CHECK_STATUS=$?
echo "restic check exit: ${CHECK_STATUS}"
echo "--- restic stats (repo size) ---"
REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
| grep -oE '"total_size":[0-9]+' \
| head -1 \
| cut -d: -f2)
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics; identical wiring to the
# loki sibling. See that file for the atomic-write
# rationale (DEV-494).
{
echo "backup_grafana_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
echo "backup_grafana_timestamp_seconds $(date +%s)"
echo "backup_grafana_check_status ${CHECK_STATUS}"
echo "restic_repo_size_bytes{repo=\"grafana\"} ${REPO_SIZE_BYTES}"
} > /metrics/backup_grafana.prom.tmp
mv /metrics/backup_grafana.prom.tmp /metrics/backup_grafana.prom
echo "=== backup-grafana-restic finished at $(date -u +%FT%TZ) ==="
exit ${CHECK_STATUS}
volumeMounts:
- name: grafana-data
mountPath: /source
readOnly: true
- name: metrics
mountPath: /metrics
- name: cache
mountPath: /root/.cache/restic
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 1500m
memory: 1Gi
volumes:
- name: grafana-data
persistentVolumeClaim:
claimName: grafana-storage
- name: metrics
hostPath:
# node-exporter's textfile-collector directory
# (DEV-494). See sibling loki cronjob for detail.
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
- name: cache
emptyDir: {}

187
apps/monitoring/backup-loki-restic-cronjob.yaml Normal file
View file

@ -0,0 +1,187 @@
---
# Loki data backup via restic to Hetzner Object Storage (DEV-485,
# DEV-482 Option 4). Step 2 of the Option 4 rollout.
#
# Streams the RWO PVC `loki-storage-encrypted` (mounted read-only)
# into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/loki`, a client-side
# encrypted restic repository.
#
# podAffinity co-schedules with the Loki pod (app=loki, topology
# kubernetes.io/hostname). RWO permits additional read-only mounts
# on the node that holds the PVC's VolumeAttachment, so this
# survives Loki being rescheduled to a different worker.
apiVersion: batch/v1
kind: CronJob
metadata:
name: backup-loki-restic
namespace: monitoring
labels:
app: backup
type: loki
backend: restic
spec:
schedule: "0 3 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
metadata:
labels:
app: backup
type: loki
backend: restic
spec:
backoffLimit: 2
activeDeadlineSeconds: 3600
template:
metadata:
labels:
app: backup
type: loki
backend: restic
spec:
restartPolicy: OnFailure
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- loki
topologyKey: kubernetes.io/hostname
containers:
- name: restic
image: harbor.basicstack.de/library/restic:0.19.1
# Mirrored from docker.io/restic/restic:0.19.1 (DEV-493) —
# deterministic ingress via Harbor. Retag procedure in
# docs/monitoring/restic-restore.md § "Tag-bump procedure".
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: access-key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: secret-key
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: restic-password
- name: S3_ENDPOINT
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: endpoint
- name: S3_BUCKET
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: bucket
- name: RESTIC_REPOSITORY
value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/loki"
command:
- /bin/sh
- -c
- |
set -eu
echo "=== backup-loki-restic started at $(date -u +%FT%TZ) ==="
echo "Repository: ${RESTIC_REPOSITORY}"
# First-run tolerance: init if the repo isn't there yet.
# `restic cat config` is the tightest existence probe; use
# `snapshots` per plan spec — either exits 0 iff the repo
# is initialised.
if restic snapshots >/dev/null 2>&1; then
echo "Repo exists, skipping init."
else
echo "Repo missing, initialising..."
restic init
fi
echo "--- restic backup /source ---"
restic backup /source \
--tag loki \
--host k3s \
--exclude '*.tmp'
echo "--- restic forget/prune ---"
restic forget --tag loki \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
--prune
echo "--- restic check --read-data-subset=5% ---"
CHECK_STATUS=0
restic check --read-data-subset=5% || CHECK_STATUS=$?
echo "restic check exit: ${CHECK_STATUS}"
echo "--- restic stats (repo size) ---"
# `restic stats --json --mode raw-data` prints e.g.
# {"total_size":123,"total_file_count":45,...}. Extract
# total_size without jq (not present in the restic image)
# via grep/cut; fall back to 0 on empty output.
REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
| grep -oE '"total_size":[0-9]+' \
| head -1 \
| cut -d: -f2)
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics. Written to the shared
# host directory that node-exporter's textfile collector
# scrapes (DEV-494). Atomic write: build the file with a
# `.tmp` extension (ignored by node-exporter) and rename
# into place, so a mid-write read never surfaces a
# truncated sample.
{
echo "backup_loki_success $([ ${CHECK_STATUS} -eq 0 ] && echo 1 || echo 0)"
echo "backup_loki_timestamp_seconds $(date +%s)"
echo "backup_loki_check_status ${CHECK_STATUS}"
echo "restic_repo_size_bytes{repo=\"loki\"} ${REPO_SIZE_BYTES}"
} > /metrics/backup_loki.prom.tmp
mv /metrics/backup_loki.prom.tmp /metrics/backup_loki.prom
echo "=== backup-loki-restic finished at $(date -u +%FT%TZ) ==="
exit ${CHECK_STATUS}
volumeMounts:
- name: loki-data
mountPath: /source
readOnly: true
- name: metrics
mountPath: /metrics
- name: cache
mountPath: /root/.cache/restic
resources:
# Requests deliberately lowered from the plan doc's
# 200m/256Mi — worker-2 (Loki node) has ~150m free CPU
# and podAffinity forces us onto it. 100m/128Mi mirrors
# the sibling backup CronJobs; limits stay generous so
# restic can burst during pack/check.
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 1500m
memory: 1Gi
volumes:
- name: loki-data
persistentVolumeClaim:
claimName: loki-storage-encrypted
- name: metrics
hostPath:
# node-exporter's textfile-collector directory
# (DEV-494). `DirectoryOrCreate` lets kubelet create
# the dir on the current node if it does not yet
# exist — nodes were pre-created out-of-band, this
# is a safety net.
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
- name: cache
emptyDir: {}

216
apps/monitoring/backup-restic-alerts.test.yaml Normal file
View file

@ -0,0 +1,216 @@
---
# promtool test file for backup-restic-alerts.yaml (DEV-490 DoD).
#
# Extract the rules from the PrometheusRule wrapper first:
# awk '/^spec:/{f=1;next} f{sub(/^ /,"");print}' \
# apps/monitoring/backup-restic-alerts.yaml > /tmp/rules.yaml
# Then run:
# promtool test rules apps/monitoring/backup-restic-alerts.test.yaml
# (this file uses `rule_files: [/tmp/rules.yaml]` — pass the extracted
# path explicitly).
rule_files:
- /tmp/backup-restic-rules.yaml
# Evaluation cadence for the test scenarios. The alerts' `for:` uses 5m/15m/30m,
# so we run 60m of series and evaluate at 60m so all alerts have had time to
# settle into the firing state.
evaluation_interval: 1m
tests:
# ---------------------------------------------------------------------------
# 1. Freshness alerts fire when timestamp is stale (>28h old).
# ---------------------------------------------------------------------------
- interval: 1m
input_series:
# Timestamp value = "current time - 29h" in Unix seconds. During the
# test, promtool's clock starts at the Unix epoch (0). We fix the
# value to a constant far in the past so `time() - value > 28*3600`
# for the entire test run.
#
# Concretely: time() returns the sample timestamp in seconds. Over a
# 60-minute test starting at t=0, time() runs 0 .. 3600. A constant
# value of -104400 (=-29h) keeps `time() - value` >= 29h across the
# whole window, safely above the 28h threshold.
- series: 'backup_loki_timestamp_seconds'
values: '-104400x60'
- series: 'backup_grafana_timestamp_seconds'
values: '-104400x60'
- series: 'backup_prometheus_timestamp_seconds'
values: '-104400x60'
alert_rule_test:
- eval_time: 30m
alertname: BackupLokiStale
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: loki
exp_annotations:
summary: "Loki restic backup is stale (>28h)"
description: |
`backup-loki-restic` has not written a fresh
`backup_loki_timestamp_seconds` sample in more than
28 hours. Expected daily at 03:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- eval_time: 30m
alertname: BackupGrafanaStale
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: grafana
exp_annotations:
summary: "Grafana restic backup is stale (>28h)"
description: |
`backup-grafana-restic` has not written a fresh
`backup_grafana_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:15 UTC.
Runbook: docs/monitoring/restic-restore.md
- eval_time: 30m
alertname: BackupPrometheusStale
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
exp_annotations:
summary: "Prometheus restic backup is stale (>28h)"
description: |
`prometheus-backup` has not written a fresh
`backup_prometheus_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:30 UTC.
Runbook: docs/monitoring/restic-restore.md
# ---------------------------------------------------------------------------
# 2. Freshness alerts stay silent when timestamp is fresh (<28h).
# ---------------------------------------------------------------------------
- interval: 1m
input_series:
# Value = -3600 (=-1h) → time() - value stays around 1-2h, well below
# the 28h threshold across the run.
- series: 'backup_loki_timestamp_seconds'
values: '-3600x60'
- series: 'backup_grafana_timestamp_seconds'
values: '-3600x60'
- series: 'backup_prometheus_timestamp_seconds'
values: '-3600x60'
alert_rule_test:
- eval_time: 30m
alertname: BackupLokiStale
exp_alerts: []
- eval_time: 30m
alertname: BackupGrafanaStale
exp_alerts: []
- eval_time: 30m
alertname: BackupPrometheusStale
exp_alerts: []
# ---------------------------------------------------------------------------
# 3. Check-status alerts fire when the metric is non-zero.
# ---------------------------------------------------------------------------
- interval: 1m
input_series:
- series: 'backup_loki_check_status'
values: '1x30'
- series: 'backup_grafana_check_status'
values: '2x30'
- series: 'backup_prometheus_check_status'
values: '1x30'
alert_rule_test:
- eval_time: 15m
alertname: BackupLokiCheckFailed
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: loki
exp_annotations:
summary: "restic check failed on loki repo"
description: |
`restic check --read-data-subset=5%` returned
exit code 1 on the loki repository.
Runbook: docs/monitoring/restic-restore.md
- eval_time: 15m
alertname: BackupGrafanaCheckFailed
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: grafana
exp_annotations:
summary: "restic check failed on grafana repo"
description: |
`restic check --read-data-subset=5%` returned
exit code 2 on the grafana repository.
Runbook: docs/monitoring/restic-restore.md
- eval_time: 15m
alertname: BackupPrometheusCheckFailed
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
exp_annotations:
summary: "restic check failed on prometheus repo"
description: |
`restic check --read-data-subset=5%` returned
exit code 1 on the prometheus
repository.
Runbook: docs/monitoring/restic-restore.md
# ---------------------------------------------------------------------------
# 4. Check-status alerts stay silent on 0.
# ---------------------------------------------------------------------------
- interval: 1m
input_series:
- series: 'backup_loki_check_status'
values: '0x30'
- series: 'backup_grafana_check_status'
values: '0x30'
- series: 'backup_prometheus_check_status'
values: '0x30'
alert_rule_test:
- eval_time: 15m
alertname: BackupLokiCheckFailed
exp_alerts: []
- eval_time: 15m
alertname: BackupGrafanaCheckFailed
exp_alerts: []
- eval_time: 15m
alertname: BackupPrometheusCheckFailed
exp_alerts: []
# ---------------------------------------------------------------------------
# 5. Repo-size alert fires when >20 GiB (21474836480 bytes).
# ---------------------------------------------------------------------------
- interval: 1m
input_series:
# 25 GiB = 26843545600 bytes.
- series: 'restic_repo_size_bytes{repo="loki"}'
values: '26843545600x45'
# 15 GiB = 16106127360 bytes (below threshold).
- series: 'restic_repo_size_bytes{repo="grafana"}'
values: '16106127360x45'
alert_rule_test:
- eval_time: 40m
alertname: ResticRepoOversize
exp_alerts:
- exp_labels:
severity: warning
service: monitoring
component: backup
repo: loki
exp_annotations:
summary: "restic repository loki exceeds 20 GiB"
description: |
Repository loki is
25GiB, above the 20 GiB
guard. Baseline is <5 GiB per repo. Investigate
retention/prune (see forget flags in the CronJob)
and dedup effectiveness.

186
apps/monitoring/backup-restic-alerts.yaml Normal file
View file

@ -0,0 +1,186 @@
---
# Prometheus alerting rules for the restic-based monitoring backups
# (DEV-490 / DEV-482 Option 4). Freshness alerts fire when a
# CronJob has not emitted its `backup_<kind>_timestamp_seconds`
# metric within 28h (schedules are daily; 28h gives one missed
# run + 4h grace before we page).
#
# Check-status alerts fire on the first non-zero exit from
# `restic check --read-data-subset=5%`.
#
# Repo-size alerts fire when a repository exceeds 20 GiB — the
# baseline is expected < 5 GiB per repo. The alert catches
# retention/prune bugs and runaway growth. The metric is emitted
# by the same CronJob step (see backup-*-restic-cronjob.yaml,
# `restic stats --json --mode raw-data`).
#
# The metrics are written to the node's textfile-collector directory
# (`/var/lib/node_exporter/textfile_collector`) and scraped by the
# kube-prometheus-stack node-exporter DaemonSet (DEV-494).
#
# Because a CronJob may run on a different worker across days (loki
# backup follows the loki pod), stale
# `.prom` files can linger on nodes the job has since left. That
# would leave a per-node series with an old timestamp/check-status
# indefinitely. The alerts below aggregate across instances so a
# single fresh sample from the node where the job currently runs is
# enough to keep the freshness alert quiet, and check/size alerts
# fire when *any* node reports a bad value (which is the correct
# behaviour — a recent failure is still a signal).
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: backup-restic-alerts
namespace: monitoring
labels:
app: backup
backend: restic
release: kube-prometheus-stack
spec:
groups:
- name: backup-restic.freshness
interval: 60s
rules:
- alert: BackupLokiStale
# `max()` collapses per-node samples so a stale `.prom` file
# left behind on a node the loki backup no longer runs on
# does not fire this alert; the freshest sample wins.
expr: time() - max(backup_loki_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: loki
annotations:
summary: "Loki restic backup is stale (>28h)"
description: |
`backup-loki-restic` has not written a fresh
`backup_loki_timestamp_seconds` sample in more than
28 hours. Expected daily at 03:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupGrafanaStale
expr: time() - max(backup_grafana_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: grafana
annotations:
summary: "Grafana restic backup is stale (>28h)"
description: |
`backup-grafana-restic` has not written a fresh
`backup_grafana_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:15 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupPrometheusStale
expr: time() - max(backup_prometheus_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
annotations:
summary: "Prometheus restic backup is stale (>28h)"
description: |
`prometheus-backup` has not written a fresh
`backup_prometheus_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:30 UTC.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupForgejoStale
expr: time() - max(backup_forgejo_timestamp_seconds) > 28 * 3600
for: 15m
labels:
severity: warning
service: monitoring
component: backup
repo: forgejo
annotations:
summary: "Forgejo restic backup is stale (>28h)"
description: |
`forgejo-backup` has not written a fresh
`backup_forgejo_timestamp_seconds` sample in more
than 28 hours. Expected daily at 03:00 UTC.
Runbook: docs/monitoring/restic-restore.md
- name: backup-restic.integrity
interval: 60s
rules:
- alert: BackupLokiCheckFailed
expr: backup_loki_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: loki
annotations:
summary: "restic check failed on loki repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the loki repository.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupGrafanaCheckFailed
expr: backup_grafana_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: grafana
annotations:
summary: "restic check failed on grafana repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the grafana repository.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupPrometheusCheckFailed
expr: backup_prometheus_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: prometheus
annotations:
summary: "restic check failed on prometheus repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the prometheus
repository.
Runbook: docs/monitoring/restic-restore.md
- alert: BackupForgejoCheckFailed
expr: backup_forgejo_check_status != 0
for: 5m
labels:
severity: warning
service: monitoring
component: backup
repo: forgejo
annotations:
summary: "restic check failed on forgejo repo"
description: |
`restic check --read-data-subset=5%` returned
exit code {{ $value }} on the forgejo
repository.
Runbook: docs/monitoring/restic-restore.md
- name: backup-restic.size
interval: 60s
rules:
- alert: ResticRepoOversize
# 20 GiB = 20 * 1024^3 = 21474836480 bytes.
expr: restic_repo_size_bytes > 21474836480
for: 30m
labels:
severity: warning
service: monitoring
component: backup
annotations:
summary: "restic repository {{ $labels.repo }} exceeds 20 GiB"
description: |
Repository {{ $labels.repo }} is
{{ $value | humanize1024 }}B, above the 20 GiB
guard. Baseline is <5 GiB per repo. Investigate
retention/prune (see forget flags in the CronJob)
and dedup effectiveness.

52
apps/monitoring/loki-deployment.yaml Normal file
View file

@ -0,0 +1,52 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: loki
namespace: monitoring
labels:
app: loki
spec:
replicas: 1
selector:
matchLabels:
app: loki
# loki-storage-encrypted is RWO on hcloud-volumes-encrypted, so a rolling
# update deadlocks (new pod cannot attach the PVC while the old pod holds
# it). Recreate drops the old pod first so the CSI detaches the volume
# before the new pod tries to attach it. Same pattern as harbor's RWO fix.
strategy:
type: Recreate
template:
metadata:
labels:
app: loki
spec:
containers:
- name: loki
image: grafana/loki:2.9.2
args:
- -config.file=/etc/loki/loki.yaml
ports:
- containerPort: 3100
name: http
- containerPort: 9096
name: grpc
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
cpu: 500m
memory: 1Gi
volumeMounts:
- name: loki-config
mountPath: /etc/loki
- name: loki-storage
mountPath: /loki
volumes:
- name: loki-config
configMap:
name: loki-config
- name: loki-storage
persistentVolumeClaim:
claimName: loki-storage-encrypted

228
apps/monitoring/prometheus-backup-cronjob.yaml Normal file
View file

@ -0,0 +1,228 @@
---
# Prometheus data backup via restic to Hetzner Object Storage
# (DEV-492, DEV-482 Option 4). Replaces the DEV-465 rclone-sync job so
# every monitoring backup ships client-side-encrypted; Hetzner Object
# Storage has no SSE-S3/SSE-KMS, so the previous plaintext-at-rest
# object layout was the only remaining gap.
#
# Streams the RWO PVC `prometheus-data-encrypted` (mounted read-only)
# into `s3:${S3_ENDPOINT}/${S3_BUCKET}/restic/prometheus`, a client-side
# encrypted restic repository (tag=`prometheus`, host=`k3s`).
#
# Node scheduling matches the previous job: podAffinity co-schedules
# with the Prometheus pod (app=prometheus, topology
# kubernetes.io/hostname). Hetzner CSI RWO permits additional read-only
# mounts on the node that holds the PVC's VolumeAttachment, so this
# survives Prometheus being rescheduled to a different worker.
#
# Prometheus TSDB compaction race
# --------------------------------
# Prometheus rewrites the on-disk store roughly every 2 h: it creates a
# new block dir, then deletes the source dirs. restic walks the source
# tree once and may catch a file that disappeared mid-walk; restic
# 0.17.3 exits 3 ("at least one source file could not be read") in
# that case, and the snapshot excludes only the missing file. The
# next daily run picks up the successor block, so the race is not a
# data-loss risk — but we must not treat exit 3 as a hard failure, or
# the daily job will alert-flap.
#
# Mitigation:
# - exclude `wal/*` (WAL is replayed from a fresh instance on
# restart; we accept losing the last ~15 s of ingested samples
# rather than snapshotting a moving segment)
# - exclude `chunks_head/*` (in-memory head block; ephemeral, would
# be rebuilt from WAL which we do not keep)
# - exclude Prometheus lock/scratch files (`lock`, `queries.active`,
# `*.tmp`, `lost+found/*`)
# - treat restic exit code 3 as a soft warning (log, continue);
# any other non-zero exit is still fatal
# - restore drill re-runs `promtool tsdb analyze` against every
# block so a corrupted snapshot is caught end-to-end
apiVersion: batch/v1
kind: CronJob
metadata:
name: prometheus-backup
namespace: monitoring
labels:
app: backup
type: prometheus
backend: restic
spec:
schedule: "30 3 * * *" # daily 03:30, offset from loki/grafana
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
metadata:
labels:
app: backup
type: prometheus
backend: restic
spec:
backoffLimit: 2
activeDeadlineSeconds: 3600
template:
metadata:
labels:
app: backup
type: prometheus
backend: restic
spec:
restartPolicy: OnFailure
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- prometheus
topologyKey: kubernetes.io/hostname
containers:
- name: restic
image: harbor.basicstack.de/library/restic:0.19.1
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: access-key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: secret-key
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: restic-password
- name: S3_ENDPOINT
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: endpoint
- name: S3_BUCKET
valueFrom:
secretKeyRef:
name: monitoring-s3-backup
key: bucket
- name: RESTIC_REPOSITORY
value: "s3:$(S3_ENDPOINT)/$(S3_BUCKET)/restic/prometheus"
command:
- /bin/sh
- -c
- |
set -eu
echo "=== backup-prometheus-restic started at $(date -u +%FT%TZ) ==="
echo "Repository: ${RESTIC_REPOSITORY}"
# First-run tolerance: init if the repo isn't there yet.
if restic snapshots >/dev/null 2>&1; then
echo "Repo exists, skipping init."
else
echo "Repo missing, initialising..."
restic init
fi
# See "Prometheus TSDB compaction race" in the file
# header. WAL + chunks_head are excluded on purpose;
# exit code 3 (source file vanished mid-walk during a
# compaction) is accepted and reported, any other
# non-zero exit is fatal.
echo "--- restic backup /source (exclude wal/chunks_head + lock files) ---"
BACKUP_STATUS=0
restic backup /source \
--tag prometheus \
--host k3s \
--exclude 'wal/*' \
--exclude 'chunks_head/*' \
--exclude 'lock' \
--exclude 'queries.active' \
--exclude 'lost+found/*' \
--exclude '*.tmp' || BACKUP_STATUS=$?
echo "restic backup exit: ${BACKUP_STATUS}"
if [ "${BACKUP_STATUS}" -eq 0 ]; then
echo "backup: all files captured cleanly"
elif [ "${BACKUP_STATUS}" -eq 3 ]; then
echo "backup: exit 3 (source files vanished mid-walk) — expected under Prometheus compaction, continuing"
else
echo "backup: FATAL exit ${BACKUP_STATUS} (not compaction-race)"
exit ${BACKUP_STATUS}
fi
echo "--- restic forget/prune ---"
restic forget --tag prometheus \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
--prune
echo "--- restic check --read-data-subset=5% ---"
CHECK_STATUS=0
restic check --read-data-subset=5% || CHECK_STATUS=$?
echo "restic check exit: ${CHECK_STATUS}"
echo "--- restic stats (repo size) ---"
REPO_SIZE_BYTES=$(restic stats --json --mode raw-data 2>/dev/null \
| grep -oE '"total_size":[0-9]+' \
| head -1 \
| cut -d: -f2)
REPO_SIZE_BYTES=${REPO_SIZE_BYTES:-0}
echo "restic repo size: ${REPO_SIZE_BYTES} bytes"
# Textfile-collector metrics. Same hostPath pattern as
# the loki/grafana siblings (DEV-494) —
# written atomically via `.tmp` + rename so a mid-write
# read never surfaces a truncated sample.
# backup_prometheus_success rolls in both stages: the
# backup step (accepting exit 3) and restic check.
BACKUP_OK=0
if [ "${BACKUP_STATUS}" -eq 0 ] || [ "${BACKUP_STATUS}" -eq 3 ]; then
BACKUP_OK=1
fi
SUCCESS=0
if [ "${BACKUP_OK}" -eq 1 ] && [ "${CHECK_STATUS}" -eq 0 ]; then
SUCCESS=1
fi
{
echo "backup_prometheus_success ${SUCCESS}"
echo "backup_prometheus_timestamp_seconds $(date +%s)"
echo "backup_prometheus_check_status ${CHECK_STATUS}"
echo "backup_prometheus_backup_status ${BACKUP_STATUS}"
echo "restic_repo_size_bytes{repo=\"prometheus\"} ${REPO_SIZE_BYTES}"
} > /metrics/backup_prometheus.prom.tmp
mv /metrics/backup_prometheus.prom.tmp /metrics/backup_prometheus.prom
echo "=== backup-prometheus-restic finished at $(date -u +%FT%TZ) ==="
exit ${CHECK_STATUS}
volumeMounts:
- name: prometheus-data
mountPath: /source
readOnly: true
- name: metrics
mountPath: /metrics
- name: cache
mountPath: /root/.cache/restic
resources:
# Prometheus TSDB is ~8-10 GiB; give restic room to
# burst during pack/check but keep steady-state small.
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 1500m
memory: 1Gi
volumes:
- name: prometheus-data
persistentVolumeClaim:
claimName: prometheus-data-encrypted
- name: metrics
hostPath:
# node-exporter's textfile-collector directory
# (DEV-494). See sibling loki cronjob for detail.
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
- name: cache
emptyDir: {}

20
apps/monitoring/prometheus-backup-sealed.yaml Normal file
View file

@ -0,0 +1,20 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: monitoring-s3-backup
namespace: monitoring
spec:
encryptedData:
access-key: AgBf1XxCF3/Uv1vVlJHyTSEMONqWBqPfSOax8jJJHy3NOpgGga3SsxDfAbPXe2rBOYSgGzRZcaT8TcNg8Ng1ipv5Dhu7LzmRb6v6fW0NimrHSU/dtzzJSIR7jQ1sf+Qn4CPS73ReDRvUJ7g4o57ClZIVFBk9oPUDJYdjtg9kAM5Me0ZyqVp4Cnyj2TRXYcQPM+RN1r/PXqK4MhlUsmhsbQCPYwocM0liqv0T8L5xR1g2845oMaMx2nmLs1c1R8nCRugK8IeBKf/CCKLYg8yTqWKS2arrfxm+dkl1mrqLdOgnXrqeeo/63lT1sbd06ub86PJr83MXWW0yh1Uc4r7DX7xK/JIGJ4w+jtQa0vRutYANDcdeVlzjVl+ghoxxw/ZizYVQ/mUFCc+ApJD9u709mI39xof77U5yYBRhnZEExRseL756APubmCc0tPfUSNA1GmMhiYaMnyWu6F5Okreto/03AgI3t7lQm9SPRRM7hcHETG/QC8PrZ0NZEN/A2Nm/ZzwDG4HPDaNxH+IV90f5X4wuh6Mh2lffidOix88rkaizQAeVY5gt8tvfSfyfRc5RoOOov+fvUVCqJHnWaD1A2lKScVeCgLv8/0CJTyR0zM6cg2XN8z0HvU/sahfZrG23fVs3E/IFM5D7//CDp+GfcwQOCBZKPLvJtTRh5nA5EbXQusGEaMOTBiiGG0IJKPejKcPRWS1g+owu+G5R6QQjZgNDoP0cPg==
bucket: AgCYDCrQalif3+XZYhe4Xxyypg2OMGvzdNyBmHzMaP+ou1g2Q0wIjH6S1+YpPu+JgmaATZVISoSZ+OIl0mNOibiFMr0kFjuMhvKRt9Pq05+6axnqnJ17U2GNsZQtY3Ic93qxkkhJKCOQ87mvAJ5AIVBc3pAqfkD5YYTllgnSKwUwyaWK3iX3hxuKLYDvsQ2WTtcc/gsnlCkm1dwPE4E7DFt4wh+jQJRD314Mt7KSpvPBpz7yVVFFX+xRKCfRxjcwshUSUI5OgagfOQvsivkxlL0oSqA9veX2MJpd0SJH0T1Y0na0evkJitxxe+ECwM4FluK/iwW4cdUu+EpQIdd5otY9C5b8e8PJp8ah1k1hYzXdX+j975MeyXWMUn/NgXfyT2CUef6hmmmImialrSYsHT72bz3xpAnZ/oQ/0qcZpNje2oJeokNgkoiC0E5XRnfwEF9jL3XL+aB6gQ3BD9YRptaxLLxOQNj51Oe82vVsOnUz+utzJO1DtqpF+8I0lvHfgcUhFQytUuLscilfBQO/ru8wh47ZI+LRiS3qCZp2Pqi9+r7Y7jJKcbb2awWp6lgrrqgLMvnbVMuvxBi2hUy0C+EkP8tayRJDzz72utx8FY/v0FWoRDl9BAi8kn11+kbbo8a2OXe+20UkmNzunz5H0ijIxhJWzVh/oLuuLFCS0YJtJNjuqUB0VctAFcR4p5Agua8LA9+73HIBsHBYexzmvEZ6Ng==
endpoint: AgCV6YtzikWYMRwHZM1tT8epq6XiSVqhbxMhS7htVjVUfdR4V7/KIcNExxSBul+TU+RCq3S7fppL7CxKbI/IsEdirRMj2PvESXEmzLY/xId7+wJFEdxeXxn59cbV2MfCyysKIwzsVXbmzO66yGaeMKU2oyqJVYu+00L9xS0RbtQRbi0UksMxtw0PhZVMee0aHfu9jynWI3lmPzC/EPRuwxQqBIjYM2v2JCDmYjP0fsBSZSJc+SnAaNmmIf7ypcFwy14qIF7C9tIg02lBx+Sg30oMPjn600AbyF6RmZ+mdclp/Qp+zXOUsbMym0Lu9BjbM7G/z0ExpR5sgE/6+05gJhPF7MuHkrYIMo9LHWWdgzNTnjikSTGEeuFzzRlThPMoUfVi/S7KhG9NtQRnNqEoVCGGW6tt5sygXVE/lWSnmmd//4jXvF9ZJHU+5LrjL7UfVIYNH4y7aYtXpoTgmUe85wWqeMdqr1EiSVueoSwCggGITYoXoKYfERfij8ln/xnX1Kt2xxuyLtpDOczBYZK6LLwTpaXVVe+lO/xBgevWaoEz6si2hyj+glluMQWxPsOnPwUfRi5qqlQxgMZK/9aALYhPLquc16iMG+4FdtoOUfQ0XlLvurHsbCt9lUy3YrsmpDhv42FfamVlcK+gMz7mqYhNhd3fVjxCiUuwvzd01QHcx8ikv48YOnDuRiweRszbPNvlaK2KH6fo7b4l9mmaa4W+MaxZCxG3KSMOafwRmyusEUVNWw==
secret-key: AgBItdfzRgcgJvK5HALDnkbGH1DveAgcmS+Q4f16VwX1KY8TzGWpjoaOdsLCwzTWOFME5xjrRyOwwAcdtP65ccWlEZvvoS4kpqv49G1ebblD1wmApnpa8vD63Rd028vUYFF3CnwntX2USxF6t4rD68sZqqzMDyuG9Av93HAKVMNdx9V4AJ45nvfF2WR7ohk37NhQifOdKuelgRV6ahl3/fm/XPRGzf+u7OER8jll9n/ywE0mSkoKBXocbkQ2Xrn+utzplh9BoR9KSPTdnmI8ZXUkDTR9p+7PN4r/HheEdTUG0aaefdEDCbdDlv1EW84l5KzhY4v9g0DW8GQQdyHWi3sMm0hSGFg9NV/Vs1HM9FZZCyjS+mzR4CbuHgkY9mvj+Svn7oS+Fy5d24s1ekmPheakBffXKy2Q2CLMRGNymjUIBQsDwoGC8o4IF+yPyDcw/OvXlr0lruxfRM6IN0oxgRmLAQiqwRzsZb3RPgILEZI9VXKxzCBEuCO7fE2pC05K/mujhUutWmJ5tQ0NFzP1mhNK59cAIVMl7y7Llq5/GZ5ax+voP0c3VandXJeaSRr6O40vyzfCMgpgEgRs9t2WfxCmVYEN2NWpFv07JNICWzJX40PLT9n/euRXzSmJld6UfjzRteaTfmo+OxFczJ6Zl8XdC+6x6wO0Nk0hVYo27s2LH4o0MsM84x9Z3StCCGl77/NnyavpxfDkj3qJQ4BgPtCw26tHTt9DIWYfgeAVqvoRZVlg8YxM3N1k
restic-password: AgBuoBFK5Y2Vlf1Vj6Q7y5OwWV2iOLMATe+Kjca8uwHgiYnpka+619ab52P1P+tkxbGyGdnwF+3q0KzvJJaNfuoduwlPuRjiYmaLLoCuQahZmTLNbzwEFst1cbm4oG6vCTBGsK9IMngT6VMfmvz029X+jsGaJHaECHlJjWnMteKN2pwxbzt4/3Wx6wP3//qAk1ah4QLRZUEWvn6O7djTFWJtzM4MRfZPI6EmM9emlQRN9iHwWSUmgBZ5zvaAylRHHUHK1lkNgWJJr7+tC4D8DbZ+c4+FO6XefXDheG6jO2Ial09KTISC0+v86xKFq/aSko5TKczpdXEdYNfyf/aPJTRd4LTcTySUdIpflmjIUnJBdn2Yn4Jox8LKc6ecUJ+IxMXhWlToFNtSFJwx744uv9WsrQ7p/lH7VTZOT+R+0wKTe9vn07h6B78xX6JJWKHMepb9aJB0gB1fn9cx1KHzm4/Y30SFwuOpYpVx19lyl1BRfSry4kmPuDKouSO6D415NJ/lMNYqTvafRho5SzlFfle/n7wAK7OIRdNbgsabO2mfHjHpw6w8Bvo9K7+3QJiwpeesY8OKAubcpxy1RBDw3fp2JQCNNqIer41vUtprsRU+tBnzDboImhOTXpP71AgTa+L68k/iGVJB0YqjFDxjcJB1f1hKGvH+yCV0GPFSeUFOZ/gmxV+Q0Fmoq/ezLrqpjoTbjp+ZbAi6qbVrMN/3t3lheKf47dU5HEOQMyZkXqDrkyuc7kXhmzN/V/ISmW0=
template:
metadata:
creationTimestamp: null
name: monitoring-s3-backup
namespace: monitoring
type: Opaque

110
apps/monitoring/prometheus-deployment.yaml Normal file
View file

@ -0,0 +1,110 @@
---
# Prometheus data volume — RWO on hcloud-volumes-encrypted (fsn1). Because the
# volume is RWO, the Deployment below must use Recreate; a rolling update would
# spin up a surge pod that cannot mount the volume while the old pod holds it
# and the rollout would stall at Init:0/1. See DEV-583 / mirrors DEV-538.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-data-encrypted
namespace: monitoring
spec:
accessModes:
- ReadWriteOnce
storageClassName: hcloud-volumes-encrypted
resources:
requests:
storage: 50Gi
---
apiVersion: v1
kind: Service
metadata:
name: prometheus
namespace: monitoring
labels:
app: prometheus
spec:
type: ClusterIP
ports:
- port: 9090
targetPort: 9090
name: web
selector:
app: prometheus
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: monitoring
labels:
app: prometheus
spec:
replicas: 1
# prometheus-data-encrypted is RWO on hcloud-volumes-encrypted, so a rolling
# update deadlocks (new pod cannot attach the PVC while the old pod holds
# it). Recreate drops the old pod first so the CSI detaches the volume
# before the new pod tries to attach it. Same pattern as loki/opencloud.
strategy:
type: Recreate
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
annotations:
# Exclude the prometheus TSDB volume from Velero fs-backup — TSDB is
# backed up separately by prometheus-backup-cronjob and streaming it
# through Kopia would double-copy tens of GiB on every backup.
backup.velero.io/backup-volumes-excludes: prometheus-storage
spec:
serviceAccountName: prometheus
initContainers:
- name: fix-permissions
image: busybox:latest
command:
- sh
- -c
- chown -R 65534:65534 /prometheus && chmod -R 755 /prometheus
volumeMounts:
- name: prometheus-storage
mountPath: /prometheus
containers:
- name: prometheus
image: prom/prometheus:v2.47.0
args:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=30d'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
- '--web.enable-lifecycle'
ports:
- containerPort: 9090
name: web
volumeMounts:
- name: prometheus-config
mountPath: /etc/prometheus
- name: prometheus-rules
mountPath: /etc/prometheus/rules
- name: prometheus-storage
mountPath: /prometheus
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 1000m
memory: 2Gi
volumes:
- name: prometheus-config
configMap:
name: prometheus-config
- name: prometheus-rules
configMap:
name: prometheus-rules
- name: prometheus-storage
persistentVolumeClaim:
claimName: prometheus-data-encrypted

38
apps/observability/patches/README.md Normal file
View file

@ -0,0 +1,38 @@
# observability patches
Strategic-merge patches applied on top of Helm-managed observability
resources. Each file is idempotent (re-applying is a no-op) and is
reasserted by hand rather than by a controller, so re-run after any
`helm upgrade` of the affected release.
## `node-exporter-textfile-collector.yaml` (DEV-494)
Enables the node-exporter textfile collector on the
`kube-prometheus-stack-prometheus-node-exporter` DaemonSet by:
1. adding `--collector.textfile.directory=/host/textfile_collector`
to the container args, and
2. mounting the host directory `/var/lib/node_exporter/textfile_collector`
read-only at `/host/textfile_collector` (`hostPath` type
`DirectoryOrCreate`, kubelet creates it on nodes where the
directory does not exist yet).
The four monitoring backup CronJobs in `apps/monitoring/` write their
textfile-collector `.prom` files into that same host directory, so the
metrics surface in Prometheus via node-exporter's normal scrape.
Apply / re-apply:
```bash
kubectl -n observability patch daemonset \
kube-prometheus-stack-prometheus-node-exporter \
--type=strategic \
--patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
kubectl -n observability rollout status daemonset \
kube-prometheus-stack-prometheus-node-exporter
```
The kube-prometheus-stack chart is not currently tracked in ArgoCD;
if it moves under GitOps, fold these values into the chart values as
`prometheus-node-exporter.extraArgs` + `.extraHostVolumeMounts`
instead of maintaining this patch.

70
apps/observability/patches/node-exporter-textfile-collector.yaml Normal file
View file

@ -0,0 +1,70 @@
---
# Strategic-merge patch enabling node-exporter's textfile collector on
# the kube-prometheus-stack node-exporter DaemonSet (DEV-494).
#
# The chart is Helm-managed (release `kube-prometheus-stack` in
# namespace `observability`, chart kube-prometheus-stack-86.2.2 /
# prometheus-node-exporter-4.55.0) and is NOT currently tracked in
# ArgoCD, so a direct DaemonSet patch is the pragmatic wiring path.
# The change:
#
# 1. adds the `--collector.textfile.directory=/host/textfile_collector`
# arg to the node-exporter container, and
# 2. mounts the host directory `/var/lib/node_exporter/textfile_collector`
# read-only at `/host/textfile_collector` (type DirectoryOrCreate so
# kubelet creates the dir on nodes where it does not yet exist).
#
# Apply with:
# kubectl -n observability patch daemonset \
# kube-prometheus-stack-prometheus-node-exporter \
# --type=strategic \
# --patch-file=apps/observability/patches/node-exporter-textfile-collector.yaml
#
# If the Helm release is ever `helm upgrade`d without folding these
# values into the chart values, this patch will be reverted — re-apply
# it after the upgrade (or move it into a repo-owned values file).
spec:
template:
spec:
containers:
- name: node-exporter
args:
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/host/root
- --path.udev.data=/host/root/run/udev/data
- --web.listen-address=[$(HOST_IP)]:9100
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
- --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$
- --collector.textfile.directory=/host/textfile_collector
volumeMounts:
- mountPath: /host/proc
name: proc
readOnly: true
- mountPath: /host/sys
name: sys
readOnly: true
- mountPath: /host/root
mountPropagation: HostToContainer
name: root
readOnly: true
- mountPath: /host/textfile_collector
name: textfile-collector
readOnly: true
volumes:
- hostPath:
path: /proc
type: ""
name: proc
- hostPath:
path: /sys
type: ""
name: sys
- hostPath:
path: /
type: ""
name: root
- hostPath:
path: /var/lib/node_exporter/textfile_collector
type: DirectoryOrCreate
name: textfile-collector

257
apps/opencloud/OPENLDAP_DEPLOYMENT.md
View file

@ -1,257 +0,0 @@
# OpenCloud with External OpenLDAP Deployment Guide
## Architecture Overview
This deployment uses:
- **External OpenLDAP** for user/group storage
- **Pocket ID** (auth.basicstack.de) for authentication via OIDC
- **OpenCloud** as the file storage platform
### Authentication Flow
1. User accesses `https://opencloud.basicstack.de`
2. OpenCloud redirects to Pocket ID for authentication
3. User logs in at `https://auth.basicstack.de`
4. Pocket ID returns OIDC token to OpenCloud
5. OpenCloud auto-provisions user in external OpenLDAP
6. User is granted access based on OIDC group claims
## Deployment Steps
### 1. Create OpenLDAP Admin Secret
First, generate strong passwords and create the sealed secret:
```bash
# Generate passwords
ADMIN_PASSWORD=$(openssl rand -base64 32)
CONFIG_PASSWORD=$(openssl rand -base64 32)
# Save them securely (e.g., password manager)
echo "Admin Password: $ADMIN_PASSWORD"
echo "Config Password: $CONFIG_PASSWORD"
# Create sealed secret
kubectl create secret generic openldap-admin-secret \
--namespace opencloud \
--from-literal=admin-password="$ADMIN_PASSWORD" \
--from-literal=config-password="$CONFIG_PASSWORD" \
--dry-run=client -o yaml | \
kubeseal --format yaml > openldap-admin-sealed.yaml
# Apply sealed secret
kubectl apply -f openldap-admin-sealed.yaml
```
### 2. Deploy OpenLDAP
```bash
# Deploy OpenLDAP
kubectl apply -f openldap-deployment.yaml
# Wait for OpenLDAP to be ready
kubectl wait --for=condition=ready pod -l app=openldap -n opencloud --timeout=120s
# Check OpenLDAP pod status
kubectl get pods -n opencloud -l app=openldap
kubectl logs -n opencloud -l app=openldap -f
```
### 3. Initialize LDAP Directory Structure
Once OpenLDAP is running, initialize the directory with required OUs:
```bash
# Get the OpenLDAP pod name
LDAP_POD=$(kubectl get pods -n opencloud -l app=openldap -o jsonpath='{.items[0].metadata.name}')
# Copy the LDIF file to the pod
kubectl cp ldap-init-structure.ldif opencloud/$LDAP_POD:/tmp/init-structure.ldif
# Apply the LDIF (replace <admin-password> with your actual password)
kubectl exec -n opencloud $LDAP_POD -- \
ldapadd -x -D "cn=admin,dc=basicstack,dc=de" -w "<admin-password>" -f /tmp/init-structure.ldif
# Verify structure was created
kubectl exec -n opencloud $LDAP_POD -- \
ldapsearch -x -D "cn=admin,dc=basicstack,dc=de" -w "<admin-password>" -b "dc=basicstack,dc=de" -LLL
```
### 4. Deploy OpenCloud
**Important:** Make sure the old OpenCloud PVC is deleted first to ensure fresh initialization with new LDAP configuration.
```bash
# Scale down existing OpenCloud deployment (if running)
kubectl scale deployment -n opencloud opencloud --replicas=0
# Delete old PVC
kubectl delete pvc -n opencloud opencloud-data
# Apply updated deployment
kubectl apply -f opencloud-deployment.yaml
kubectl apply -f opencloud-configmap.yaml
# Scale up
kubectl scale deployment -n opencloud opencloud --replicas=1
# Watch initialization
kubectl logs -n opencloud -l app=opencloud -f
```
### 5. Verify Configuration
```bash
# Check all pods are running
kubectl get pods -n opencloud
# Verify OpenCloud can connect to LDAP
kubectl exec -n opencloud deployment/opencloud -- \
ldapsearch -x -H ldap://openldap.opencloud.svc.cluster.local:389 \
-D "cn=admin,dc=basicstack,dc=de" -w "<admin-password>" \
-b "dc=basicstack,dc=de" -LLL
# Check OpenCloud logs for LDAP connection
kubectl logs -n opencloud -l app=opencloud | grep -i ldap
```
### 6. Test OIDC Authentication
1. Open browser to `https://opencloud.basicstack.de`
2. Should redirect to `https://auth.basicstack.de`
3. Login with Pocket ID credentials
4. Should redirect back to OpenCloud
5. User auto-provisioned in OpenLDAP
## LDAP Directory Structure
```
dc=basicstack,dc=de
├── cn=admin (admin user)
├── ou=users
│ └── (auto-provisioned users from OIDC)
└── ou=groups
├── cn=opencloudUsers (default users group)
└── cn=opencloudAdmins (administrators group)
```
## Configuration Details
### OpenLDAP Connection Details
- **Service:** `openldap.opencloud.svc.cluster.local`
- **Port:** 389 (LDAP), 636 (LDAPS - disabled for internal use)
- **Base DN:** `dc=basicstack,dc=de`
- **Admin DN:** `cn=admin,dc=basicstack,dc=de`
- **User Base:** `ou=users,dc=basicstack,dc=de`
- **Group Base:** `ou=groups,dc=basicstack,dc=de`
### OpenCloud Services Excluded
- **search** - Broken in v7.2.0
- **idp** - Using external Pocket ID for authentication
- **idm** - Using external OpenLDAP for user storage
### OIDC Configuration
- **Issuer:** `https://auth.basicstack.de`
- **Client ID:** (from `opencloud-oidc-secret`)
- **Scopes:** `openid profile email groups offline_access`
- **Auto-provisioning:** Enabled
- **User claim:** `email`
- **Role claim:** `groups`
### Role Mapping
OIDC groups → OpenCloud roles:
- `opencloudAdmin` → admin
- `opencloudSpaceAdmin` → spaceadmin
- `opencloudUser` → user
- `opencloudGuest` → guest
## Troubleshooting
### OpenLDAP not starting
```bash
# Check logs
kubectl logs -n opencloud -l app=openldap
# Check PVCs
kubectl get pvc -n opencloud
# Check events
kubectl get events -n opencloud --sort-by='.lastTimestamp' | tail -20
```
### OpenCloud can't connect to LDAP
```bash
# Test LDAP connectivity from OpenCloud pod
kubectl exec -n opencloud deployment/opencloud -- \
nc -zv openldap.opencloud.svc.cluster.local 389
# Check LDAP service
kubectl get svc -n opencloud openldap
# Check LDAP endpoints
kubectl get endpoints -n opencloud openldap
```
### Users not auto-provisioning
```bash
# Check OpenCloud proxy logs
kubectl logs -n opencloud -l app=opencloud | grep -i provision
# Check OIDC flow
kubectl logs -n opencloud -l app=opencloud | grep -i oidc
# Verify LDAP directory structure
kubectl exec -n opencloud deployment/openldap -- \
ldapsearch -x -D "cn=admin,dc=basicstack,dc=de" -w "<password>" \
-b "ou=users,dc=basicstack,dc=de" -LLL
```
### View auto-provisioned users
```bash
# List all users in LDAP
kubectl exec -n opencloud deployment/openldap -- \
ldapsearch -x -D "cn=admin,dc=basicstack,dc=de" -w "<password>" \
-b "ou=users,dc=basicstack,dc=de" -LLL "(objectClass=inetOrgPerson)"
```
## Backup and Maintenance
### Backup LDAP Data
```bash
# Backup entire LDAP directory
kubectl exec -n opencloud deployment/openldap -- \
slapcat -l /tmp/backup.ldif
kubectl cp opencloud/$LDAP_POD:/tmp/backup.ldif ./ldap-backup-$(date +%Y%m%d).ldif
```
### Monitor LDAP Performance
```bash
# Check LDAP stats
kubectl exec -n opencloud deployment/openldap -- \
ldapsearch -x -H ldapi:/// -Y EXTERNAL -b "cn=Monitor" -LLL
```
## Files
- `openldap-deployment.yaml` - OpenLDAP Kubernetes deployment
- `openldap-admin-sealed.yaml` - Admin credentials (sealed secret)
- `ldap-init-structure.ldif` - Initial directory structure
- `opencloud-deployment.yaml` - OpenCloud deployment (updated for external LDAP)
- `opencloud-configmap.yaml` - OpenCloud configuration (updated for external LDAP)
## References
- OpenLDAP: https://www.openldap.org/
- OpenCloud LDAP Configuration: https://docs.opencloud.eu/
- Pocket ID: https://github.com/stonith404/pocket-id

4
apps/opencloud/backup-cronjob.yaml
View file

@ -90,8 +90,8 @@ spec:
readOnly: true readOnly: true
resources: resources:
requests: requests:
memory: "256Mi" memory: "16Mi"
cpu: "100m" cpu: "20m"
limits: limits:
memory: "512Mi" memory: "512Mi"
cpu: "500m" cpu: "500m"

29
apps/opencloud/init-job.yaml
View file

@ -1,29 +0,0 @@
apiVersion: batch/v1
kind: Job
metadata:
name: opencloud-init-generator
namespace: opencloud
spec:
ttlSecondsAfterFinished: 300
template:
spec:
restartPolicy: Never
containers:
- name: init
image: quay.io/opencloudeu/opencloud:7.2.0
command: ["/bin/sh"]
args:
- -c
- |
echo "=== Running opencloud init ==="
OPENCLOUD_URL=https://opencloud.basicstack.de \
/usr/bin/opencloud init --insecure true
echo ""
echo "=== Generated opencloud.yaml ==="
if [ -f /etc/opencloud/opencloud.yaml ]; then
cat /etc/opencloud/opencloud.yaml
else
echo "ERROR: opencloud.yaml not found at /etc/opencloud/"
ls -la /etc/opencloud/ || echo "Directory doesn't exist"
fi

37
apps/opencloud/ldap-init-structure.ldif
View file

@ -1,37 +0,0 @@
# OpenLDAP Directory Structure Initialization
# This LDIF creates the organizational units needed by OpenCloud
#
# Apply with:
# kubectl exec -n opencloud deployment/openldap -- \
# ldapadd -x -D "cn=admin,dc=basicstack,dc=de" -w <admin-password> -f /tmp/init-structure.ldif
#
# Or copy to pod and apply:
# kubectl cp ldap-init-structure.ldif opencloud/openldap-<pod-id>:/tmp/init-structure.ldif
# kubectl exec -n opencloud openldap-<pod-id> -- \
# ldapadd -x -D "cn=admin,dc=basicstack,dc=de" -w <admin-password> -f /tmp/init-structure.ldif
# Create Users Organizational Unit
dn: ou=users,dc=basicstack,dc=de
objectClass: organizationalUnit
ou: users
description: OpenCloud Users
# Create Groups Organizational Unit
dn: ou=groups,dc=basicstack,dc=de
objectClass: organizationalUnit
ou: groups
description: OpenCloud Groups
# Example: Create default user group
dn: cn=opencloudUsers,ou=groups,dc=basicstack,dc=de
objectClass: groupOfNames
cn: opencloudUsers
description: Default OpenCloud Users Group
member: cn=admin,dc=basicstack,dc=de
# Example: Create admin group
dn: cn=opencloudAdmins,ou=groups,dc=basicstack,dc=de
objectClass: groupOfNames
cn: opencloudAdmins
description: OpenCloud Administrators
member: cn=admin,dc=basicstack,dc=de

Some files were not shown because too many files have changed in this diff Show more