basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions
CD/CI deployment manifests and configurations for basicstack.de cluster
358 commits 8 branches 0 tags 2.8 MiB
Find a file
CTO (Paperclip Agent) da9777358b feat(os-update): SSH-key drift preflight (DEV-587) 
DEV-586 (2026-09-27) aborted mid-drain because cp-1's pubkey had drifted out
of four nodes' /root/.ssh/authorized_keys — probably Hetzner cloud-init
re-syncing from the HC SSH-keys API on reboot (the operator key is not in
that keyset). Both the apt-phase SSH and the CP peer-probe SSH died once
they tried to reach the drifted targets, and recovery cost real time.

os-update.sh now runs a preflight against the fully-computed plan:
- Reads cp-1's pubkey once (fails loudly if operator can't reach cp-1).
- For every planned worker + CP: proves operator SSH works, then grep-checks
  cp-1's pubkey in /root/.ssh/authorized_keys. Missing pubkeys are appended
  via the operator's own SSH (auto-heal). Verify via grep after the write.
- Nodes still unreachable at the end block the whole cycle with the exact
  fix command in the log — nothing gets drained until this passes.
- Preflight report: /tmp/os-update-<UTC-ts>/ssh-preflight.txt (ok/repaired/
  problem lists) as audit trail.
- Escape hatch: --skip-ssh-preflight for bootstrap scenarios.

Also factors the node->ssh-endpoint case into a shared node_ssh_endpoint()
helper so the preflight, finalize/apt-history loop, and any future call site
can't drift apart.

Verified end-to-end against the live cluster: all planned nodes reachable
and cp-1's pubkey currently present on cp-2/cp-3, worker-1..6.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-11 02:57:31 +00:00
apps fix(opencloud): reduce CPU request 250m→50m to free worker-2 slot for backup CronJob (DEV-594) 2026-10-11 01:11:29 +00:00
docs chore(restic): bump image 0.17.3 -> 0.19.1 (DEV-541) 2026-08-23 15:52:15 +00:00
infrastructure feat(os-update): SSH-key drift preflight (DEV-587) 2026-10-11 02:57:31 +00:00
velero fix(velero-drill): switch webhook from bearer to github_hmac (DEV-590) 2026-10-01 04:45:51 +00:00
.gitignore Convert all secrets to SealedSecrets for enhanced security 2026-07-01 18:38:27 +00:00
add-user-andreas.ldif OpenCloud: Use native bash substitution in config file 2026-07-05 14:03:49 +00:00
DEV-349-progress.md Consolidate Forgejo backup into forgejo namespace 2026-07-26 09:35:31 +00:00
README.md Add comprehensive Argo CD documentation 2026-07-12 10:04:51 +00:00

README.md

stack.basicstack.de

CD/CI deployment manifests and configurations for the basicstack.de Kubernetes cluster.

Repository Structure

stack.basicstack.de/
├── apps/                    # Application deployments
│   ├── stalwart/           # Stalwart mail server (example)
│   └── forgejo/            # Forgejo Git service (placeholder)
├── infrastructure/          # Infrastructure-level configurations
│   ├── networking/         # Network policies, ingress, DNS
│   └── monitoring/         # Monitoring, logging, observability
└── docs/                   # Documentation and guides

Purpose

This repository serves as the central source of truth for all deployment configurations targeting the basicstack.de Kubernetes cluster. It follows GitOps principles where infrastructure and application state is declaratively defined and version-controlled.

Directory Details

apps/

Contains deployment configurations for individual applications and services running on the cluster. Each application should have its own subdirectory with:

  • Kubernetes manifests (Deployments, StatefulSets, Services, etc.)
  • Helm values files
  • Configuration files
  • Application-specific documentation

Example: The stalwart/ directory contains the complete deployment configuration for the Stalwart mail server, including multiple deployment variants, monitoring setup, and operational guides.

infrastructure/

Contains cluster-wide infrastructure configurations:

  • networking/: Ingress controllers, network policies, DNS configurations, load balancers
  • monitoring/: Prometheus, Grafana, logging infrastructure, observability tools

docs/

General documentation including:

  • Deployment procedures
  • Cluster architecture
  • Troubleshooting guides
  • Best practices

GitOps with Argo CD

This repository is managed via Argo CD, the GitOps deployment platform for the cluster.

All changes pushed to the main branch are automatically synchronized to the cluster. Applications are defined in apps/app-*.yaml files and reference subdirectories for their manifests.

For details on managing applications, repository credentials, troubleshooting, and emergency procedures, see the Argo CD documentation.

Getting Started

  1. Clone this repository
  2. Review the example Stalwart deployment in apps/stalwart/
  3. Follow the pattern for new application deployments
  4. Ensure all manifests are tested before committing
  5. Argo CD will automatically sync changes to the cluster (or use manual sync for critical changes)

Contributing

All changes should be:

  1. Committed with clear, descriptive messages
  2. Tested in a development environment when possible
  3. Documented appropriately
  4. Reviewed before deployment to production

Cluster Information

  • Cluster: basicstack.de
  • Platform: K3s on Hetzner Cloud
  • Namespace Strategy: One namespace per application (recommended)
  • Ingress: Traefik (default K3s ingress controller)