The 180s post-uncordon health check in `update-node.sh` step 7 was single-shot. Any transient yellow signal (Init:*, Running-but-not-Ready, deployment below replicas during reschedule) at the exact moment of the check would FATAL the whole cycle and require operator wake + manual `--start-from <next-node>`. During DEV-593 this fired three times in one cycle (opencloud init, harbor jobservice reinit) for conditions that cleared within seconds. Wrap the health check in a bounded retry loop: - First check at `POST_UNCORDON_WAIT_SECONDS` (default 180s, unchanged). - Up to `HEALTH_MAX_RETRIES` (default 10) more attempts at `HEALTH_RETRY_INTERVAL_SECONDS` (default 30s) = +5 min max. - Short-circuit halt immediately on persistent pod errors (CrashLoopBackOff, ImagePullBackOff, ErrImagePull, CreateContainerConfigError, InvalidImageName) — these don't recover on their own and shouldn't burn the retry budget. - Each retry calls `cluster-health.sh` with `RETRY_ON_TRANSIENT=0` so the outer loop controls cadence (no double 30s sleeps per attempt). No k3s or manifest changes. Preflight path is untouched. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|---|---|---|
| apps | ||
| docs | ||
| infrastructure | ||
| velero | ||
| .gitignore | ||
| add-user-andreas.ldif | ||
| DEV-349-progress.md | ||
| README.md | ||
stack.basicstack.de
CD/CI deployment manifests and configurations for the basicstack.de Kubernetes cluster.
Repository Structure
stack.basicstack.de/
├── apps/ # Application deployments
│ ├── stalwart/ # Stalwart mail server (example)
│ └── forgejo/ # Forgejo Git service (placeholder)
├── infrastructure/ # Infrastructure-level configurations
│ ├── networking/ # Network policies, ingress, DNS
│ └── monitoring/ # Monitoring, logging, observability
└── docs/ # Documentation and guides
Purpose
This repository serves as the central source of truth for all deployment configurations targeting the basicstack.de Kubernetes cluster. It follows GitOps principles where infrastructure and application state is declaratively defined and version-controlled.
Directory Details
apps/
Contains deployment configurations for individual applications and services running on the cluster. Each application should have its own subdirectory with:
- Kubernetes manifests (Deployments, StatefulSets, Services, etc.)
- Helm values files
- Configuration files
- Application-specific documentation
Example: The stalwart/ directory contains the complete deployment configuration for the Stalwart mail server, including multiple deployment variants, monitoring setup, and operational guides.
infrastructure/
Contains cluster-wide infrastructure configurations:
- networking/: Ingress controllers, network policies, DNS configurations, load balancers
- monitoring/: Prometheus, Grafana, logging infrastructure, observability tools
docs/
General documentation including:
- Deployment procedures
- Cluster architecture
- Troubleshooting guides
- Best practices
GitOps with Argo CD
This repository is managed via Argo CD, the GitOps deployment platform for the cluster.
- Argo CD UI: https://argo.basicstack.de
- Authentication: Pocket ID SSO (https://auth.basicstack.de)
- Documentation: docs/argocd.md
All changes pushed to the main branch are automatically synchronized to the cluster. Applications are defined in apps/app-*.yaml files and reference subdirectories for their manifests.
For details on managing applications, repository credentials, troubleshooting, and emergency procedures, see the Argo CD documentation.
Getting Started
- Clone this repository
- Review the example Stalwart deployment in
apps/stalwart/ - Follow the pattern for new application deployments
- Ensure all manifests are tested before committing
- Argo CD will automatically sync changes to the cluster (or use manual sync for critical changes)
Contributing
All changes should be:
- Committed with clear, descriptive messages
- Tested in a development environment when possible
- Documented appropriately
- Reviewed before deployment to production
Cluster Information
- Cluster: basicstack.de
- Platform: K3s on Hetzner Cloud
- Namespace Strategy: One namespace per application (recommended)
- Ingress: Traefik (default K3s ingress controller)