b0f2acf5f3
Root cause: k3s service ClusterIP routing instability causing intermittent
failures despite healthy pods. This is the 5th incident - prior fixes treated
symptoms, not the systemic networking fragility.
Changes:
- Add startup probe (60s delay, prevents premature service registration)
- Fix backup job env var substitution (use shell ${VAR}, not K8s $(VAR))
- Add comprehensive monitoring (ServiceMonitor, PrometheusRule, blackbox probes)
- Add alerting for service failures, high latency, pod restarts, backup failures
Evidence:
- Pod healthy (4d15h uptime, 0 restarts) but service ClusterIP routing broken
- Direct pod IP worked, service ClusterIP failed with "Connection reset by peer"
- Iptables rules correct, endpoints correct, but packets not flowing
- Required pod restart + Traefik restart to restore service
Monitoring now tests full service path from outside cluster, not just pod health.
Will alert immediately on failures instead of relying on reactive discovery.
Related: DEV-213, DEV-221, DEV-223, DEV-224, DEV-230, DEV-231
Co-Authored-By: Paperclip <noreply@paperclip.ing>
|
||
|---|---|---|
| .. | ||
| backup | ||
| bookstack | ||
| directus | ||
| forgejo | ||
| opencloud | ||
| paperclip | ||
| passbolt | ||
| platform-prod | ||
| pocket-id | ||
| stalwart | ||
| README.md | ||
Applications
This directory contains deployment configurations for all applications running on the basicstack.de cluster.
Structure
Each application should have its own subdirectory containing:
- Kubernetes manifests: Deployment, StatefulSet, Service, ConfigMap, Secret definitions
- Helm values: If using Helm charts, include values.yaml files
- Configuration files: Application-specific configs (TOML, JSON, YAML)
- Documentation: README or guide specific to the application deployment
- Patches: Any kubectl patches or modifications needed
Example: Stalwart
The stalwart/ directory serves as a reference implementation, containing:
- Multiple deployment variants (basic, with OIDC, etc.)
- Helm values files
- Monitoring dashboard configurations
- Backup/restore procedures
- Operational documentation
Adding a New Application
- Create a new directory:
apps/<application-name>/ - Add your Kubernetes manifests
- Include a README.md explaining:
- What the application does
- How to deploy it
- Configuration options
- Troubleshooting steps
- Test the deployment in a dev environment
- Commit with a descriptive message
Naming Conventions
- Directory names: lowercase, hyphen-separated (e.g.,
my-app) - Manifest files: descriptive names indicating resource type (e.g.,
deployment.yaml,service.yaml) - Use consistent naming across applications