da9777358b
DEV-586 (2026-09-27) aborted mid-drain because cp-1's pubkey had drifted out of four nodes' /root/.ssh/authorized_keys — probably Hetzner cloud-init re-syncing from the HC SSH-keys API on reboot (the operator key is not in that keyset). Both the apt-phase SSH and the CP peer-probe SSH died once they tried to reach the drifted targets, and recovery cost real time. os-update.sh now runs a preflight against the fully-computed plan: - Reads cp-1's pubkey once (fails loudly if operator can't reach cp-1). - For every planned worker + CP: proves operator SSH works, then grep-checks cp-1's pubkey in /root/.ssh/authorized_keys. Missing pubkeys are appended via the operator's own SSH (auto-heal). Verify via grep after the write. - Nodes still unreachable at the end block the whole cycle with the exact fix command in the log — nothing gets drained until this passes. - Preflight report: /tmp/os-update-<UTC-ts>/ssh-preflight.txt (ok/repaired/ problem lists) as audit trail. - Escape hatch: --skip-ssh-preflight for bootstrap scenarios. Also factors the node->ssh-endpoint case into a shared node_ssh_endpoint() helper so the preflight, finalize/apt-history loop, and any future call site can't drift apart. Verified end-to-end against the live cluster: all planned nodes reachable and cp-1's pubkey currently present on cp-2/cp-3, worker-1..6. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|---|---|---|
| .. | ||
| k3s-upgrade | ||
| networking | ||
| scripts | ||
| ADD_WORKER_NODE.md | ||
| CLUSTER_ACCESS.md | ||
| CP1_UPDATE_PROCEDURE.md | ||
| CP_UPDATE_PROCEDURE.md | ||
| K3S_OPERATIONS.md | ||
| OS_UPDATE_PROCEDURE.md | ||
| OS_UPDATE_ROUTINE.md | ||
| README.md | ||
Infrastructure
This directory contains cluster-wide infrastructure configurations that support all applications.
Operational procedures (agent-facing)
- OS_UPDATE_PROCEDURE.md — weekly rolling Ubuntu OS update for all cluster nodes (drain → apt → reboot → verify → uncordon → cluster health → next). Never touches k3s config.
- OS_UPDATE_ROUTINE.md — the Paperclip routine that fires the above weekly.
- K3S_OPERATIONS.md — k3s version upgrades (separate concern from OS updates).
- ADD_WORKER_NODE.md — adding a worker node.
- CLUSTER_ACCESS.md — SSH / kubectl access.
Structure
networking/
Network-level configurations including:
- Network Architecture - Comprehensive network architecture documentation
- DNS Requirements - DNS records and configuration guide
- Ingress controller configurations (Traefik)
- Network policies
- Load balancer configurations (k3s ServiceLB)
- Certificate management (cert-manager, TLS)
- Certificate reloading (Stakater Reloader)
monitoring/
Observability infrastructure:
- Prometheus operator and configurations
- Grafana dashboards and datasources
- Logging stack (Loki, Promtail, etc.)
- Alert rules and notification channels
- Service monitors and pod monitors
Purpose
Infrastructure configurations in this directory are shared across all applications. Changes here can affect the entire cluster, so:
- Test thoroughly before applying
- Document all changes
- Consider the impact on existing deployments
- Coordinate with other team members
Adding Infrastructure Components
When adding new infrastructure components:
- Create appropriate subdirectories if needed
- Include clear documentation
- Define dependencies and prerequisites
- Provide rollback procedures
- Update this README with the new component