e848d225cd
Replaces single-CP CP1_UPDATE_PROCEDURE.md + update-cp-1.sh with an HA-aware procedure that covers all three CPs. Rationale: after DEV-510 the cluster runs embedded etcd 3.6.12 across cp-1/cp-2/cp-3 and CPs are NoSchedule-tainted, so: - Kine cascade is no longer the driving risk; embedded 3-node etcd tolerates one member down. The kubectl-latency guardrail stays as a soft signal. - CPs host no StatefulSets or single-replica Deployments. The old Phase C batched stateful eviction dance is obsolete. - CP reboots no longer lose the api-server; two peers keep serving. /livez is polled from the operator machine and cross-checked with etcdctl endpoint status from a peer CP. New CP ordering rule: one CP at a time, leader last, health gate between CPs. Refuses to start if another CP is already cordoned. Automation: - New scripts/os-update/update-cp-node.sh <node> is the entry point. Supports --add-swap / --preflight / --drain / --apt / --reboot / --finalize / --run / --dry-run. Phase A also installs etcd-client on cp-2/cp-3 so the etcd-quorum probes work when cp-1 is later the target. Peer picker prefers a CP that has etcdctl; refuses cp-1 target with a targeted error if no other CP has etcdctl yet. - scripts/os-update/update-cp-1.sh is now a thin wrapper that forwards to update-cp-node.sh k3s-cp-1 "$@" for backwards compat. - CP1_UPDATE_PROCEDURE.md is a redirect stub pointing at the new doc. - OS_UPDATE_PROCEDURE.md §8 topology block, kine cascade section, and automation entry point list all point at the new HA-aware artifacts. Verified: bash -n clean on both scripts; --dry-run walks all six phases for each of cp-1/cp-2/cp-3; --preflight against live cluster correctly aborts cp-1 with "install etcdctl on another CP first" and cp-2/cp-3 with "run --add-swap first" (their current state). Execution is board-approval-gated; DEV-497 (cp-1-only executor) is superseded by a successor ticket. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|---|---|---|
| .. | ||
| k3s-upgrade | ||
| networking | ||
| scripts | ||
| ADD_WORKER_NODE.md | ||
| CLUSTER_ACCESS.md | ||
| CP1_UPDATE_PROCEDURE.md | ||
| CP_UPDATE_PROCEDURE.md | ||
| K3S_OPERATIONS.md | ||
| OS_UPDATE_PROCEDURE.md | ||
| OS_UPDATE_ROUTINE.md | ||
| README.md | ||
Infrastructure
This directory contains cluster-wide infrastructure configurations that support all applications.
Operational procedures (agent-facing)
- OS_UPDATE_PROCEDURE.md — weekly rolling Ubuntu OS update for all cluster nodes (drain → apt → reboot → verify → uncordon → cluster health → next). Never touches k3s config.
- OS_UPDATE_ROUTINE.md — the Paperclip routine that fires the above weekly.
- K3S_OPERATIONS.md — k3s version upgrades (separate concern from OS updates).
- ADD_WORKER_NODE.md — adding a worker node.
- CLUSTER_ACCESS.md — SSH / kubectl access.
Structure
networking/
Network-level configurations including:
- Network Architecture - Comprehensive network architecture documentation
- DNS Requirements - DNS records and configuration guide
- Ingress controller configurations (Traefik)
- Network policies
- Load balancer configurations (k3s ServiceLB)
- Certificate management (cert-manager, TLS)
- Certificate reloading (Stakater Reloader)
monitoring/
Observability infrastructure:
- Prometheus operator and configurations
- Grafana dashboards and datasources
- Logging stack (Loki, Promtail, etc.)
- Alert rules and notification channels
- Service monitors and pod monitors
Purpose
Infrastructure configurations in this directory are shared across all applications. Changes here can affect the entire cluster, so:
- Test thoroughly before applying
- Document all changes
- Consider the impact on existing deployments
- Coordinate with other team members
Adding Infrastructure Components
When adding new infrastructure components:
- Create appropriate subdirectories if needed
- Include clear documentation
- Define dependencies and prerequisites
- Provide rollback procedures
- Update this README with the new component