diff --git a/infrastructure/CLUSTER_ACCESS.md b/infrastructure/CLUSTER_ACCESS.md new file mode 100644 index 0000000..f6972f1 --- /dev/null +++ b/infrastructure/CLUSTER_ACCESS.md @@ -0,0 +1,384 @@ +# k3s Cluster Access Guide + +This guide provides practical instructions for accessing and managing the k3s cluster. + +## Prerequisites + +- SSH access to the control plane node (k3s-cp-1) +- SSH key configured for root access +- kubectl installed locally (optional, can use kubectl on control plane) + +## Cluster Information + +| Component | Value | +|-----------|-------| +| Control Plane | k3s-cp-1 (178.105.17.239) | +| API Server | https://178.105.17.239:6443 | +| Current Version | v1.36.2+k3s1 | +| Pod CIDR | 10.244.0.0/16 | +| Service CIDR | 10.43.0.0/16 | + +## Accessing the Cluster + +### Option 1: Direct SSH to Control Plane + +The simplest method for quick operations: + +```bash +# SSH to control plane +ssh root@178.105.17.239 + +# Now you can use kubectl directly +kubectl get nodes +kubectl get pods --all-namespaces +``` + +### Option 2: Local kubectl with Remote kubeconfig + +For working from your local machine: + +```bash +# Copy kubeconfig from control plane +scp root@178.105.17.239:/etc/rancher/k3s/k3s.yaml ~/.kube/config-basicstack + +# Edit the server URL in the config +sed -i 's/127.0.0.1/178.105.17.239/g' ~/.kube/config-basicstack + +# Use the config +export KUBECONFIG=~/.kube/config-basicstack +kubectl get nodes +``` + +**Note:** This requires the API server to be accessible from your location. If you're behind a firewall, use Option 1 or Option 3. + +### Option 3: SSH Tunnel for kubectl + +Secure access through SSH tunnel: + +```bash +# Create SSH tunnel (run in background) +ssh -N -L 6443:localhost:6443 root@178.105.17.239 & + +# Copy and modify kubeconfig +scp root@178.105.17.239:/etc/rancher/k3s/k3s.yaml ~/.kube/config-basicstack + +# Server URL stays as 127.0.0.1:6443 (using tunnel) +export KUBECONFIG=~/.kube/config-basicstack +kubectl get nodes +``` + +## Essential kubectl Commands + +### Cluster Status + +```bash +# View all nodes +kubectl get nodes -o wide + +# Check node resource usage +kubectl top nodes + +# View cluster info +kubectl cluster-info + +# Check component status +kubectl get componentstatuses +``` + +### Pod Management + +```bash +# List all pods in all namespaces +kubectl get pods --all-namespaces -o wide + +# List pods in specific namespace +kubectl get pods -n + +# Get pod details +kubectl describe pod -n + +# View pod logs +kubectl logs -n + +# Follow pod logs in real-time +kubectl logs -f -n + +# Execute command in pod +kubectl exec -it -n -- /bin/bash +``` + +### Namespace Operations + +```bash +# List all namespaces +kubectl get namespaces + +# Create namespace +kubectl create namespace + +# Set default namespace for current context +kubectl config set-context --current --namespace= +``` + +### Service and Ingress + +```bash +# List services +kubectl get services --all-namespaces + +# List ingresses +kubectl get ingress --all-namespaces + +# Describe service +kubectl describe service -n +``` + +### Deployments and StatefulSets + +```bash +# List deployments +kubectl get deployments --all-namespaces + +# Scale deployment +kubectl scale deployment -n --replicas=3 + +# Restart deployment +kubectl rollout restart deployment -n + +# Check rollout status +kubectl rollout status deployment -n +``` + +### ConfigMaps and Secrets + +```bash +# List configmaps +kubectl get configmaps -n + +# View configmap +kubectl describe configmap -n + +# List secrets +kubectl get secrets -n + +# View secret (base64 encoded) +kubectl get secret -n -o yaml +``` + +## Node Access + +### Control Plane Node + +```bash +# Direct SSH +ssh root@178.105.17.239 + +# Check k3s service status +systemctl status k3s + +# View k3s logs +journalctl -u k3s -f + +# Check k3s version +k3s --version +``` + +### Worker Nodes (via Control Plane) + +Worker nodes are on private IPs and must be accessed through the control plane: + +```bash +# SSH to control plane first +ssh root@178.105.17.239 + +# Then SSH to worker using private IP +ssh root@10.42.1.2 # k3s-worker-1 +ssh root@10.42.1.3 # k3s-worker-2 +ssh root@10.42.1.5 # k3s-worker-3 + +# Or use direct SSH for nodes with public IPs +ssh root@167.233.79.65 # k3s-update-runner +ssh root@167.233.121.121 # k3s-worker-3 +``` + +### Node-Level Commands + +```bash +# Check k3s-agent service (on worker) +systemctl status k3s-agent + +# View agent logs +journalctl -u k3s-agent -f + +# Check containerd containers +k3s crictl ps + +# View container logs +k3s crictl logs + +# Check node disk usage +df -h +du -sh /var/lib/rancher/k3s +``` + +## Troubleshooting Commands + +### API Server Issues + +```bash +# Check API server health (on control plane) +curl -k https://localhost:6443/healthz + +# Check k3s service status +systemctl status k3s + +# Restart k3s service (if needed) +systemctl restart k3s + +# View detailed k3s logs +journalctl -u k3s -n 100 --no-pager +``` + +### Network Troubleshooting + +```bash +# Check flannel pods +kubectl get pods -n kube-system -l app=flannel + +# View flannel logs +kubectl logs -n kube-system -l app=flannel + +# Check pod network connectivity (from within a pod) +kubectl run -it --rm debug --image=busybox --restart=Never -- sh +# Inside the pod: +ping +nslookup kubernetes.default +``` + +### Resource Issues + +```bash +# Check resource usage by namespace +kubectl top pods --all-namespaces + +# Identify pods with high resource usage +kubectl top pods --all-namespaces --sort-by=cpu +kubectl top pods --all-namespaces --sort-by=memory + +# Check node disk pressure +kubectl describe nodes | grep -A5 "Conditions:" + +# View events (useful for troubleshooting) +kubectl get events --all-namespaces --sort-by='.lastTimestamp' +``` + +### Pod Troubleshooting + +```bash +# Check why a pod is not starting +kubectl describe pod -n + +# View pod events +kubectl get events -n --field-selector involvedObject.name= + +# Check pod resource limits +kubectl get pod -n -o yaml | grep -A10 resources: + +# Debug with ephemeral container +kubectl debug -n -it --image=busybox +``` + +## Cluster Maintenance + +### Viewing Cluster Certificates + +```bash +# Check certificate expiration (on control plane) +ssh root@178.105.17.239 +for cert in /var/lib/rancher/k3s/server/tls/*.crt; do + echo "=== $cert ===" + openssl x509 -in "$cert" -text -noout | grep -A2 "Validity" +done +``` + +### Backup Operations + +```bash +# Backup etcd snapshot (k3s uses embedded SQLite by default) +ssh root@178.105.17.239 +k3s etcd-snapshot save + +# List snapshots +k3s etcd-snapshot list + +# Restore from snapshot (emergency only) +k3s server --cluster-reset --cluster-reset-restore-path= +``` + +### Checking Cluster Health + +```bash +# Comprehensive cluster health check +kubectl get nodes +kubectl get pods --all-namespaces | grep -v Running | grep -v Completed +kubectl get componentstatuses +kubectl top nodes +kubectl top pods --all-namespaces +``` + +## Important Configuration Files + +### Control Plane (k3s-cp-1) + +| File | Purpose | +|------|---------| +| `/etc/systemd/system/k3s.service` | k3s service definition | +| `/etc/systemd/system/k3s.service.env` | k3s environment variables | +| `/etc/rancher/k3s/config.yaml` | k3s server configuration | +| `/var/lib/rancher/k3s/server/node-token` | Join token for new nodes | +| `/var/lib/rancher/k3s/server/db/state.db` | k3s datastore (SQLite) | +| `/etc/rancher/k3s/k3s.yaml` | Admin kubeconfig | + +### Worker Nodes + +| File | Purpose | +|------|---------| +| `/etc/systemd/system/k3s-agent.service` | k3s agent service definition | +| `/etc/systemd/system/k3s-agent.service.env` | Agent configuration (server URL, token) | +| `/var/lib/rancher/k3s/agent/` | Agent data directory | + +## Security Notes + +- The kubeconfig file (`/etc/rancher/k3s/k3s.yaml`) contains admin credentials +- Never commit kubeconfig files to version control +- The node-token grants full cluster join permissions +- Keep SSH keys secure and use key-based authentication only +- Regularly rotate the node-token if compromised + +## Quick Reference Card + +```bash +# Most common operations +kubectl get nodes # Check cluster nodes +kubectl get pods -A # List all pods +kubectl logs -f -n # Follow pod logs +kubectl exec -it -n -- bash # Shell into pod +kubectl describe pod -n # Debug pod issues +kubectl rollout restart deployment -n # Restart deployment + +# Cluster access +ssh root@178.105.17.239 # SSH to control plane +ssh -J root@178.105.17.239 root@10.42.1.2 # Jump to worker + +# Emergency +systemctl restart k3s # Restart control plane +kubectl drain --ignore-daemonsets # Evacuate node +kubectl cordon # Prevent scheduling +kubectl uncordon # Allow scheduling +``` + +## Additional Resources + +- [k3s Documentation](https://docs.k3s.io/) +- [kubectl Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/) +- [K3S_OPERATIONS.md](./K3S_OPERATIONS.md) - Detailed operations guide diff --git a/infrastructure/K3S_OPERATIONS.md b/infrastructure/K3S_OPERATIONS.md index 8ea4fca..45e7886 100644 --- a/infrastructure/K3S_OPERATIONS.md +++ b/infrastructure/K3S_OPERATIONS.md @@ -330,6 +330,56 @@ systemctl restart k3s-agent kubectl uncordon ``` +### API Server Instability / Connection Refused Errors + +**Symptom:** Intermittent API server connection failures with errors like: +``` +The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port? +Error from server (ServiceUnavailable): the server is currently unable to handle the request +``` + +**Root Cause:** Cluster CIDR mismatch between bootstrap configuration and actual node PodCIDR allocations. This causes flannel to crash repeatedly with: +``` +failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16" specified in the flannel net config doesn't contain "Y.Y.Y.Y/24" PodCIDR +``` + +**Fix:** +1. Verify the actual pod CIDR being used by nodes: + ```bash + kubectl get nodes -o custom-columns=NAME:.metadata.name,POD-CIDR:.spec.podCIDR + ``` + +2. Edit the k3s service file to explicitly set the cluster-cidr: + ```bash + ssh root@178.105.17.239 + + # Backup the service file + cp /etc/systemd/system/k3s.service /etc/systemd/system/k3s.service.backup + + # Edit the service file + nano /etc/systemd/system/k3s.service + + # Add --cluster-cidr flag to the ExecStart line: + ExecStart=/usr/local/bin/k3s \ + server \ + --cluster-cidr=10.244.0.0/16 \ + ``` + +3. Reload and restart k3s: + ```bash + systemctl daemon-reload + systemctl restart k3s + ``` + +4. Verify stability: + ```bash + # Test API server with multiple consecutive calls + for i in {1..20}; do kubectl get nodes --no-headers | wc -l; sleep 2; done + # Should return node count consistently without errors + ``` + +**Prevention:** Always explicitly set `--cluster-cidr` in the k3s server configuration to match the actual network setup. + --- ## Best Practices @@ -349,10 +399,12 @@ kubectl uncordon ### Network Configuration - **Pod CIDR:** 10.244.0.0/16 -- **Service CIDR:** 10.96.0.0/12 +- **Service CIDR:** 10.43.0.0/16 - **CNI:** Flannel (VXLAN mode) - **Flannel Backend:** Uses **public IPs** for VXLAN tunnels (not private IPs) +**Important:** The cluster-cidr must be explicitly set in the k3s service configuration to prevent bootstrap/runtime mismatch. See Troubleshooting section for details. + ### Node CIDRs | Node | Pod CIDR | Private IP | Public IP | @@ -373,6 +425,7 @@ kubectl uncordon | Date | Version | Changes | By | |------|---------|---------|-----| +| 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1, corrected Service CIDR documentation to 10.43.0.0/16 | CTO Agent | | 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent | | 2026-06-10 | v1.35.5+k3s1 | Original cluster deployment | - |