basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Update k3s cluster documentation

Browse source 
- Add CLUSTER_ACCESS.md with comprehensive cluster access guide
- Fix Service CIDR in K3S_OPERATIONS.md (10.43.0.0/16, not 10.96.0.0/12)
- Document API server instability fix (cluster-cidr configuration)
- Add troubleshooting section for CIDR mismatch issues
- Update change history with cluster update details

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-07-06 17:54:53 +00:00
parent 2c0e7f22b1
commit f3034af04a
2 changed files with 438 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

384
infrastructure/CLUSTER_ACCESS.md Normal file
View file

@ -0,0 +1,384 @@
# k3s Cluster Access Guide
This guide provides practical instructions for accessing and managing the k3s cluster.
## Prerequisites
- SSH access to the control plane node (k3s-cp-1)
- SSH key configured for root access
- kubectl installed locally (optional, can use kubectl on control plane)
## Cluster Information
| Component | Value |
|-----------|-------|
| Control Plane | k3s-cp-1 (178.105.17.239) |
| API Server | https://178.105.17.239:6443 |
| Current Version | v1.36.2+k3s1 |
| Pod CIDR | 10.244.0.0/16 |
| Service CIDR | 10.43.0.0/16 |
## Accessing the Cluster
### Option 1: Direct SSH to Control Plane
The simplest method for quick operations:
```bash
# SSH to control plane
ssh root@178.105.17.239
# Now you can use kubectl directly
kubectl get nodes
kubectl get pods --all-namespaces
```
### Option 2: Local kubectl with Remote kubeconfig
For working from your local machine:
```bash
# Copy kubeconfig from control plane
scp root@178.105.17.239:/etc/rancher/k3s/k3s.yaml ~/.kube/config-basicstack
# Edit the server URL in the config
sed -i 's/127.0.0.1/178.105.17.239/g' ~/.kube/config-basicstack
# Use the config
export KUBECONFIG=~/.kube/config-basicstack
kubectl get nodes
```
**Note:** This requires the API server to be accessible from your location. If you're behind a firewall, use Option 1 or Option 3.
### Option 3: SSH Tunnel for kubectl
Secure access through SSH tunnel:
```bash
# Create SSH tunnel (run in background)
ssh -N -L 6443:localhost:6443 root@178.105.17.239 &
# Copy and modify kubeconfig
scp root@178.105.17.239:/etc/rancher/k3s/k3s.yaml ~/.kube/config-basicstack
# Server URL stays as 127.0.0.1:6443 (using tunnel)
export KUBECONFIG=~/.kube/config-basicstack
kubectl get nodes
```
## Essential kubectl Commands
### Cluster Status
```bash
# View all nodes
kubectl get nodes -o wide
# Check node resource usage
kubectl top nodes
# View cluster info
kubectl cluster-info
# Check component status
kubectl get componentstatuses
```
### Pod Management
```bash
# List all pods in all namespaces
kubectl get pods --all-namespaces -o wide
# List pods in specific namespace
kubectl get pods -n <namespace>
# Get pod details
kubectl describe pod <pod-name> -n <namespace>
# View pod logs
kubectl logs <pod-name> -n <namespace>
# Follow pod logs in real-time
kubectl logs -f <pod-name> -n <namespace>
# Execute command in pod
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash
```
### Namespace Operations
```bash
# List all namespaces
kubectl get namespaces
# Create namespace
kubectl create namespace <namespace>
# Set default namespace for current context
kubectl config set-context --current --namespace=<namespace>
```
### Service and Ingress
```bash
# List services
kubectl get services --all-namespaces
# List ingresses
kubectl get ingress --all-namespaces
# Describe service
kubectl describe service <service-name> -n <namespace>
```
### Deployments and StatefulSets
```bash
# List deployments
kubectl get deployments --all-namespaces
# Scale deployment
kubectl scale deployment <deployment-name> -n <namespace> --replicas=3
# Restart deployment
kubectl rollout restart deployment <deployment-name> -n <namespace>
# Check rollout status
kubectl rollout status deployment <deployment-name> -n <namespace>
```
### ConfigMaps and Secrets
```bash
# List configmaps
kubectl get configmaps -n <namespace>
# View configmap
kubectl describe configmap <configmap-name> -n <namespace>
# List secrets
kubectl get secrets -n <namespace>
# View secret (base64 encoded)
kubectl get secret <secret-name> -n <namespace> -o yaml
```
## Node Access
### Control Plane Node
```bash
# Direct SSH
ssh root@178.105.17.239
# Check k3s service status
systemctl status k3s
# View k3s logs
journalctl -u k3s -f
# Check k3s version
k3s --version
```
### Worker Nodes (via Control Plane)
Worker nodes are on private IPs and must be accessed through the control plane:
```bash
# SSH to control plane first
ssh root@178.105.17.239
# Then SSH to worker using private IP
ssh root@10.42.1.2 # k3s-worker-1
ssh root@10.42.1.3 # k3s-worker-2
ssh root@10.42.1.5 # k3s-worker-3
# Or use direct SSH for nodes with public IPs
ssh root@167.233.79.65 # k3s-update-runner
ssh root@167.233.121.121 # k3s-worker-3
```
### Node-Level Commands
```bash
# Check k3s-agent service (on worker)
systemctl status k3s-agent
# View agent logs
journalctl -u k3s-agent -f
# Check containerd containers
k3s crictl ps
# View container logs
k3s crictl logs <container-id>
# Check node disk usage
df -h
du -sh /var/lib/rancher/k3s
```
## Troubleshooting Commands
### API Server Issues
```bash
# Check API server health (on control plane)
curl -k https://localhost:6443/healthz
# Check k3s service status
systemctl status k3s
# Restart k3s service (if needed)
systemctl restart k3s
# View detailed k3s logs
journalctl -u k3s -n 100 --no-pager
```
### Network Troubleshooting
```bash
# Check flannel pods
kubectl get pods -n kube-system -l app=flannel
# View flannel logs
kubectl logs -n kube-system -l app=flannel
# Check pod network connectivity (from within a pod)
kubectl run -it --rm debug --image=busybox --restart=Never -- sh
# Inside the pod:
ping <service-ip>
nslookup kubernetes.default
```
### Resource Issues
```bash
# Check resource usage by namespace
kubectl top pods --all-namespaces
# Identify pods with high resource usage
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top pods --all-namespaces --sort-by=memory
# Check node disk pressure
kubectl describe nodes | grep -A5 "Conditions:"
# View events (useful for troubleshooting)
kubectl get events --all-namespaces --sort-by='.lastTimestamp'
```
### Pod Troubleshooting
```bash
# Check why a pod is not starting
kubectl describe pod <pod-name> -n <namespace>
# View pod events
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>
# Check pod resource limits
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A10 resources:
# Debug with ephemeral container
kubectl debug <pod-name> -n <namespace> -it --image=busybox
```
## Cluster Maintenance
### Viewing Cluster Certificates
```bash
# Check certificate expiration (on control plane)
ssh root@178.105.17.239
for cert in /var/lib/rancher/k3s/server/tls/*.crt; do
echo "=== $cert ==="
openssl x509 -in "$cert" -text -noout | grep -A2 "Validity"
done
```
### Backup Operations
```bash
# Backup etcd snapshot (k3s uses embedded SQLite by default)
ssh root@178.105.17.239
k3s etcd-snapshot save
# List snapshots
k3s etcd-snapshot list
# Restore from snapshot (emergency only)
k3s server --cluster-reset --cluster-reset-restore-path=<snapshot-file>
```
### Checking Cluster Health
```bash
# Comprehensive cluster health check
kubectl get nodes
kubectl get pods --all-namespaces | grep -v Running | grep -v Completed
kubectl get componentstatuses
kubectl top nodes
kubectl top pods --all-namespaces
```
## Important Configuration Files
### Control Plane (k3s-cp-1)
| File | Purpose |
|------|---------|
| `/etc/systemd/system/k3s.service` | k3s service definition |
| `/etc/systemd/system/k3s.service.env` | k3s environment variables |
| `/etc/rancher/k3s/config.yaml` | k3s server configuration |
| `/var/lib/rancher/k3s/server/node-token` | Join token for new nodes |
| `/var/lib/rancher/k3s/server/db/state.db` | k3s datastore (SQLite) |
| `/etc/rancher/k3s/k3s.yaml` | Admin kubeconfig |
### Worker Nodes
| File | Purpose |
|------|---------|
| `/etc/systemd/system/k3s-agent.service` | k3s agent service definition |
| `/etc/systemd/system/k3s-agent.service.env` | Agent configuration (server URL, token) |
| `/var/lib/rancher/k3s/agent/` | Agent data directory |
## Security Notes
- The kubeconfig file (`/etc/rancher/k3s/k3s.yaml`) contains admin credentials
- Never commit kubeconfig files to version control
- The node-token grants full cluster join permissions
- Keep SSH keys secure and use key-based authentication only
- Regularly rotate the node-token if compromised
## Quick Reference Card
```bash
# Most common operations
kubectl get nodes # Check cluster nodes
kubectl get pods -A # List all pods
kubectl logs -f <pod> -n <ns> # Follow pod logs
kubectl exec -it <pod> -n <ns> -- bash # Shell into pod
kubectl describe pod <pod> -n <ns> # Debug pod issues
kubectl rollout restart deployment <dep> -n <ns> # Restart deployment
# Cluster access
ssh root@178.105.17.239 # SSH to control plane
ssh -J root@178.105.17.239 root@10.42.1.2 # Jump to worker
# Emergency
systemctl restart k3s # Restart control plane
kubectl drain <node> --ignore-daemonsets # Evacuate node
kubectl cordon <node> # Prevent scheduling
kubectl uncordon <node> # Allow scheduling
```
## Additional Resources
- [k3s Documentation](https://docs.k3s.io/)
- [kubectl Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/)
- [K3S_OPERATIONS.md](./K3S_OPERATIONS.md) - Detailed operations guide

55
infrastructure/K3S_OPERATIONS.md
View file

@ -330,6 +330,56 @@ systemctl restart k3s-agent
kubectl uncordon <node-name>
```
### API Server Instability / Connection Refused Errors
**Symptom:** Intermittent API server connection failures with errors like:
```
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
Error from server (ServiceUnavailable): the server is currently unable to handle the request
```
**Root Cause:** Cluster CIDR mismatch between bootstrap configuration and actual node PodCIDR allocations. This causes flannel to crash repeatedly with:
```
failed to register flannel network: failed to acquire lease: subnet "X.X.X.X/16" specified in the flannel net config doesn't contain "Y.Y.Y.Y/24" PodCIDR
```
**Fix:**
1. Verify the actual pod CIDR being used by nodes:
```bash
kubectl get nodes -o custom-columns=NAME:.metadata.name,POD-CIDR:.spec.podCIDR
```
2. Edit the k3s service file to explicitly set the cluster-cidr:
```bash
ssh root@178.105.17.239
# Backup the service file
cp /etc/systemd/system/k3s.service /etc/systemd/system/k3s.service.backup
# Edit the service file
nano /etc/systemd/system/k3s.service
# Add --cluster-cidr flag to the ExecStart line:
ExecStart=/usr/local/bin/k3s \
server \
--cluster-cidr=10.244.0.0/16 \
```
3. Reload and restart k3s:
```bash
systemctl daemon-reload
systemctl restart k3s
```
4. Verify stability:
```bash
# Test API server with multiple consecutive calls
for i in {1..20}; do kubectl get nodes --no-headers | wc -l; sleep 2; done
# Should return node count consistently without errors
```
**Prevention:** Always explicitly set `--cluster-cidr` in the k3s server configuration to match the actual network setup.
---
## Best Practices
@ -349,10 +399,12 @@ kubectl uncordon <node-name>
### Network Configuration
- **Pod CIDR:** 10.244.0.0/16
- **Service CIDR:** 10.96.0.0/12
- **Service CIDR:** 10.43.0.0/16
- **CNI:** Flannel (VXLAN mode)
- **Flannel Backend:** Uses **public IPs** for VXLAN tunnels (not private IPs)
**Important:** The cluster-cidr must be explicitly set in the k3s service configuration to prevent bootstrap/runtime mismatch. See Troubleshooting section for details.
### Node CIDRs
| Node | Pod CIDR | Private IP | Public IP |
@ -373,6 +425,7 @@ kubectl uncordon <node-name>
| Date | Version | Changes | By |
|------|---------|---------|-----|
| 2026-07-06 | v1.36.2+k3s1 | Fixed API server instability by adding explicit --cluster-cidr=10.244.0.0/16 to k3s service, updated all nodes to v1.36.2+k3s1, corrected Service CIDR documentation to 10.43.0.0/16 | CTO Agent |
| 2026-07-06 | v1.36.2+k3s1 | Initial documentation, upgraded from v1.35.5, installed system-upgrade-controller | CTO Agent |
| 2026-06-10 | v1.35.5+k3s1 | Original cluster deployment | - |