stack.basicstack.de/infrastructure/CLUSTER_ACCESS.md

# k3s Cluster Access Guide

This guide provides practical instructions for accessing and managing the k3s cluster.

## Prerequisites

- SSH access to the control plane node (k3s-cp-1)
- SSH key configured for root access
- kubectl installed locally (optional, can use kubectl on control plane)

## Cluster Information

| Component | Value |
|-----------|-------|
| Control Plane | k3s-cp-1 (178.105.17.239) |
| API Server | https://178.105.17.239:6443 |
| Current Version | v1.36.2+k3s1 |
| Pod CIDR | 10.244.0.0/16 |
| Service CIDR | 10.43.0.0/16 |

## Accessing the Cluster

### Option 1: Direct SSH to Control Plane

The simplest method for quick operations:

```bash
# SSH to control plane
ssh root@178.105.17.239

# Now you can use kubectl directly
kubectl get nodes
kubectl get pods --all-namespaces
```

### Option 2: Local kubectl with Remote kubeconfig

For working from your local machine:

```bash
# Copy kubeconfig from control plane
scp root@178.105.17.239:/etc/rancher/k3s/k3s.yaml ~/.kube/config-basicstack

# Edit the server URL in the config
sed -i 's/127.0.0.1/178.105.17.239/g' ~/.kube/config-basicstack

# Use the config
export KUBECONFIG=~/.kube/config-basicstack
kubectl get nodes
```

**Note:** This requires the API server to be accessible from your location. If you're behind a firewall, use Option 1 or Option 3.

### Option 3: SSH Tunnel for kubectl

Secure access through SSH tunnel:

```bash
# Create SSH tunnel (run in background)
ssh -N -L 6443:localhost:6443 root@178.105.17.239 &

# Copy and modify kubeconfig
scp root@178.105.17.239:/etc/rancher/k3s/k3s.yaml ~/.kube/config-basicstack

# Server URL stays as 127.0.0.1:6443 (using tunnel)
export KUBECONFIG=~/.kube/config-basicstack
kubectl get nodes
```

## Essential kubectl Commands

### Cluster Status

```bash
# View all nodes
kubectl get nodes -o wide

# Check node resource usage
kubectl top nodes

# View cluster info
kubectl cluster-info

# Check component status
kubectl get componentstatuses
```

### Pod Management

```bash
# List all pods in all namespaces
kubectl get pods --all-namespaces -o wide

# List pods in specific namespace
kubectl get pods -n <namespace>

# Get pod details
kubectl describe pod <pod-name> -n <namespace>

# View pod logs
kubectl logs <pod-name> -n <namespace>

# Follow pod logs in real-time
kubectl logs -f <pod-name> -n <namespace>

# Execute command in pod
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash
```

### Namespace Operations

```bash
# List all namespaces
kubectl get namespaces

# Create namespace
kubectl create namespace <namespace>

# Set default namespace for current context
kubectl config set-context --current --namespace=<namespace>
```

### Service and Ingress

```bash
# List services
kubectl get services --all-namespaces

# List ingresses
kubectl get ingress --all-namespaces

# Describe service
kubectl describe service <service-name> -n <namespace>
```

### Deployments and StatefulSets

```bash
# List deployments
kubectl get deployments --all-namespaces

# Scale deployment
kubectl scale deployment <deployment-name> -n <namespace> --replicas=3

# Restart deployment
kubectl rollout restart deployment <deployment-name> -n <namespace>

# Check rollout status
kubectl rollout status deployment <deployment-name> -n <namespace>
```

### ConfigMaps and Secrets

```bash
# List configmaps
kubectl get configmaps -n <namespace>

# View configmap
kubectl describe configmap <configmap-name> -n <namespace>

# List secrets
kubectl get secrets -n <namespace>

# View secret (base64 encoded)
kubectl get secret <secret-name> -n <namespace> -o yaml
```

## Node Access

### Control Plane Node

```bash
# Direct SSH
ssh root@178.105.17.239

# Check k3s service status
systemctl status k3s

# View k3s logs
journalctl -u k3s -f

# Check k3s version
k3s --version
```

### Worker Nodes (via Control Plane)

Worker nodes are on private IPs and must be accessed through the control plane:

```bash
# SSH to control plane first
ssh root@178.105.17.239

# Then SSH to worker using private IP
ssh root@10.42.1.2  # k3s-worker-1
ssh root@10.42.1.3  # k3s-worker-2
ssh root@10.42.1.5  # k3s-worker-3

# Or use direct SSH for nodes with public IPs
ssh root@167.233.79.65   # k3s-update-runner
ssh root@167.233.121.121 # k3s-worker-3
```

### Node-Level Commands

```bash
# Check k3s-agent service (on worker)
systemctl status k3s-agent

# View agent logs
journalctl -u k3s-agent -f

# Check containerd containers
k3s crictl ps

# View container logs
k3s crictl logs <container-id>

# Check node disk usage
df -h
du -sh /var/lib/rancher/k3s
```

## Troubleshooting Commands

### API Server Issues

```bash
# Check API server health (on control plane)
curl -k https://localhost:6443/healthz

# Check k3s service status
systemctl status k3s

# Restart k3s service (if needed)
systemctl restart k3s

# View detailed k3s logs
journalctl -u k3s -n 100 --no-pager
```

### Network Troubleshooting

```bash
# Check flannel pods
kubectl get pods -n kube-system -l app=flannel

# View flannel logs
kubectl logs -n kube-system -l app=flannel

# Check pod network connectivity (from within a pod)
kubectl run -it --rm debug --image=busybox --restart=Never -- sh
# Inside the pod:
ping <service-ip>
nslookup kubernetes.default
```

### Resource Issues

```bash
# Check resource usage by namespace
kubectl top pods --all-namespaces

# Identify pods with high resource usage
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top pods --all-namespaces --sort-by=memory

# Check node disk pressure
kubectl describe nodes | grep -A5 "Conditions:"

# View events (useful for troubleshooting)
kubectl get events --all-namespaces --sort-by='.lastTimestamp'
```

### Pod Troubleshooting

```bash
# Check why a pod is not starting
kubectl describe pod <pod-name> -n <namespace>

# View pod events
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>

# Check pod resource limits
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A10 resources:

# Debug with ephemeral container
kubectl debug <pod-name> -n <namespace> -it --image=busybox
```

## Cluster Maintenance

### Viewing Cluster Certificates

```bash
# Check certificate expiration (on control plane)
ssh root@178.105.17.239
for cert in /var/lib/rancher/k3s/server/tls/*.crt; do
  echo "=== $cert ==="
  openssl x509 -in "$cert" -text -noout | grep -A2 "Validity"
done
```

### Backup Operations

```bash
# Backup etcd snapshot (k3s uses embedded SQLite by default)
ssh root@178.105.17.239
k3s etcd-snapshot save

# List snapshots
k3s etcd-snapshot list

# Restore from snapshot (emergency only)
k3s server --cluster-reset --cluster-reset-restore-path=<snapshot-file>
```

### Checking Cluster Health

```bash
# Comprehensive cluster health check
kubectl get nodes
kubectl get pods --all-namespaces | grep -v Running | grep -v Completed
kubectl get componentstatuses
kubectl top nodes
kubectl top pods --all-namespaces
```

## Important Configuration Files

### Control Plane (k3s-cp-1)

| File | Purpose |
|------|---------|
| `/etc/systemd/system/k3s.service` | k3s service definition |
| `/etc/systemd/system/k3s.service.env` | k3s environment variables |
| `/etc/rancher/k3s/config.yaml` | k3s server configuration |
| `/var/lib/rancher/k3s/server/node-token` | Join token for new nodes |
| `/var/lib/rancher/k3s/server/db/state.db` | k3s datastore (SQLite) |
| `/etc/rancher/k3s/k3s.yaml` | Admin kubeconfig |

### Worker Nodes

| File | Purpose |
|------|---------|
| `/etc/systemd/system/k3s-agent.service` | k3s agent service definition |
| `/etc/systemd/system/k3s-agent.service.env` | Agent configuration (server URL, token) |
| `/var/lib/rancher/k3s/agent/` | Agent data directory |

## Security Notes

- The kubeconfig file (`/etc/rancher/k3s/k3s.yaml`) contains admin credentials
- Never commit kubeconfig files to version control
- The node-token grants full cluster join permissions
- Keep SSH keys secure and use key-based authentication only
- Regularly rotate the node-token if compromised

## Quick Reference Card

```bash
# Most common operations
kubectl get nodes                              # Check cluster nodes
kubectl get pods -A                            # List all pods
kubectl logs -f <pod> -n <ns>                  # Follow pod logs
kubectl exec -it <pod> -n <ns> -- bash         # Shell into pod
kubectl describe pod <pod> -n <ns>             # Debug pod issues
kubectl rollout restart deployment <dep> -n <ns> # Restart deployment

# Cluster access
ssh root@178.105.17.239                        # SSH to control plane
ssh -J root@178.105.17.239 root@10.42.1.2     # Jump to worker

# Emergency
systemctl restart k3s                          # Restart control plane
kubectl drain <node> --ignore-daemonsets       # Evacuate node
kubectl cordon <node>                          # Prevent scheduling
kubectl uncordon <node>                        # Allow scheduling
```

## Additional Resources

- [k3s Documentation](https://docs.k3s.io/)
- [kubectl Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/)
- [K3S_OPERATIONS.md](./K3S_OPERATIONS.md) - Detailed operations guide