stack.basicstack.de/infrastructure/CLUSTER_ACCESS.md

k3s Cluster Access Guide

This guide provides practical instructions for accessing and managing the k3s cluster.

Prerequisites

• SSH access to the control plane node (k3s-cp-1)
• SSH key configured for root access
• kubectl installed locally (optional, can use kubectl on control plane)

Cluster Information

Component	Value
Control Plane	k3s-cp-1 (178.105.17.239)
API Server	https://178.105.17.239:6443
Current Version	v1.36.2+k3s1
Pod CIDR	10.244.0.0/16
Service CIDR	10.43.0.0/16

Accessing the Cluster

Option 1: Direct SSH to Control Plane

The simplest method for quick operations:

# SSH to control plane
ssh root@178.105.17.239

# Now you can use kubectl directly
kubectl get nodes
kubectl get pods --all-namespaces

Option 2: Local kubectl with Remote kubeconfig

For working from your local machine:

# Copy kubeconfig from control plane
scp root@178.105.17.239:/etc/rancher/k3s/k3s.yaml ~/.kube/config-basicstack

# Edit the server URL in the config
sed -i 's/127.0.0.1/178.105.17.239/g' ~/.kube/config-basicstack

# Use the config
export KUBECONFIG=~/.kube/config-basicstack
kubectl get nodes

Note: This requires the API server to be accessible from your location. If you're behind a firewall, use Option 1 or Option 3.

Option 3: SSH Tunnel for kubectl

Secure access through SSH tunnel:

# Create SSH tunnel (run in background)
ssh -N -L 6443:localhost:6443 root@178.105.17.239 &

# Copy and modify kubeconfig
scp root@178.105.17.239:/etc/rancher/k3s/k3s.yaml ~/.kube/config-basicstack

# Server URL stays as 127.0.0.1:6443 (using tunnel)
export KUBECONFIG=~/.kube/config-basicstack
kubectl get nodes

Essential kubectl Commands

Cluster Status

# View all nodes
kubectl get nodes -o wide

# Check node resource usage
kubectl top nodes

# View cluster info
kubectl cluster-info

# Check component status
kubectl get componentstatuses

Pod Management

# List all pods in all namespaces
kubectl get pods --all-namespaces -o wide

# List pods in specific namespace
kubectl get pods -n <namespace>

# Get pod details
kubectl describe pod <pod-name> -n <namespace>

# View pod logs
kubectl logs <pod-name> -n <namespace>

# Follow pod logs in real-time
kubectl logs -f <pod-name> -n <namespace>

# Execute command in pod
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash

Namespace Operations

# List all namespaces
kubectl get namespaces

# Create namespace
kubectl create namespace <namespace>

# Set default namespace for current context
kubectl config set-context --current --namespace=<namespace>

Service and Ingress

# List services
kubectl get services --all-namespaces

# List ingresses
kubectl get ingress --all-namespaces

# Describe service
kubectl describe service <service-name> -n <namespace>

Deployments and StatefulSets

# List deployments
kubectl get deployments --all-namespaces

# Scale deployment
kubectl scale deployment <deployment-name> -n <namespace> --replicas=3

# Restart deployment
kubectl rollout restart deployment <deployment-name> -n <namespace>

# Check rollout status
kubectl rollout status deployment <deployment-name> -n <namespace>

ConfigMaps and Secrets

# List configmaps
kubectl get configmaps -n <namespace>

# View configmap
kubectl describe configmap <configmap-name> -n <namespace>

# List secrets
kubectl get secrets -n <namespace>

# View secret (base64 encoded)
kubectl get secret <secret-name> -n <namespace> -o yaml

Node Access

Control Plane Node

# Direct SSH
ssh root@178.105.17.239

# Check k3s service status
systemctl status k3s

# View k3s logs
journalctl -u k3s -f

# Check k3s version
k3s --version

Worker Nodes (via Control Plane)

Worker nodes are on private IPs and must be accessed through the control plane:

# SSH to control plane first
ssh root@178.105.17.239

# Then SSH to worker using private IP
ssh root@10.42.1.2  # k3s-worker-1
ssh root@10.42.1.3  # k3s-worker-2
ssh root@10.42.1.5  # k3s-worker-3

# Or use direct SSH for nodes with public IPs
ssh root@167.233.79.65   # k3s-update-runner
ssh root@167.233.121.121 # k3s-worker-3

Node-Level Commands

# Check k3s-agent service (on worker)
systemctl status k3s-agent

# View agent logs
journalctl -u k3s-agent -f

# Check containerd containers
k3s crictl ps

# View container logs
k3s crictl logs <container-id>

# Check node disk usage
df -h
du -sh /var/lib/rancher/k3s

Troubleshooting Commands

API Server Issues

# Check API server health (on control plane)
curl -k https://localhost:6443/healthz

# Check k3s service status
systemctl status k3s

# Restart k3s service (if needed)
systemctl restart k3s

# View detailed k3s logs
journalctl -u k3s -n 100 --no-pager

Network Troubleshooting

# Check flannel pods
kubectl get pods -n kube-system -l app=flannel

# View flannel logs
kubectl logs -n kube-system -l app=flannel

# Check pod network connectivity (from within a pod)
kubectl run -it --rm debug --image=busybox --restart=Never -- sh
# Inside the pod:
ping <service-ip>
nslookup kubernetes.default

Resource Issues

# Check resource usage by namespace
kubectl top pods --all-namespaces

# Identify pods with high resource usage
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top pods --all-namespaces --sort-by=memory

# Check node disk pressure
kubectl describe nodes | grep -A5 "Conditions:"

# View events (useful for troubleshooting)
kubectl get events --all-namespaces --sort-by='.lastTimestamp'

Pod Troubleshooting

# Check why a pod is not starting
kubectl describe pod <pod-name> -n <namespace>

# View pod events
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>

# Check pod resource limits
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A10 resources:

# Debug with ephemeral container
kubectl debug <pod-name> -n <namespace> -it --image=busybox

Cluster Maintenance

Viewing Cluster Certificates

# Check certificate expiration (on control plane)
ssh root@178.105.17.239
for cert in /var/lib/rancher/k3s/server/tls/*.crt; do
  echo "=== $cert ==="
  openssl x509 -in "$cert" -text -noout | grep -A2 "Validity"
done

Backup Operations

# Backup etcd snapshot (k3s uses embedded SQLite by default)
ssh root@178.105.17.239
k3s etcd-snapshot save

# List snapshots
k3s etcd-snapshot list

# Restore from snapshot (emergency only)
k3s server --cluster-reset --cluster-reset-restore-path=<snapshot-file>

Checking Cluster Health

# Comprehensive cluster health check
kubectl get nodes
kubectl get pods --all-namespaces | grep -v Running | grep -v Completed
kubectl get componentstatuses
kubectl top nodes
kubectl top pods --all-namespaces

Important Configuration Files

Control Plane (k3s-cp-1)

File	Purpose
/etc/systemd/system/k3s.service	k3s service definition
/etc/systemd/system/k3s.service.env	k3s environment variables
/etc/rancher/k3s/config.yaml	k3s server configuration
/var/lib/rancher/k3s/server/node-token	Join token for new nodes
/var/lib/rancher/k3s/server/db/state.db	k3s datastore (SQLite)
/etc/rancher/k3s/k3s.yaml	Admin kubeconfig

Worker Nodes

File	Purpose
/etc/systemd/system/k3s-agent.service	k3s agent service definition
/etc/systemd/system/k3s-agent.service.env	Agent configuration (server URL, token)
/var/lib/rancher/k3s/agent/	Agent data directory

Security Notes

• The kubeconfig file (/etc/rancher/k3s/k3s.yaml) contains admin credentials
• Never commit kubeconfig files to version control
• The node-token grants full cluster join permissions
• Keep SSH keys secure and use key-based authentication only
• Regularly rotate the node-token if compromised

Quick Reference Card

# Most common operations
kubectl get nodes                              # Check cluster nodes
kubectl get pods -A                            # List all pods
kubectl logs -f <pod> -n <ns>                  # Follow pod logs
kubectl exec -it <pod> -n <ns> -- bash         # Shell into pod
kubectl describe pod <pod> -n <ns>             # Debug pod issues
kubectl rollout restart deployment <dep> -n <ns> # Restart deployment

# Cluster access
ssh root@178.105.17.239                        # SSH to control plane
ssh -J root@178.105.17.239 root@10.42.1.2     # Jump to worker

# Emergency
systemctl restart k3s                          # Restart control plane
kubectl drain <node> --ignore-daemonsets       # Evacuate node
kubectl cordon <node>                          # Prevent scheduling
kubectl uncordon <node>                        # Allow scheduling

Additional Resources

• k3s Documentation
• kubectl Cheat Sheet
• K3S_OPERATIONS.md - Detailed operations guide