129496d78d
- Add Hetzner Load Balancer (138.199.128.63) to network architecture diagram - Update DNS configuration to show all domains pointing to Hetzner LB IP - Update HTTP/HTTPS traffic flow to show traffic routing through Hetzner LB - Update last modified date to 2026-08-02 Related to DEV-439. Co-Authored-By: Paperclip <noreply@paperclip.ing>
729 lines
25 KiB
Markdown
729 lines
25 KiB
Markdown
# BasicStack k3s Cluster Network Architecture
|
|
|
|
**Last Updated:** 2026-08-02
|
|
**Cluster Version:** k3s v1.36.2+k3s1
|
|
|
|
## Overview
|
|
|
|
This document provides comprehensive documentation of the network configuration for the BasicStack k3s cluster running on Hetzner Cloud infrastructure.
|
|
|
|
## Network Architecture Diagram
|
|
|
|
```
|
|
Internet
|
|
|
|
|
┌──────────┴──────────┐
|
|
│ DNS (basicstack.de) │
|
|
│ *.basicstack.de │
|
|
│ → 138.199.128.63 │
|
|
└──────────┬──────────┘
|
|
|
|
|
┌─────────────────┴─────────────────┐
|
|
│ Hetzner Load Balancer │
|
|
│ IP: 138.199.128.63 │
|
|
│ Targets: k3s-cp-1 + workers 1-5 │
|
|
└─────────────────┬─────────────────┘
|
|
|
|
|
┌─────────────────┴─────────────────┐
|
|
│ Hetzner Cloud (fw-k3s) │
|
|
│ Firewall: Port 80, 443, 25, │
|
|
│ 587, 465, 143, 993 │
|
|
└─────────────────┬─────────────────┘
|
|
|
|
|
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
|
|
┃ k3s Cluster Network (10.42.0.0/16) ┃
|
|
┃ ┃
|
|
┃ ┌────────────────────────────────────────────────────┐ ┃
|
|
┃ │ k3s ServiceLB (LoadBalancer Controller) │ ┃
|
|
┃ │ Distributes LoadBalancer IPs across worker nodes │ ┃
|
|
┃ └────────────────────────────────────────────────────┘ ┃
|
|
┃ | ┃
|
|
┃ ┌───────────────┼───────────────┐ ┃
|
|
┃ | | | ┃
|
|
┃ ┌────▼────┐ ┌────▼────┐ ┌────▼────┐ ┃
|
|
┃ │ worker-1│ │ worker-2│ │ worker-3│ ┃
|
|
┃ │10.42.1.2│ │10.42.1.3│ │10.42.1.5│ ┃
|
|
┃ └────┬────┘ └────┬────┘ └────┬────┘ ┃
|
|
┃ | | | ┃
|
|
┃ ┌────▼───────────────▼───────────────▼────┐ ┃
|
|
┃ │ Traefik Ingress Controller │ ┃
|
|
┃ │ (HTTP/HTTPS routing) │ ┃
|
|
┃ └────┬──────────────────────────────────────┘ ┃
|
|
┃ | ┃
|
|
┃ ┌────▼────────────────────────────────────┐ ┃
|
|
┃ │ Application Services │ ┃
|
|
┃ │ - Paperclip, Forgejo, Directus, etc. │ ┃
|
|
┃ │ - Stalwart SMTP/IMAP │ ┃
|
|
┃ │ - Nextcloud, BookStack, etc. │ ┃
|
|
┃ └──────────────────────────────────────────┘ ┃
|
|
┃ ┃
|
|
┃ CNI: Flannel (VXLAN backend over public IPs) ┃
|
|
┃ Pod Network: 10.244.0.0/16 ┃
|
|
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
|
|
```
|
|
|
|
## Cluster Nodes
|
|
|
|
| Node | Private IP | Public IP | Role | node-ip Config | Flannel Backend |
|
|
|------|-----------|-----------|------|----------------|-----------------|
|
|
| k3s-cp-1 | 178.105.17.239 | 178.105.17.239 | control-plane | (uses public) | vxlan |
|
|
| k3s-worker-1 | 10.42.1.2 | 178.105.216.48 | worker | `--node-ip=10.42.1.2` | vxlan |
|
|
| k3s-worker-2 | 10.42.1.3 | 49.13.134.255 | worker | `--node-ip=10.42.1.3` | vxlan |
|
|
| k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | worker | `--node-ip=10.42.1.5` | vxlan |
|
|
| k3s-update-runner | 167.233.79.65 | 167.233.79.65 | update-runner | (uses public) | vxlan |
|
|
|
|
### Critical Node Configuration
|
|
|
|
**Worker Nodes MUST have `--node-ip` set to private IP:**
|
|
- Without this flag, k3s defaults to the public IP
|
|
- LoadBalancer services advertise node IPs as EXTERNAL-IPs
|
|
- If workers use public IPs as node-ip, routing fails because internal cluster networking expects private IPs
|
|
- Fixed in DEV-224
|
|
|
|
**Configuration Location:** `/etc/systemd/system/k3s-agent.service`
|
|
|
|
Example:
|
|
```bash
|
|
ExecStart=/usr/local/bin/k3s agent --node-ip=10.42.1.2
|
|
```
|
|
|
|
After modifying:
|
|
```bash
|
|
systemctl daemon-reload
|
|
systemctl restart k3s-agent
|
|
```
|
|
|
|
## CNI: Flannel Configuration
|
|
|
|
### Overview
|
|
- **CNI Plugin:** Flannel (built-in with k3s)
|
|
- **Backend Type:** VXLAN
|
|
- **Pod Network CIDR:** 10.244.0.0/16
|
|
- **Cluster Network CIDR:** 10.42.0.0/16
|
|
|
|
### Critical: Flannel Uses Public IPs for VXLAN Tunnels
|
|
|
|
**Key Fact:** Flannel VXLAN uses PUBLIC IPs for tunnels, not private IPs.
|
|
|
|
Each node has a Flannel annotation:
|
|
```
|
|
flannel.alpha.coreos.com/public-ip: <node-public-ip>
|
|
flannel.alpha.coreos.com/backend-type: vxlan
|
|
```
|
|
|
|
**Impact:**
|
|
- When adding a new node, its **public IP** must be added to Hetzner firewall `fw-k3s`
|
|
- If public IP is not in firewall, pods on that node cannot reach DNS or other pods
|
|
- Firewall must allow VXLAN traffic (UDP port 8472) between all node public IPs
|
|
|
|
### Hetzner Firewall Configuration
|
|
|
|
**Firewall Name:** `fw-k3s`
|
|
|
|
**Required Rules:**
|
|
1. Allow public IPs of ALL cluster nodes for VXLAN (UDP 8472)
|
|
2. Allow external traffic: HTTP (80), HTTPS (443)
|
|
3. Allow mail traffic: SMTP (25, 587, 465), IMAP (143, 993)
|
|
|
|
**When adding a new worker node:**
|
|
```bash
|
|
# Get new node public IP first
|
|
hcloud server describe <server-name> -o json | jq -r '.public_net.ipv4.ip'
|
|
|
|
# Update firewall to include new public IP
|
|
hcloud firewall replace-rules fw-k3s \
|
|
--rules-file fw-k3s-rules.json
|
|
```
|
|
|
|
## Ingress Controller: Traefik
|
|
|
|
### Configuration
|
|
- **Type:** LoadBalancer Service
|
|
- **Controller:** traefik.io/ingress-controller
|
|
- **IngressClass:** `traefik` (default)
|
|
- **Namespace:** kube-system
|
|
- **Service Name:** traefik
|
|
|
|
### LoadBalancer IPs
|
|
|
|
Traefik LoadBalancer service exposes on:
|
|
```
|
|
10.42.1.2 (worker-1)
|
|
10.42.1.3 (worker-2)
|
|
10.42.1.5 (worker-3)
|
|
167.233.79.65 (update-runner)
|
|
178.105.17.239 (control-plane)
|
|
```
|
|
|
|
**Ports:**
|
|
- **web:** 80/TCP → 8000/TCP (NodePort: 31538)
|
|
- **websecure:** 443/TCP → 8443/TCP (NodePort: 32457)
|
|
|
|
### k3s ServiceLB
|
|
|
|
k3s includes a built-in LoadBalancer controller (ServiceLB / Klipper-LB) that:
|
|
- Deploys DaemonSet pods on all nodes for each LoadBalancer service
|
|
- Advertises node IPs as LoadBalancer EXTERNAL-IPs
|
|
- Routes traffic to target pods via NodePort
|
|
|
|
**Example DaemonSet:** `svclb-traefik-ee4e1a78` in kube-system namespace
|
|
|
|
### Ingress Resources
|
|
|
|
All HTTP/HTTPS services use Traefik ingress with TLS:
|
|
|
|
| Namespace | Ingress Name | Host(s) | TLS Certificate |
|
|
|-----------|-------------|---------|-----------------|
|
|
| basicstack-web | basicstack-web | basicstack.org, www.basicstack.org | basicstack-org-tls |
|
|
| bookstack | bookstack | books.basicstack.de | books-tls |
|
|
| container-registry | registry | registry.basicstack.de | registry-tls |
|
|
| directus | directus | directus.basicstack.de | directus-tls |
|
|
| forgejo | forgejo | forgejo.basicstack.de | forgejo-tls |
|
|
| monitoring | prometheus-ingress | prometheus.basicstack.de | prometheus-tls |
|
|
| monitoring | alertmanager-ingress | alertmanager.basicstack.de | alertmanager-tls |
|
|
| nextcloud | nextcloud | nextcloud.basicstack.de | nextcloud-tls |
|
|
| observability | grafana-ingress | grafana.basicstack.de | grafana-tls |
|
|
| opencloud | opencloud | opencloud.basicstack.de | opencloud-tls |
|
|
| paperclip | paperclip | paperclip.basicstack.de | paperclip-tls |
|
|
| passbolt | passbolt-ingress | passbolt.basicstack.de | passbolt-tls |
|
|
| planka | planka | planka.basicstack.de | planka-tls |
|
|
| pocket-id | pocket-id | auth.basicstack.de | pocket-id-tls |
|
|
| stalwart | stalwart-web | mail.basicstack.de | stalwart-tls |
|
|
| vaultwarden | vaultwarden | vaultwarden.basicstack.de | vaultwarden-tls |
|
|
|
|
## LoadBalancer Services
|
|
|
|
### Traefik Ingress Controller
|
|
```yaml
|
|
Service: kube-system/traefik
|
|
Type: LoadBalancer
|
|
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 167.233.79.65, 178.105.17.239
|
|
Ports:
|
|
- 80/TCP (web)
|
|
- 443/TCP (websecure)
|
|
```
|
|
|
|
### Stalwart Mail Server
|
|
|
|
Stalwart uses a **two-tier load balancing architecture** combining Hetzner Cloud Load Balancer with Kubernetes LoadBalancer services.
|
|
|
|
#### Architecture
|
|
|
|
```
|
|
External Mail Client
|
|
↓
|
|
Hetzner Cloud Load Balancer (Managed)
|
|
- Targets: k3s-cp-1, k3s-worker-1 through k3s-worker-5
|
|
- Health checks enabled
|
|
↓
|
|
k3s Node NodePorts (auto-assigned by k3s ServiceLB)
|
|
↓
|
|
Kubernetes LoadBalancer Services (stalwart-smtp / stalwart-imap)
|
|
↓
|
|
Stalwart Pod (fsn1 datacenter, PV affinity)
|
|
```
|
|
|
|
#### Why This Architecture?
|
|
|
|
The Kubernetes services **MUST** be type `LoadBalancer` (not NodePort) because:
|
|
- **Hetzner CCM Integration**: The Hetzner Cloud Controller Manager automatically configures the Hetzner Load Balancer when it detects a Kubernetes LoadBalancer service
|
|
- **Automatic Target Management**: CCM keeps the Hetzner LB targets synchronized with cluster node changes
|
|
- **Health Check Automation**: CCM configures health checks based on the service configuration
|
|
- **No Manual Intervention**: Changing to NodePort would require manual Hetzner LB management
|
|
|
|
See DEV-439 for detailed investigation.
|
|
|
|
#### SMTP Service
|
|
```yaml
|
|
Service: stalwart/stalwart-smtp
|
|
Type: LoadBalancer # Required for Hetzner CCM integration
|
|
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
|
|
Ports:
|
|
- 25/TCP (SMTP)
|
|
- 587/TCP (Submission)
|
|
```
|
|
|
|
#### IMAP Service
|
|
```yaml
|
|
Service: stalwart/stalwart-imap
|
|
Type: LoadBalancer # Required for Hetzner CCM integration
|
|
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
|
|
Ports:
|
|
- 993/TCP (IMAPS) # Port 143 disabled per DEV-359
|
|
```
|
|
|
|
#### Historical Context
|
|
|
|
- **Initial Setup**: Used k3s ServiceLB with hostPort bindings (DEV-359)
|
|
- **Migration**: Migrated to Hetzner Load Balancer for stability (DEV-357)
|
|
- **Current State**: Hetzner LB + Kubernetes LoadBalancer services (managed by Hetzner CCM)
|
|
- **Key Fix**: Resolved CNI-HOSTPORT orphaned rules issue by adding external LB layer
|
|
|
|
## DNS Configuration
|
|
|
|
### Domain: basicstack.de
|
|
|
|
**DNS Provider:** Managed via Hetzner DNS
|
|
|
|
#### Required DNS Records
|
|
|
|
**A Records (HTTP/HTTPS Services):**
|
|
All services point to the Hetzner Load Balancer IP address.
|
|
|
|
**Current Configuration:**
|
|
All domains and subdomains point to: **138.199.128.63** (Hetzner Load Balancer)
|
|
|
|
**Wildcard Option:**
|
|
```
|
|
*.basicstack.de → 138.199.128.63
|
|
```
|
|
|
|
**Specific Records:**
|
|
```
|
|
basicstack.org → 138.199.128.63
|
|
www.basicstack.org → 138.199.128.63
|
|
paperclip.basicstack.de → 138.199.128.63
|
|
mail.basicstack.de → 138.199.128.63
|
|
auth.basicstack.de → 138.199.128.63
|
|
forgejo.basicstack.de → 138.199.128.63
|
|
directus.basicstack.de → 138.199.128.63
|
|
books.basicstack.de → 138.199.128.63
|
|
(etc. for all services)
|
|
```
|
|
|
|
**MX Records (Mail):**
|
|
```
|
|
basicstack.de MX 10 mail.basicstack.de
|
|
```
|
|
|
|
**SPF Record:**
|
|
```
|
|
basicstack.de TXT "v=spf1 a:mail.basicstack.de ~all"
|
|
```
|
|
|
|
**DKIM Record:**
|
|
```
|
|
default._domainkey.basicstack.de TXT "v=DKIM1; k=rsa; p=<public-key>"
|
|
```
|
|
|
|
**DMARC Record:**
|
|
```
|
|
_dmarc.basicstack.de TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@basicstack.de"
|
|
```
|
|
|
|
### Internal DNS: CoreDNS
|
|
|
|
**Service:** kube-system/kube-dns
|
|
**Cluster Domain:** cluster.local
|
|
|
|
CoreDNS configuration:
|
|
```
|
|
.:53 {
|
|
errors
|
|
health
|
|
ready
|
|
kubernetes cluster.local in-addr.arpa ip6.arpa {
|
|
pods insecure
|
|
fallthrough in-addr.arpa ip6.arpa
|
|
}
|
|
hosts /etc/coredns/NodeHosts {
|
|
ttl 60
|
|
reload 15s
|
|
fallthrough
|
|
}
|
|
prometheus :9153
|
|
cache 30
|
|
loop
|
|
reload
|
|
loadbalance
|
|
forward . /etc/resolv.conf
|
|
}
|
|
```
|
|
|
|
Services within the cluster are accessible via:
|
|
- `<service-name>.<namespace>.svc.cluster.local`
|
|
- `<service-name>.<namespace>`
|
|
|
|
## TLS Certificate Management
|
|
|
|
### cert-manager
|
|
|
|
**Purpose:** Automated TLS certificate provisioning and renewal via Let's Encrypt
|
|
|
|
**Certificate Status:**
|
|
|
|
All ingress resources have associated Certificate resources. Current status:
|
|
- Most certificates: **Ready = True**
|
|
- `container-registry/registry-tls`: **Ready = False** (requires troubleshooting)
|
|
|
|
### Certificate Reloading: Stakater Reloader
|
|
|
|
**Purpose:** Automatically restarts pods when their ConfigMaps or Secrets change
|
|
|
|
**Why:** When cert-manager renews a certificate, pods need to reload the updated TLS certificate. Reloader watches for Secret changes and triggers rolling restarts.
|
|
|
|
**Configuration:** See `infrastructure/networking/reloader.yaml`
|
|
|
|
**How to use:**
|
|
Add annotation to Deployment/StatefulSet:
|
|
```yaml
|
|
metadata:
|
|
annotations:
|
|
reloader.stakater.com/auto: "true"
|
|
```
|
|
|
|
Or watch specific secrets:
|
|
```yaml
|
|
metadata:
|
|
annotations:
|
|
reloader.stakater.com/search: "true"
|
|
```
|
|
|
|
## Network Policies
|
|
|
|
### Current Policies
|
|
|
|
Network policies restrict pod-to-pod communication for security:
|
|
|
|
| Namespace | Policy Name | Pod Selector | Purpose |
|
|
|-----------|-------------|--------------|---------|
|
|
| kube-system | allow-all | (none) | Default allow-all (cluster-wide) |
|
|
| nextcloud | nextcloud-postgresql | PostgreSQL pods | Restrict DB access |
|
|
| nextcloud | nextcloud-redis | Redis pods | Restrict cache access |
|
|
| planka | planka-postgresql | PostgreSQL pods | Restrict DB access |
|
|
|
|
### Default Policy
|
|
|
|
The `kube-system/allow-all` policy allows all traffic by default. This is a permissive configuration suitable for trusted internal cluster traffic.
|
|
|
|
### Service-Specific Policies
|
|
|
|
Database and cache services (PostgreSQL, Redis) have restrictive policies that only allow traffic from their respective application pods.
|
|
|
|
## Service Types Decision Matrix
|
|
|
|
| Service Type | When to Use | LoadBalancer IPs | External Access |
|
|
|--------------|-------------|------------------|-----------------|
|
|
| **ClusterIP** | Internal services only | N/A | No direct access |
|
|
| **NodePort** | Direct node access needed (rare) | N/A | Via NodeIP:NodePort |
|
|
| **LoadBalancer** | External access needed (HTTP/HTTPS/SMTP/IMAP) | Advertised from all nodes | Yes, via LoadBalancer IP |
|
|
| **Ingress** | HTTP/HTTPS with routing | Via Traefik LoadBalancer | Yes, with TLS termination |
|
|
|
|
### Decision Tree
|
|
|
|
1. **Is the service HTTP/HTTPS?**
|
|
- Yes → Use **Ingress** (Traefik handles routing + TLS)
|
|
- No → Continue
|
|
|
|
2. **Does the service need external access?**
|
|
- No → Use **ClusterIP**
|
|
- Yes → Continue
|
|
|
|
3. **Is it a standard protocol (SMTP, IMAP, etc.)?**
|
|
- Yes → Use **LoadBalancer** service
|
|
- No → Consider **NodePort** or **LoadBalancer**
|
|
|
|
### Current Service Distribution
|
|
|
|
- **ClusterIP:** Most internal services (databases, caches, APIs)
|
|
- **LoadBalancer:** Traefik (3x), Stalwart SMTP, Stalwart IMAP
|
|
- **Ingress:** All HTTP/HTTPS services (via Traefik)
|
|
|
|
## Traffic Flows
|
|
|
|
### Ingress Traffic (HTTP/HTTPS)
|
|
|
|
```
|
|
Internet → DNS (→ 138.199.128.63)
|
|
→ Hetzner Load Balancer
|
|
→ k3s Node (k3s-cp-1 or k3s-worker-1 through k3s-worker-5)
|
|
→ Hetzner Firewall (fw-k3s)
|
|
→ NodePort (auto-assigned by k3s ServiceLB)
|
|
→ Traefik Ingress Controller
|
|
→ Backend Service (ClusterIP)
|
|
→ Application Pods
|
|
```
|
|
|
|
### Mail Traffic (SMTP/IMAP) - Stalwart
|
|
|
|
**Two-Tier Architecture** (Hetzner Load Balancer + Kubernetes LoadBalancer):
|
|
|
|
```
|
|
Internet → DNS (mail.basicstack.de)
|
|
→ Hetzner Cloud Load Balancer (managed by Hetzner CCM)
|
|
→ k3s Node (any target: k3s-cp-1 or k3s-worker-1 through k3s-worker-5)
|
|
→ NodePort (auto-assigned by k3s ServiceLB)
|
|
→ Kubernetes LoadBalancer Service (stalwart-smtp / stalwart-imap)
|
|
→ Service DaemonSet Pod (svclb-stalwart-*)
|
|
→ Stalwart Pod (in fsn1 datacenter due to PV affinity)
|
|
```
|
|
|
|
**Key Points:**
|
|
- Hetzner CCM automatically manages the Hetzner Load Balancer configuration
|
|
- Service type MUST be LoadBalancer (not NodePort) for CCM integration
|
|
- k3s ServiceLB creates the NodePorts that the Hetzner LB targets
|
|
- Traffic flows through two load balancing layers for reliability
|
|
|
|
### Internal Service-to-Service
|
|
|
|
```
|
|
Source Pod → ClusterIP Service → Target Pods
|
|
(via Flannel VXLAN overlay network)
|
|
```
|
|
|
|
### External API Calls (Egress)
|
|
|
|
```
|
|
Application Pod → Default Gateway → Node → Internet
|
|
(No egress restrictions by default)
|
|
```
|
|
|
|
## Network Troubleshooting
|
|
|
|
### Common Issues and Solutions
|
|
|
|
#### 1. Service Not Accessible Externally
|
|
|
|
**Symptoms:**
|
|
- Service times out or connection refused from external clients
|
|
- Works from within cluster
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
# Check service type and external IPs
|
|
kubectl get svc -n <namespace> <service-name>
|
|
|
|
# Check if LoadBalancer has assigned IPs
|
|
kubectl describe svc -n <namespace> <service-name> | grep "LoadBalancer Ingress"
|
|
|
|
# Check if Traefik ingress is configured
|
|
kubectl get ingress -n <namespace>
|
|
|
|
# Check Traefik logs
|
|
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik
|
|
```
|
|
|
|
**Solutions:**
|
|
- Verify DNS points to correct LoadBalancer IP
|
|
- Check Hetzner firewall allows required ports
|
|
- Verify ingress configuration and TLS certificate status
|
|
- Check that worker nodes have correct `--node-ip` configuration
|
|
|
|
#### 2. DNS Resolution Failing
|
|
|
|
**Symptoms:**
|
|
- Pods cannot resolve external domains
|
|
- Inter-service DNS not working
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
# Check CoreDNS pods
|
|
kubectl get pods -n kube-system -l k8s-app=kube-dns
|
|
|
|
# Test DNS from a pod
|
|
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup kubernetes.default
|
|
|
|
# Check CoreDNS logs
|
|
kubectl logs -n kube-system -l k8s-app=kube-dns
|
|
```
|
|
|
|
**Solutions:**
|
|
- Verify CoreDNS pods are running
|
|
- Check node's `/etc/resolv.conf` configuration
|
|
- Verify Flannel public IPs are in Hetzner firewall
|
|
|
|
#### 3. Pod-to-Pod Communication Failing
|
|
|
|
**Symptoms:**
|
|
- Services cannot reach each other
|
|
- Random connection timeouts between pods
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
# Check Flannel annotations on nodes
|
|
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.flannel\.alpha\.coreos\.com/public-ip}{"\n"}{end}'
|
|
|
|
# Check network policies
|
|
kubectl get networkpolicies -A
|
|
|
|
# Test connectivity from a pod
|
|
kubectl run -it --rm debug --image=busybox --restart=Never -- ping <other-pod-ip>
|
|
```
|
|
|
|
**Solutions:**
|
|
- Verify all node public IPs are in Hetzner firewall `fw-k3s`
|
|
- Check network policies aren't blocking traffic
|
|
- Verify Flannel VXLAN (UDP 8472) is allowed between all nodes
|
|
|
|
#### 4. Certificate Issues
|
|
|
|
**Symptoms:**
|
|
- Browser shows certificate warnings
|
|
- Service returns 404 or connection refused on HTTPS
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
# Check certificate status
|
|
kubectl get certificate -n <namespace>
|
|
|
|
# Check certificate details
|
|
kubectl describe certificate -n <namespace> <cert-name>
|
|
|
|
# Check cert-manager logs
|
|
kubectl logs -n cert-manager -l app=cert-manager
|
|
```
|
|
|
|
**Solutions:**
|
|
- Verify cert-manager is running and healthy
|
|
- Check that HTTP-01 challenge can reach the ingress
|
|
- Verify DNS points to correct LoadBalancer IP
|
|
- Manual certificate reissue: Delete certificate and let cert-manager recreate
|
|
|
|
#### 5. LoadBalancer Service Not Getting IPs
|
|
|
|
**Symptoms:**
|
|
- LoadBalancer service shows `<pending>` for EXTERNAL-IP
|
|
- No IPs assigned after several minutes
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
# Check service status
|
|
kubectl describe svc -n <namespace> <service-name>
|
|
|
|
# Check ServiceLB DaemonSets
|
|
kubectl get ds -n kube-system | grep svclb
|
|
|
|
# Check ServiceLB controller logs (part of k3s)
|
|
kubectl logs -n kube-system -l svccontroller.k3s.cattle.io/svcname=<service-name>
|
|
```
|
|
|
|
**Solutions:**
|
|
- Verify worker nodes have `--node-ip` set to private IPs
|
|
- Check that nodes are in Ready state
|
|
- Restart k3s on affected nodes if needed
|
|
|
|
### Diagnostic Commands
|
|
|
|
```bash
|
|
# Show all services with their types and IPs
|
|
kubectl get svc -A -o wide
|
|
|
|
# Show all ingress resources
|
|
kubectl get ingress -A
|
|
|
|
# Show all certificates
|
|
kubectl get certificate -A
|
|
|
|
# Show node details including IPs and annotations
|
|
kubectl get nodes -o json | jq -r '.items[] | {name: .metadata.name, internal_ip: (.status.addresses[] | select(.type=="InternalIP") | .address), external_ip: (.status.addresses[] | select(.type=="ExternalIP") | .address), flannel_public_ip: .metadata.annotations."flannel.alpha.coreos.com/public-ip"}'
|
|
|
|
# Check Flannel VXLAN interfaces on nodes (requires SSH)
|
|
ssh <node> ip -d link show flannel.1
|
|
|
|
# Check ServiceLB DaemonSets
|
|
kubectl get ds -n kube-system -l svccontroller.k3s.cattle.io/svcnamespace
|
|
|
|
# Test external connectivity from within cluster
|
|
kubectl run -it --rm curl --image=curlimages/curl --restart=Never -- curl -I https://<service>.<domain>
|
|
```
|
|
|
|
## Maintenance Procedures
|
|
|
|
### Adding a New Node
|
|
|
|
1. **Provision node on Hetzner Cloud**
|
|
2. **Get node public IP:**
|
|
```bash
|
|
hcloud server describe <server-name> -o json | jq -r '.public_net.ipv4.ip'
|
|
```
|
|
|
|
3. **Update Hetzner firewall:**
|
|
```bash
|
|
hcloud firewall replace-rules fw-k3s --rules-file fw-k3s-rules.json
|
|
```
|
|
(Include new node's public IP)
|
|
|
|
4. **Install k3s agent on new node:**
|
|
```bash
|
|
# For worker node - MUST include --node-ip
|
|
curl -sfL https://get.k3s.io | K3S_URL=https://<control-plane>:6443 \
|
|
K3S_TOKEN=<token> \
|
|
INSTALL_K3S_EXEC="agent --node-ip=<private-ip>" sh -
|
|
```
|
|
|
|
5. **Verify node joined:**
|
|
```bash
|
|
kubectl get nodes
|
|
kubectl get nodes <node-name> -o jsonpath='{.metadata.annotations}'
|
|
```
|
|
|
|
### Removing a Node
|
|
|
|
1. **Drain node:**
|
|
```bash
|
|
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
|
|
```
|
|
|
|
2. **Delete node from cluster:**
|
|
```bash
|
|
kubectl delete node <node-name>
|
|
```
|
|
|
|
3. **Delete server on Hetzner Cloud**
|
|
|
|
4. **Update Hetzner firewall** (remove node's public IP if no longer needed)
|
|
|
|
### Updating DNS Records
|
|
|
|
**Important:** When updating DNS records for services, ensure:
|
|
1. DNS points to stable LoadBalancer IPs (control-plane or worker private IPs)
|
|
2. TTL is set appropriately (300-3600 seconds)
|
|
3. MX, SPF, DKIM, DMARC records are correct for mail services
|
|
|
|
### Certificate Renewal
|
|
|
|
Certificates are automatically renewed by cert-manager 30 days before expiration. Reloader automatically restarts pods when certificates are updated.
|
|
|
|
**Manual renewal (if needed):**
|
|
```bash
|
|
# Delete certificate to trigger recreation
|
|
kubectl delete certificate -n <namespace> <cert-name>
|
|
|
|
# cert-manager will automatically recreate and issue
|
|
kubectl get certificate -n <namespace> <cert-name> -w
|
|
```
|
|
|
|
## Security Considerations
|
|
|
|
### Firewall Rules
|
|
- Only required ports are open (80, 443, 25, 587, 465, 143, 993)
|
|
- VXLAN traffic (UDP 8472) restricted to known node public IPs
|
|
- Regular review of firewall rules required
|
|
|
|
### Network Policies
|
|
- Database and cache services have restrictive policies
|
|
- Consider implementing more restrictive default-deny policies for production
|
|
- Regularly audit network policy effectiveness
|
|
|
|
### TLS/SSL
|
|
- All HTTP services must use HTTPS (enforced by Traefik)
|
|
- Certificate auto-renewal reduces manual intervention and expiration risk
|
|
- Regular monitoring of certificate status required
|
|
|
|
### Node Access
|
|
- Worker nodes should use private IPs for `--node-ip`
|
|
- SSH access should be restricted to authorized IPs
|
|
- Regular security updates required
|
|
|
|
## Related Documentation
|
|
|
|
- [K3S Operations Guide](../K3S_OPERATIONS.md)
|
|
- [Stalwart Mail Setup](../../apps/stalwart/README.md)
|
|
- [Certificate Renewal](../../apps/stalwart/CERTIFICATE-RENEWAL.md)
|
|
|
|
## Changelog
|
|
|
|
| Date | Change | Author |
|
|
|------|--------|--------|
|
|
| 2026-08-02 | Document Hetzner Load Balancer architecture for Stalwart mail services | CTO (DEV-439) |
|
|
| 2026-07-06 | Initial comprehensive network architecture documentation | CTO (DEV-225) |
|