# BasicStack k3s Cluster Network Architecture

**Last Updated:** 2026-07-06  
**Cluster Version:** k3s v1.36.2+k3s1

## Overview

This document provides comprehensive documentation of the network configuration for the BasicStack k3s cluster running on Hetzner Cloud infrastructure.

## Network Architecture Diagram

```
                                    Internet
                                       |
                            ┌──────────┴──────────┐
                            │  DNS (basicstack.de) │
                            │  *.basicstack.de     │
                            └──────────┬──────────┘
                                       |
                     ┌─────────────────┴─────────────────┐
                     │   Hetzner Cloud (fw-k3s)          │
                     │   Firewall: Port 80, 443, 25,     │
                     │             587, 465, 143, 993     │
                     └─────────────────┬─────────────────┘
                                       |
        ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
        ┃           k3s Cluster Network (10.42.0.0/16)                ┃
        ┃                                                              ┃
        ┃  ┌────────────────────────────────────────────────────┐    ┃
        ┃  │  k3s ServiceLB (LoadBalancer Controller)           │    ┃
        ┃  │  Distributes LoadBalancer IPs across worker nodes  │    ┃
        ┃  └────────────────────────────────────────────────────┘    ┃
        ┃                         |                                    ┃
        ┃         ┌───────────────┼───────────────┐                  ┃
        ┃         |               |               |                   ┃
        ┃    ┌────▼────┐    ┌────▼────┐    ┌────▼────┐             ┃
        ┃    │ worker-1│    │ worker-2│    │ worker-3│              ┃
        ┃    │10.42.1.2│    │10.42.1.3│    │10.42.1.5│              ┃
        ┃    └────┬────┘    └────┬────┘    └────┬────┘              ┃
        ┃         |               |               |                   ┃
        ┃    ┌────▼───────────────▼───────────────▼────┐            ┃
        ┃    │      Traefik Ingress Controller          │            ┃
        ┃    │      (HTTP/HTTPS routing)                 │            ┃
        ┃    └────┬──────────────────────────────────────┘           ┃
        ┃         |                                                    ┃
        ┃    ┌────▼────────────────────────────────────┐             ┃
        ┃    │  Application Services                    │             ┃
        ┃    │  - Paperclip, Forgejo, Directus, etc.   │             ┃
        ┃    │  - Stalwart SMTP/IMAP                    │             ┃
        ┃    │  - Nextcloud, BookStack, etc.            │             ┃
        ┃    └──────────────────────────────────────────┘            ┃
        ┃                                                              ┃
        ┃  CNI: Flannel (VXLAN backend over public IPs)              ┃
        ┃  Pod Network: 10.244.0.0/16                                 ┃
        ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```

## Cluster Nodes

| Node | Private IP | Public IP | Role | node-ip Config | Flannel Backend |
|------|-----------|-----------|------|----------------|-----------------|
| k3s-cp-1 | 178.105.17.239 | 178.105.17.239 | control-plane | (uses public) | vxlan |
| k3s-worker-1 | 10.42.1.2 | 178.105.216.48 | worker | `--node-ip=10.42.1.2` | vxlan |
| k3s-worker-2 | 10.42.1.3 | 49.13.134.255 | worker | `--node-ip=10.42.1.3` | vxlan |
| k3s-worker-3 | 10.42.1.5 | 167.233.121.121 | worker | `--node-ip=10.42.1.5` | vxlan |
| k3s-update-runner | 167.233.79.65 | 167.233.79.65 | update-runner | (uses public) | vxlan |

### Critical Node Configuration

**Worker Nodes MUST have `--node-ip` set to private IP:**
- Without this flag, k3s defaults to the public IP
- LoadBalancer services advertise node IPs as EXTERNAL-IPs
- If workers use public IPs as node-ip, routing fails because internal cluster networking expects private IPs
- Fixed in DEV-224

**Configuration Location:** `/etc/systemd/system/k3s-agent.service`

Example:
```bash
ExecStart=/usr/local/bin/k3s agent --node-ip=10.42.1.2
```

After modifying:
```bash
systemctl daemon-reload
systemctl restart k3s-agent
```

## CNI: Flannel Configuration

### Overview
- **CNI Plugin:** Flannel (built-in with k3s)
- **Backend Type:** VXLAN
- **Pod Network CIDR:** 10.244.0.0/16
- **Cluster Network CIDR:** 10.42.0.0/16

### Critical: Flannel Uses Public IPs for VXLAN Tunnels

**Key Fact:** Flannel VXLAN uses PUBLIC IPs for tunnels, not private IPs.

Each node has a Flannel annotation:
```
flannel.alpha.coreos.com/public-ip: <node-public-ip>
flannel.alpha.coreos.com/backend-type: vxlan
```

**Impact:**
- When adding a new node, its **public IP** must be added to Hetzner firewall `fw-k3s`
- If public IP is not in firewall, pods on that node cannot reach DNS or other pods
- Firewall must allow VXLAN traffic (UDP port 8472) between all node public IPs

### Hetzner Firewall Configuration

**Firewall Name:** `fw-k3s`

**Required Rules:**
1. Allow public IPs of ALL cluster nodes for VXLAN (UDP 8472)
2. Allow external traffic: HTTP (80), HTTPS (443)
3. Allow mail traffic: SMTP (25, 587, 465), IMAP (143, 993)

**When adding a new worker node:**
```bash
# Get new node public IP first
hcloud server describe <server-name> -o json | jq -r '.public_net.ipv4.ip'

# Update firewall to include new public IP
hcloud firewall replace-rules fw-k3s \
  --rules-file fw-k3s-rules.json
```

## Ingress Controller: Traefik

### Configuration
- **Type:** LoadBalancer Service
- **Controller:** traefik.io/ingress-controller
- **IngressClass:** `traefik` (default)
- **Namespace:** kube-system
- **Service Name:** traefik

### LoadBalancer IPs

Traefik LoadBalancer service exposes on:
```
10.42.1.2 (worker-1)
10.42.1.3 (worker-2)
10.42.1.5 (worker-3)
167.233.79.65 (update-runner)
178.105.17.239 (control-plane)
```

**Ports:**
- **web:** 80/TCP → 8000/TCP (NodePort: 31538)
- **websecure:** 443/TCP → 8443/TCP (NodePort: 32457)

### k3s ServiceLB

k3s includes a built-in LoadBalancer controller (ServiceLB / Klipper-LB) that:
- Deploys DaemonSet pods on all nodes for each LoadBalancer service
- Advertises node IPs as LoadBalancer EXTERNAL-IPs
- Routes traffic to target pods via NodePort

**Example DaemonSet:** `svclb-traefik-ee4e1a78` in kube-system namespace

### Ingress Resources

All HTTP/HTTPS services use Traefik ingress with TLS:

| Namespace | Ingress Name | Host(s) | TLS Certificate |
|-----------|-------------|---------|-----------------|
| basicstack-web | basicstack-web | basicstack.org, www.basicstack.org | basicstack-org-tls |
| bookstack | bookstack | books.basicstack.de | books-tls |
| container-registry | registry | registry.basicstack.de | registry-tls |
| directus | directus | directus.basicstack.de | directus-tls |
| forgejo | forgejo | forgejo.basicstack.de | forgejo-tls |
| monitoring | prometheus-ingress | prometheus.basicstack.de | prometheus-tls |
| monitoring | alertmanager-ingress | alertmanager.basicstack.de | alertmanager-tls |
| nextcloud | nextcloud | nextcloud.basicstack.de | nextcloud-tls |
| observability | grafana-ingress | grafana.basicstack.de | grafana-tls |
| opencloud | opencloud | opencloud.basicstack.de | opencloud-tls |
| paperclip | paperclip | paperclip.basicstack.de | paperclip-tls |
| passbolt | passbolt-ingress | passbolt.basicstack.de | passbolt-tls |
| planka | planka | planka.basicstack.de | planka-tls |
| pocket-id | pocket-id | auth.basicstack.de | pocket-id-tls |
| stalwart | stalwart-web | mail.basicstack.de | stalwart-tls |
| vaultwarden | vaultwarden | vaultwarden.basicstack.de | vaultwarden-tls |

## LoadBalancer Services

### Traefik Ingress Controller
```yaml
Service: kube-system/traefik
Type: LoadBalancer
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 167.233.79.65, 178.105.17.239
Ports:
  - 80/TCP (web)
  - 443/TCP (websecure)
```

### Stalwart Mail Server

#### SMTP Service
```yaml
Service: stalwart/stalwart-smtp
Type: LoadBalancer
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
Ports:
  - 25/TCP (SMTP)
  - 587/TCP (Submission)
  - 465/TCP (SMTPS)
```

#### IMAP Service
```yaml
Service: stalwart/stalwart-imap
Type: LoadBalancer
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
Ports:
  - 143/TCP (IMAP)
  - 993/TCP (IMAPS)
```

## DNS Configuration

### Domain: basicstack.de

**DNS Provider:** Managed via Hetzner DNS

#### Required DNS Records

**A Records (HTTP/HTTPS Services):**
All HTTPS services point to LoadBalancer IPs. Since we use multiple LoadBalancer IPs, DNS should point to the most stable/accessible IPs:

- Recommended: Point to worker node private IPs (10.42.1.2, 10.42.1.3, 10.42.1.5) OR
- Alternative: Point to control-plane public IP (178.105.17.239)

**Wildcard Option:**
```
*.basicstack.de → 178.105.17.239 (or round-robin to worker IPs)
```

**Specific Records:**
```
basicstack.org → 178.105.17.239
www.basicstack.org → 178.105.17.239
paperclip.basicstack.de → 178.105.17.239
mail.basicstack.de → 178.105.17.239
auth.basicstack.de → 178.105.17.239
(etc. for all services)
```

**MX Records (Mail):**
```
basicstack.de MX 10 mail.basicstack.de
```

**SPF Record:**
```
basicstack.de TXT "v=spf1 a:mail.basicstack.de ~all"
```

**DKIM Record:**
```
default._domainkey.basicstack.de TXT "v=DKIM1; k=rsa; p=<public-key>"
```

**DMARC Record:**
```
_dmarc.basicstack.de TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@basicstack.de"
```

### Internal DNS: CoreDNS

**Service:** kube-system/kube-dns
**Cluster Domain:** cluster.local

CoreDNS configuration:
```
.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      fallthrough in-addr.arpa ip6.arpa
    }
    hosts /etc/coredns/NodeHosts {
      ttl 60
      reload 15s
      fallthrough
    }
    prometheus :9153
    cache 30
    loop
    reload
    loadbalance
    forward . /etc/resolv.conf
}
```

Services within the cluster are accessible via:
- `<service-name>.<namespace>.svc.cluster.local`
- `<service-name>.<namespace>`

## TLS Certificate Management

### cert-manager

**Purpose:** Automated TLS certificate provisioning and renewal via Let's Encrypt

**Certificate Status:**

All ingress resources have associated Certificate resources. Current status:
- Most certificates: **Ready = True**
- `container-registry/registry-tls`: **Ready = False** (requires troubleshooting)

### Certificate Reloading: Stakater Reloader

**Purpose:** Automatically restarts pods when their ConfigMaps or Secrets change

**Why:** When cert-manager renews a certificate, pods need to reload the updated TLS certificate. Reloader watches for Secret changes and triggers rolling restarts.

**Configuration:** See `infrastructure/networking/reloader.yaml`

**How to use:**
Add annotation to Deployment/StatefulSet:
```yaml
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
```

Or watch specific secrets:
```yaml
metadata:
  annotations:
    reloader.stakater.com/search: "true"
```

## Network Policies

### Current Policies

Network policies restrict pod-to-pod communication for security:

| Namespace | Policy Name | Pod Selector | Purpose |
|-----------|-------------|--------------|---------|
| kube-system | allow-all | (none) | Default allow-all (cluster-wide) |
| nextcloud | nextcloud-postgresql | PostgreSQL pods | Restrict DB access |
| nextcloud | nextcloud-redis | Redis pods | Restrict cache access |
| planka | planka-postgresql | PostgreSQL pods | Restrict DB access |

### Default Policy

The `kube-system/allow-all` policy allows all traffic by default. This is a permissive configuration suitable for trusted internal cluster traffic.

### Service-Specific Policies

Database and cache services (PostgreSQL, Redis) have restrictive policies that only allow traffic from their respective application pods.

## Service Types Decision Matrix

| Service Type | When to Use | LoadBalancer IPs | External Access |
|--------------|-------------|------------------|-----------------|
| **ClusterIP** | Internal services only | N/A | No direct access |
| **NodePort** | Direct node access needed (rare) | N/A | Via NodeIP:NodePort |
| **LoadBalancer** | External access needed (HTTP/HTTPS/SMTP/IMAP) | Advertised from all nodes | Yes, via LoadBalancer IP |
| **Ingress** | HTTP/HTTPS with routing | Via Traefik LoadBalancer | Yes, with TLS termination |

### Decision Tree

1. **Is the service HTTP/HTTPS?**
   - Yes → Use **Ingress** (Traefik handles routing + TLS)
   - No → Continue

2. **Does the service need external access?**
   - No → Use **ClusterIP**
   - Yes → Continue

3. **Is it a standard protocol (SMTP, IMAP, etc.)?**
   - Yes → Use **LoadBalancer** service
   - No → Consider **NodePort** or **LoadBalancer**

### Current Service Distribution

- **ClusterIP:** Most internal services (databases, caches, APIs)
- **LoadBalancer:** Traefik (3x), Stalwart SMTP, Stalwart IMAP
- **Ingress:** All HTTP/HTTPS services (via Traefik)

## Traffic Flows

### Ingress Traffic (HTTP/HTTPS)

```
Internet → DNS → Hetzner Firewall (fw-k3s)
    → LoadBalancer IP (any worker node)
    → Traefik Ingress Controller (Pod on control-plane)
    → Backend Service (ClusterIP)
    → Application Pods
```

### Direct LoadBalancer Traffic (SMTP/IMAP)

```
Internet → DNS → Hetzner Firewall (fw-k3s)
    → LoadBalancer IP (any worker or control-plane node)
    → Service DaemonSet Pod (svclb-stalwart-*)
    → Backend Service (ClusterIP)
    → Stalwart Pods
```

### Internal Service-to-Service

```
Source Pod → ClusterIP Service → Target Pods
  (via Flannel VXLAN overlay network)
```

### External API Calls (Egress)

```
Application Pod → Default Gateway → Node → Internet
  (No egress restrictions by default)
```

## Network Troubleshooting

### Common Issues and Solutions

#### 1. Service Not Accessible Externally

**Symptoms:**
- Service times out or connection refused from external clients
- Works from within cluster

**Diagnosis:**
```bash
# Check service type and external IPs
kubectl get svc -n <namespace> <service-name>

# Check if LoadBalancer has assigned IPs
kubectl describe svc -n <namespace> <service-name> | grep "LoadBalancer Ingress"

# Check if Traefik ingress is configured
kubectl get ingress -n <namespace>

# Check Traefik logs
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik
```

**Solutions:**
- Verify DNS points to correct LoadBalancer IP
- Check Hetzner firewall allows required ports
- Verify ingress configuration and TLS certificate status
- Check that worker nodes have correct `--node-ip` configuration

#### 2. DNS Resolution Failing

**Symptoms:**
- Pods cannot resolve external domains
- Inter-service DNS not working

**Diagnosis:**
```bash
# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Test DNS from a pod
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup kubernetes.default

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns
```

**Solutions:**
- Verify CoreDNS pods are running
- Check node's `/etc/resolv.conf` configuration
- Verify Flannel public IPs are in Hetzner firewall

#### 3. Pod-to-Pod Communication Failing

**Symptoms:**
- Services cannot reach each other
- Random connection timeouts between pods

**Diagnosis:**
```bash
# Check Flannel annotations on nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.flannel\.alpha\.coreos\.com/public-ip}{"\n"}{end}'

# Check network policies
kubectl get networkpolicies -A

# Test connectivity from a pod
kubectl run -it --rm debug --image=busybox --restart=Never -- ping <other-pod-ip>
```

**Solutions:**
- Verify all node public IPs are in Hetzner firewall `fw-k3s`
- Check network policies aren't blocking traffic
- Verify Flannel VXLAN (UDP 8472) is allowed between all nodes

#### 4. Certificate Issues

**Symptoms:**
- Browser shows certificate warnings
- Service returns 404 or connection refused on HTTPS

**Diagnosis:**
```bash
# Check certificate status
kubectl get certificate -n <namespace>

# Check certificate details
kubectl describe certificate -n <namespace> <cert-name>

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager
```

**Solutions:**
- Verify cert-manager is running and healthy
- Check that HTTP-01 challenge can reach the ingress
- Verify DNS points to correct LoadBalancer IP
- Manual certificate reissue: Delete certificate and let cert-manager recreate

#### 5. LoadBalancer Service Not Getting IPs

**Symptoms:**
- LoadBalancer service shows `<pending>` for EXTERNAL-IP
- No IPs assigned after several minutes

**Diagnosis:**
```bash
# Check service status
kubectl describe svc -n <namespace> <service-name>

# Check ServiceLB DaemonSets
kubectl get ds -n kube-system | grep svclb

# Check ServiceLB controller logs (part of k3s)
kubectl logs -n kube-system -l svccontroller.k3s.cattle.io/svcname=<service-name>
```

**Solutions:**
- Verify worker nodes have `--node-ip` set to private IPs
- Check that nodes are in Ready state
- Restart k3s on affected nodes if needed

### Diagnostic Commands

```bash
# Show all services with their types and IPs
kubectl get svc -A -o wide

# Show all ingress resources
kubectl get ingress -A

# Show all certificates
kubectl get certificate -A

# Show node details including IPs and annotations
kubectl get nodes -o json | jq -r '.items[] | {name: .metadata.name, internal_ip: (.status.addresses[] | select(.type=="InternalIP") | .address), external_ip: (.status.addresses[] | select(.type=="ExternalIP") | .address), flannel_public_ip: .metadata.annotations."flannel.alpha.coreos.com/public-ip"}'

# Check Flannel VXLAN interfaces on nodes (requires SSH)
ssh <node> ip -d link show flannel.1

# Check ServiceLB DaemonSets
kubectl get ds -n kube-system -l svccontroller.k3s.cattle.io/svcnamespace

# Test external connectivity from within cluster
kubectl run -it --rm curl --image=curlimages/curl --restart=Never -- curl -I https://<service>.<domain>
```

## Maintenance Procedures

### Adding a New Node

1. **Provision node on Hetzner Cloud**
2. **Get node public IP:**
   ```bash
   hcloud server describe <server-name> -o json | jq -r '.public_net.ipv4.ip'
   ```

3. **Update Hetzner firewall:**
   ```bash
   hcloud firewall replace-rules fw-k3s --rules-file fw-k3s-rules.json
   ```
   (Include new node's public IP)

4. **Install k3s agent on new node:**
   ```bash
   # For worker node - MUST include --node-ip
   curl -sfL https://get.k3s.io | K3S_URL=https://<control-plane>:6443 \
     K3S_TOKEN=<token> \
     INSTALL_K3S_EXEC="agent --node-ip=<private-ip>" sh -
   ```

5. **Verify node joined:**
   ```bash
   kubectl get nodes
   kubectl get nodes <node-name> -o jsonpath='{.metadata.annotations}'
   ```

### Removing a Node

1. **Drain node:**
   ```bash
   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
   ```

2. **Delete node from cluster:**
   ```bash
   kubectl delete node <node-name>
   ```

3. **Delete server on Hetzner Cloud**

4. **Update Hetzner firewall** (remove node's public IP if no longer needed)

### Updating DNS Records

**Important:** When updating DNS records for services, ensure:
1. DNS points to stable LoadBalancer IPs (control-plane or worker private IPs)
2. TTL is set appropriately (300-3600 seconds)
3. MX, SPF, DKIM, DMARC records are correct for mail services

### Certificate Renewal

Certificates are automatically renewed by cert-manager 30 days before expiration. Reloader automatically restarts pods when certificates are updated.

**Manual renewal (if needed):**
```bash
# Delete certificate to trigger recreation
kubectl delete certificate -n <namespace> <cert-name>

# cert-manager will automatically recreate and issue
kubectl get certificate -n <namespace> <cert-name> -w
```

## Security Considerations

### Firewall Rules
- Only required ports are open (80, 443, 25, 587, 465, 143, 993)
- VXLAN traffic (UDP 8472) restricted to known node public IPs
- Regular review of firewall rules required

### Network Policies
- Database and cache services have restrictive policies
- Consider implementing more restrictive default-deny policies for production
- Regularly audit network policy effectiveness

### TLS/SSL
- All HTTP services must use HTTPS (enforced by Traefik)
- Certificate auto-renewal reduces manual intervention and expiration risk
- Regular monitoring of certificate status required

### Node Access
- Worker nodes should use private IPs for `--node-ip`
- SSH access should be restricted to authorized IPs
- Regular security updates required

## Related Documentation

- [K3S Operations Guide](../K3S_OPERATIONS.md)
- [Stalwart Mail Setup](../../apps/stalwart/README.md)
- [Certificate Renewal](../../apps/stalwart/CERTIFICATE-RENEWAL.md)

## Changelog

| Date | Change | Author |
|------|--------|--------|
| 2026-07-06 | Initial comprehensive network architecture documentation | CTO (DEV-225) |