stack.basicstack.de/infrastructure/networking/NETWORK_ARCHITECTURE.md

BasicStack k3s Cluster Network Architecture

Last Updated: 2026-08-02
Cluster Version: k3s v1.36.2+k3s1

Overview

This document provides comprehensive documentation of the network configuration for the BasicStack k3s cluster running on Hetzner Cloud infrastructure.

Network Architecture Diagram

                                    Internet
                                       |
                            ┌──────────┴──────────┐
                            │  DNS (basicstack.de) │
                            │  *.basicstack.de     │
                            │  → 138.199.128.63    │
                            └──────────┬──────────┘
                                       |
                     ┌─────────────────┴─────────────────┐
                     │   Hetzner Load Balancer           │
                     │   IP: 138.199.128.63              │
                     │   Targets: k3s-cp-1 + workers 1-5 │
                     └─────────────────┬─────────────────┘
                                       |
                     ┌─────────────────┴─────────────────┐
                     │   Hetzner Cloud (fw-k3s)          │
                     │   Firewall: Port 80, 443, 25,     │
                     │             587, 465, 143, 993     │
                     └─────────────────┬─────────────────┘
                                       |
        ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
        ┃           k3s Cluster Network (10.42.0.0/16)                ┃
        ┃                                                              ┃
        ┃  ┌────────────────────────────────────────────────────┐    ┃
        ┃  │  k3s ServiceLB (LoadBalancer Controller)           │    ┃
        ┃  │  Distributes LoadBalancer IPs across worker nodes  │    ┃
        ┃  └────────────────────────────────────────────────────┘    ┃
        ┃                         |                                    ┃
        ┃         ┌───────────────┼───────────────┐                  ┃
        ┃         |               |               |                   ┃
        ┃    ┌────▼────┐    ┌────▼────┐    ┌────▼────┐             ┃
        ┃    │ worker-1│    │ worker-2│    │ worker-3│              ┃
        ┃    │10.42.1.2│    │10.42.1.3│    │10.42.1.5│              ┃
        ┃    └────┬────┘    └────┬────┘    └────┬────┘              ┃
        ┃         |               |               |                   ┃
        ┃    ┌────▼───────────────▼───────────────▼────┐            ┃
        ┃    │      Traefik Ingress Controller          │            ┃
        ┃    │      (HTTP/HTTPS routing)                 │            ┃
        ┃    └────┬──────────────────────────────────────┘           ┃
        ┃         |                                                    ┃
        ┃    ┌────▼────────────────────────────────────┐             ┃
        ┃    │  Application Services                    │             ┃
        ┃    │  - Paperclip, Forgejo, Directus, etc.   │             ┃
        ┃    │  - Stalwart SMTP/IMAP                    │             ┃
        ┃    │  - Nextcloud, BookStack, etc.            │             ┃
        ┃    └──────────────────────────────────────────┘            ┃
        ┃                                                              ┃
        ┃  CNI: Flannel (VXLAN backend over public IPs)              ┃
        ┃  Pod Network: 10.244.0.0/16                                 ┃
        ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Cluster Nodes

Node	Private IP	Public IP	Role	node-ip Config	Flannel Backend
k3s-cp-1	178.105.17.239	178.105.17.239	control-plane	(uses public)	vxlan
k3s-worker-1	10.42.1.2	178.105.216.48	worker	--node-ip=10.42.1.2	vxlan
k3s-worker-2	10.42.1.3	49.13.134.255	worker	--node-ip=10.42.1.3	vxlan
k3s-worker-3	10.42.1.5	167.233.121.121	worker	--node-ip=10.42.1.5	vxlan
k3s-update-runner	167.233.79.65	167.233.79.65	update-runner	(uses public)	vxlan

Critical Node Configuration

Worker Nodes MUST have --node-ip set to private IP:

• Without this flag, k3s defaults to the public IP
• LoadBalancer services advertise node IPs as EXTERNAL-IPs
• If workers use public IPs as node-ip, routing fails because internal cluster networking expects private IPs
• Fixed in DEV-224

Configuration Location: /etc/systemd/system/k3s-agent.service

Example:

ExecStart=/usr/local/bin/k3s agent --node-ip=10.42.1.2

After modifying:

systemctl daemon-reload
systemctl restart k3s-agent

CNI: Flannel Configuration

Overview

• CNI Plugin: Flannel (built-in with k3s)
• Backend Type: VXLAN
• Pod Network CIDR: 10.244.0.0/16
• Cluster Network CIDR: 10.42.0.0/16

Critical: Flannel Uses Public IPs for VXLAN Tunnels

Key Fact: Flannel VXLAN uses PUBLIC IPs for tunnels, not private IPs.

Each node has a Flannel annotation:

flannel.alpha.coreos.com/public-ip: <node-public-ip>
flannel.alpha.coreos.com/backend-type: vxlan

Impact:

• When adding a new node, its public IP must be added to Hetzner firewall fw-k3s
• If public IP is not in firewall, pods on that node cannot reach DNS or other pods
• Firewall must allow VXLAN traffic (UDP port 8472) between all node public IPs

Hetzner Firewall Configuration

Firewall Name: fw-k3s

Required Rules:

1. Allow public IPs of ALL cluster nodes for VXLAN (UDP 8472)
2. Allow external traffic: HTTP (80), HTTPS (443)
3. Allow mail traffic: SMTP (25, 587, 465), IMAP (143, 993)

When adding a new worker node:

# Get new node public IP first
hcloud server describe <server-name> -o json | jq -r '.public_net.ipv4.ip'

# Update firewall to include new public IP
hcloud firewall replace-rules fw-k3s \
  --rules-file fw-k3s-rules.json

Ingress Controller: Traefik

Configuration

• Type: LoadBalancer Service
• Controller: traefik.io/ingress-controller
• IngressClass: traefik (default)
• Namespace: kube-system
• Service Name: traefik

LoadBalancer IPs

Traefik LoadBalancer service exposes on:

10.42.1.2 (worker-1)
10.42.1.3 (worker-2)
10.42.1.5 (worker-3)
167.233.79.65 (update-runner)
178.105.17.239 (control-plane)

Ports:

• web: 80/TCP → 8000/TCP (NodePort: 31538)
• websecure: 443/TCP → 8443/TCP (NodePort: 32457)

k3s ServiceLB

k3s includes a built-in LoadBalancer controller (ServiceLB / Klipper-LB) that:

• Deploys DaemonSet pods on all nodes for each LoadBalancer service
• Advertises node IPs as LoadBalancer EXTERNAL-IPs
• Routes traffic to target pods via NodePort

Example DaemonSet: svclb-traefik-ee4e1a78 in kube-system namespace

Ingress Resources

All HTTP/HTTPS services use Traefik ingress with TLS:

Namespace	Ingress Name	Host(s)	TLS Certificate
basicstack-web	basicstack-web	basicstack.org, www.basicstack.org	basicstack-org-tls
bookstack	bookstack	books.basicstack.de	books-tls
container-registry	registry	registry.basicstack.de	registry-tls
directus	directus	directus.basicstack.de	directus-tls
forgejo	forgejo	forgejo.basicstack.de	forgejo-tls
monitoring	prometheus-ingress	prometheus.basicstack.de	prometheus-tls
monitoring	alertmanager-ingress	alertmanager.basicstack.de	alertmanager-tls
nextcloud	nextcloud	nextcloud.basicstack.de	nextcloud-tls
observability	grafana-ingress	grafana.basicstack.de	grafana-tls
opencloud	opencloud	opencloud.basicstack.de	opencloud-tls
paperclip	paperclip	paperclip.basicstack.de	paperclip-tls
passbolt	passbolt-ingress	passbolt.basicstack.de	passbolt-tls
planka	planka	planka.basicstack.de	planka-tls
pocket-id	pocket-id	auth.basicstack.de	pocket-id-tls
stalwart	stalwart-web	mail.basicstack.de	stalwart-tls
vaultwarden	vaultwarden	vaultwarden.basicstack.de	vaultwarden-tls

LoadBalancer Services

Traefik Ingress Controller

Service: kube-system/traefik
Type: LoadBalancer
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 167.233.79.65, 178.105.17.239
Ports:
  - 80/TCP (web)
  - 443/TCP (websecure)

Stalwart Mail Server

Stalwart uses a two-tier load balancing architecture combining Hetzner Cloud Load Balancer with Kubernetes LoadBalancer services.

Architecture

External Mail Client
    ↓
Hetzner Cloud Load Balancer (Managed)
  - Targets: k3s-cp-1, k3s-worker-1 through k3s-worker-5
  - Health checks enabled
    ↓
k3s Node NodePorts (auto-assigned by k3s ServiceLB)
    ↓
Kubernetes LoadBalancer Services (stalwart-smtp / stalwart-imap)
    ↓
Stalwart Pod (fsn1 datacenter, PV affinity)

Why This Architecture?

The Kubernetes services MUST be type LoadBalancer (not NodePort) because:

• Hetzner CCM Integration: The Hetzner Cloud Controller Manager automatically configures the Hetzner Load Balancer when it detects a Kubernetes LoadBalancer service
• Automatic Target Management: CCM keeps the Hetzner LB targets synchronized with cluster node changes
• Health Check Automation: CCM configures health checks based on the service configuration
• No Manual Intervention: Changing to NodePort would require manual Hetzner LB management

See DEV-439 for detailed investigation.

SMTP Service

Service: stalwart/stalwart-smtp
Type: LoadBalancer  # Required for Hetzner CCM integration
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
Ports:
  - 25/TCP (SMTP)
  - 587/TCP (Submission)

IMAP Service

Service: stalwart/stalwart-imap
Type: LoadBalancer  # Required for Hetzner CCM integration
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
Ports:
  - 993/TCP (IMAPS)  # Port 143 disabled per DEV-359

Historical Context

• Initial Setup: Used k3s ServiceLB with hostPort bindings (DEV-359)
• Migration: Migrated to Hetzner Load Balancer for stability (DEV-357)
• Current State: Hetzner LB + Kubernetes LoadBalancer services (managed by Hetzner CCM)
• Key Fix: Resolved CNI-HOSTPORT orphaned rules issue by adding external LB layer

DNS Configuration

Domain: basicstack.de

DNS Provider: Managed via Hetzner DNS

Required DNS Records

A Records (HTTP/HTTPS Services): All services point to the Hetzner Load Balancer IP address.

Current Configuration: All domains and subdomains point to: 138.199.128.63 (Hetzner Load Balancer)

Wildcard Option:

*.basicstack.de → 138.199.128.63

Specific Records:

basicstack.org → 138.199.128.63
www.basicstack.org → 138.199.128.63
paperclip.basicstack.de → 138.199.128.63
mail.basicstack.de → 138.199.128.63
auth.basicstack.de → 138.199.128.63
forgejo.basicstack.de → 138.199.128.63
directus.basicstack.de → 138.199.128.63
books.basicstack.de → 138.199.128.63
(etc. for all services)

MX Records (Mail):

basicstack.de MX 10 mail.basicstack.de

SPF Record:

basicstack.de TXT "v=spf1 a:mail.basicstack.de ~all"

DKIM Record:

default._domainkey.basicstack.de TXT "v=DKIM1; k=rsa; p=<public-key>"

DMARC Record:

_dmarc.basicstack.de TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@basicstack.de"

Internal DNS: CoreDNS

Service: kube-system/kube-dns Cluster Domain: cluster.local

CoreDNS configuration:

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      fallthrough in-addr.arpa ip6.arpa
    }
    hosts /etc/coredns/NodeHosts {
      ttl 60
      reload 15s
      fallthrough
    }
    prometheus :9153
    cache 30
    loop
    reload
    loadbalance
    forward . /etc/resolv.conf
}

Services within the cluster are accessible via:

• <service-name>.<namespace>.svc.cluster.local
• <service-name>.<namespace>

TLS Certificate Management

cert-manager

Purpose: Automated TLS certificate provisioning and renewal via Let's Encrypt

Certificate Status:

All ingress resources have associated Certificate resources. Current status:

• Most certificates: Ready = True
• container-registry/registry-tls: Ready = False (requires troubleshooting)

Certificate Reloading: Stakater Reloader

Purpose: Automatically restarts pods when their ConfigMaps or Secrets change

Why: When cert-manager renews a certificate, pods need to reload the updated TLS certificate. Reloader watches for Secret changes and triggers rolling restarts.

Configuration: See infrastructure/networking/reloader.yaml

How to use: Add annotation to Deployment/StatefulSet:

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

Or watch specific secrets:

metadata:
  annotations:
    reloader.stakater.com/search: "true"

Network Policies

Current Policies

Network policies restrict pod-to-pod communication for security:

Namespace	Policy Name	Pod Selector	Purpose
kube-system	allow-all	(none)	Default allow-all (cluster-wide)
nextcloud	nextcloud-postgresql	PostgreSQL pods	Restrict DB access
nextcloud	nextcloud-redis	Redis pods	Restrict cache access
planka	planka-postgresql	PostgreSQL pods	Restrict DB access

Default Policy

The kube-system/allow-all policy allows all traffic by default. This is a permissive configuration suitable for trusted internal cluster traffic.

Service-Specific Policies

Database and cache services (PostgreSQL, Redis) have restrictive policies that only allow traffic from their respective application pods.

Service Types Decision Matrix

Service Type	When to Use	LoadBalancer IPs	External Access
ClusterIP	Internal services only	N/A	No direct access
NodePort	Direct node access needed (rare)	N/A	Via NodeIP:NodePort
LoadBalancer	External access needed (HTTP/HTTPS/SMTP/IMAP)	Advertised from all nodes	Yes, via LoadBalancer IP
Ingress	HTTP/HTTPS with routing	Via Traefik LoadBalancer	Yes, with TLS termination

Decision Tree

1. Is the service HTTP/HTTPS?

   • Yes → Use Ingress (Traefik handles routing + TLS)
   • No → Continue

2. Does the service need external access?

   • No → Use ClusterIP
   • Yes → Continue

3. Is it a standard protocol (SMTP, IMAP, etc.)?

   • Yes → Use LoadBalancer service
   • No → Consider NodePort or LoadBalancer

Current Service Distribution

• ClusterIP: Most internal services (databases, caches, APIs)
• LoadBalancer: Traefik (3x), Stalwart SMTP, Stalwart IMAP
• Ingress: All HTTP/HTTPS services (via Traefik)

Traffic Flows

Ingress Traffic (HTTP/HTTPS)

Internet → DNS (→ 138.199.128.63)
    → Hetzner Load Balancer
    → k3s Node (k3s-cp-1 or k3s-worker-1 through k3s-worker-5)
    → Hetzner Firewall (fw-k3s)
    → NodePort (auto-assigned by k3s ServiceLB)
    → Traefik Ingress Controller
    → Backend Service (ClusterIP)
    → Application Pods

Mail Traffic (SMTP/IMAP) - Stalwart

Two-Tier Architecture (Hetzner Load Balancer + Kubernetes LoadBalancer):

Internet → DNS (mail.basicstack.de)
    → Hetzner Cloud Load Balancer (managed by Hetzner CCM)
    → k3s Node (any target: k3s-cp-1 or k3s-worker-1 through k3s-worker-5)
    → NodePort (auto-assigned by k3s ServiceLB)
    → Kubernetes LoadBalancer Service (stalwart-smtp / stalwart-imap)
    → Service DaemonSet Pod (svclb-stalwart-*)
    → Stalwart Pod (in fsn1 datacenter due to PV affinity)

Key Points:

• Hetzner CCM automatically manages the Hetzner Load Balancer configuration
• Service type MUST be LoadBalancer (not NodePort) for CCM integration
• k3s ServiceLB creates the NodePorts that the Hetzner LB targets
• Traffic flows through two load balancing layers for reliability

Internal Service-to-Service

Source Pod → ClusterIP Service → Target Pods
  (via Flannel VXLAN overlay network)

External API Calls (Egress)

Application Pod → Default Gateway → Node → Internet
  (No egress restrictions by default)

Network Troubleshooting

Common Issues and Solutions

1. Service Not Accessible Externally

Symptoms:

• Service times out or connection refused from external clients
• Works from within cluster

Diagnosis:

# Check service type and external IPs
kubectl get svc -n <namespace> <service-name>

# Check if LoadBalancer has assigned IPs
kubectl describe svc -n <namespace> <service-name> | grep "LoadBalancer Ingress"

# Check if Traefik ingress is configured
kubectl get ingress -n <namespace>

# Check Traefik logs
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik

Solutions:

• Verify DNS points to correct LoadBalancer IP
• Check Hetzner firewall allows required ports
• Verify ingress configuration and TLS certificate status
• Check that worker nodes have correct --node-ip configuration

2. DNS Resolution Failing

Symptoms:

• Pods cannot resolve external domains
• Inter-service DNS not working

Diagnosis:

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Test DNS from a pod
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup kubernetes.default

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

Solutions:

• Verify CoreDNS pods are running
• Check node's /etc/resolv.conf configuration
• Verify Flannel public IPs are in Hetzner firewall

3. Pod-to-Pod Communication Failing

Symptoms:

• Services cannot reach each other
• Random connection timeouts between pods

Diagnosis:

# Check Flannel annotations on nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.flannel\.alpha\.coreos\.com/public-ip}{"\n"}{end}'

# Check network policies
kubectl get networkpolicies -A

# Test connectivity from a pod
kubectl run -it --rm debug --image=busybox --restart=Never -- ping <other-pod-ip>

Solutions:

• Verify all node public IPs are in Hetzner firewall fw-k3s
• Check network policies aren't blocking traffic
• Verify Flannel VXLAN (UDP 8472) is allowed between all nodes

4. Certificate Issues

Symptoms:

• Browser shows certificate warnings
• Service returns 404 or connection refused on HTTPS

Diagnosis:

# Check certificate status
kubectl get certificate -n <namespace>

# Check certificate details
kubectl describe certificate -n <namespace> <cert-name>

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager

Solutions:

• Verify cert-manager is running and healthy
• Check that HTTP-01 challenge can reach the ingress
• Verify DNS points to correct LoadBalancer IP
• Manual certificate reissue: Delete certificate and let cert-manager recreate

5. LoadBalancer Service Not Getting IPs

Symptoms:

• LoadBalancer service shows <pending> for EXTERNAL-IP
• No IPs assigned after several minutes

Diagnosis:

# Check service status
kubectl describe svc -n <namespace> <service-name>

# Check ServiceLB DaemonSets
kubectl get ds -n kube-system | grep svclb

# Check ServiceLB controller logs (part of k3s)
kubectl logs -n kube-system -l svccontroller.k3s.cattle.io/svcname=<service-name>

Solutions:

• Verify worker nodes have --node-ip set to private IPs
• Check that nodes are in Ready state
• Restart k3s on affected nodes if needed

Diagnostic Commands

# Show all services with their types and IPs
kubectl get svc -A -o wide

# Show all ingress resources
kubectl get ingress -A

# Show all certificates
kubectl get certificate -A

# Show node details including IPs and annotations
kubectl get nodes -o json | jq -r '.items[] | {name: .metadata.name, internal_ip: (.status.addresses[] | select(.type=="InternalIP") | .address), external_ip: (.status.addresses[] | select(.type=="ExternalIP") | .address), flannel_public_ip: .metadata.annotations."flannel.alpha.coreos.com/public-ip"}'

# Check Flannel VXLAN interfaces on nodes (requires SSH)
ssh <node> ip -d link show flannel.1

# Check ServiceLB DaemonSets
kubectl get ds -n kube-system -l svccontroller.k3s.cattle.io/svcnamespace

# Test external connectivity from within cluster
kubectl run -it --rm curl --image=curlimages/curl --restart=Never -- curl -I https://<service>.<domain>

Maintenance Procedures

Adding a New Node

1. Provision node on Hetzner Cloud

2. Get node public IP:

   hcloud server describe <server-name> -o json | jq -r '.public_net.ipv4.ip'
   

3. Update Hetzner firewall:

   hcloud firewall replace-rules fw-k3s --rules-file fw-k3s-rules.json
   

   (Include new node's public IP)

4. Install k3s agent on new node:

   # For worker node - MUST include --node-ip
   curl -sfL https://get.k3s.io | K3S_URL=https://<control-plane>:6443 \
     K3S_TOKEN=<token> \
     INSTALL_K3S_EXEC="agent --node-ip=<private-ip>" sh -
   

5. Verify node joined:

   kubectl get nodes
   kubectl get nodes <node-name> -o jsonpath='{.metadata.annotations}'
   

Removing a Node

1. Drain node:

   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
   

2. Delete node from cluster:

   kubectl delete node <node-name>
   

3. Delete server on Hetzner Cloud

4. Update Hetzner firewall (remove node's public IP if no longer needed)

Updating DNS Records

Important: When updating DNS records for services, ensure:

1. DNS points to stable LoadBalancer IPs (control-plane or worker private IPs)
2. TTL is set appropriately (300-3600 seconds)
3. MX, SPF, DKIM, DMARC records are correct for mail services

Certificate Renewal

Certificates are automatically renewed by cert-manager 30 days before expiration. Reloader automatically restarts pods when certificates are updated.

Manual renewal (if needed):

# Delete certificate to trigger recreation
kubectl delete certificate -n <namespace> <cert-name>

# cert-manager will automatically recreate and issue
kubectl get certificate -n <namespace> <cert-name> -w

Security Considerations

Firewall Rules

• Only required ports are open (80, 443, 25, 587, 465, 143, 993)
• VXLAN traffic (UDP 8472) restricted to known node public IPs
• Regular review of firewall rules required

Network Policies

• Database and cache services have restrictive policies
• Consider implementing more restrictive default-deny policies for production
• Regularly audit network policy effectiveness

TLS/SSL

• All HTTP services must use HTTPS (enforced by Traefik)
• Certificate auto-renewal reduces manual intervention and expiration risk
• Regular monitoring of certificate status required

Node Access

• Worker nodes should use private IPs for --node-ip
• SSH access should be restricted to authorized IPs
• Regular security updates required

Related Documentation

• K3S Operations Guide
• Stalwart Mail Setup
• Certificate Renewal

Changelog

Date	Change	Author
2026-08-02	Document Hetzner Load Balancer architecture for Stalwart mail services	CTO (DEV-439)
2026-07-06	Initial comprehensive network architecture documentation	CTO (DEV-225)