stack.basicstack.de/infrastructure/networking/NETWORK_ARCHITECTURE.md

BasicStack k3s Cluster Network Architecture

Last Updated: 2026-07-06
Cluster Version: k3s v1.36.2+k3s1

Overview

This document provides comprehensive documentation of the network configuration for the BasicStack k3s cluster running on Hetzner Cloud infrastructure.

Network Architecture Diagram

                                    Internet
                                       |
                            ┌──────────┴──────────┐
                            │  DNS (basicstack.de) │
                            │  *.basicstack.de     │
                            └──────────┬──────────┘
                                       |
                     ┌─────────────────┴─────────────────┐
                     │   Hetzner Cloud (fw-k3s)          │
                     │   Firewall: Port 80, 443, 25,     │
                     │             587, 465, 143, 993     │
                     └─────────────────┬─────────────────┘
                                       |
        ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
        ┃           k3s Cluster Network (10.42.0.0/16)                ┃
        ┃                                                              ┃
        ┃  ┌────────────────────────────────────────────────────┐    ┃
        ┃  │  k3s ServiceLB (LoadBalancer Controller)           │    ┃
        ┃  │  Distributes LoadBalancer IPs across worker nodes  │    ┃
        ┃  └────────────────────────────────────────────────────┘    ┃
        ┃                         |                                    ┃
        ┃         ┌───────────────┼───────────────┐                  ┃
        ┃         |               |               |                   ┃
        ┃    ┌────▼────┐    ┌────▼────┐    ┌────▼────┐             ┃
        ┃    │ worker-1│    │ worker-2│    │ worker-3│              ┃
        ┃    │10.42.1.2│    │10.42.1.3│    │10.42.1.5│              ┃
        ┃    └────┬────┘    └────┬────┘    └────┬────┘              ┃
        ┃         |               |               |                   ┃
        ┃    ┌────▼───────────────▼───────────────▼────┐            ┃
        ┃    │      Traefik Ingress Controller          │            ┃
        ┃    │      (HTTP/HTTPS routing)                 │            ┃
        ┃    └────┬──────────────────────────────────────┘           ┃
        ┃         |                                                    ┃
        ┃    ┌────▼────────────────────────────────────┐             ┃
        ┃    │  Application Services                    │             ┃
        ┃    │  - Paperclip, Forgejo, Directus, etc.   │             ┃
        ┃    │  - Stalwart SMTP/IMAP                    │             ┃
        ┃    │  - Nextcloud, BookStack, etc.            │             ┃
        ┃    └──────────────────────────────────────────┘            ┃
        ┃                                                              ┃
        ┃  CNI: Flannel (VXLAN backend over public IPs)              ┃
        ┃  Pod Network: 10.244.0.0/16                                 ┃
        ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Cluster Nodes

Node	Private IP	Public IP	Role	node-ip Config	Flannel Backend
k3s-cp-1	178.105.17.239	178.105.17.239	control-plane	(uses public)	vxlan
k3s-worker-1	10.42.1.2	178.105.216.48	worker	--node-ip=10.42.1.2	vxlan
k3s-worker-2	10.42.1.3	49.13.134.255	worker	--node-ip=10.42.1.3	vxlan
k3s-worker-3	10.42.1.5	167.233.121.121	worker	--node-ip=10.42.1.5	vxlan
k3s-update-runner	167.233.79.65	167.233.79.65	update-runner	(uses public)	vxlan

Critical Node Configuration

Worker Nodes MUST have --node-ip set to private IP:

• Without this flag, k3s defaults to the public IP
• LoadBalancer services advertise node IPs as EXTERNAL-IPs
• If workers use public IPs as node-ip, routing fails because internal cluster networking expects private IPs
• Fixed in DEV-224

Configuration Location: /etc/systemd/system/k3s-agent.service

Example:

ExecStart=/usr/local/bin/k3s agent --node-ip=10.42.1.2

After modifying:

systemctl daemon-reload
systemctl restart k3s-agent

CNI: Flannel Configuration

Overview

• CNI Plugin: Flannel (built-in with k3s)
• Backend Type: VXLAN
• Pod Network CIDR: 10.244.0.0/16
• Cluster Network CIDR: 10.42.0.0/16

Critical: Flannel Uses Public IPs for VXLAN Tunnels

Key Fact: Flannel VXLAN uses PUBLIC IPs for tunnels, not private IPs.

Each node has a Flannel annotation:

flannel.alpha.coreos.com/public-ip: <node-public-ip>
flannel.alpha.coreos.com/backend-type: vxlan

Impact:

• When adding a new node, its public IP must be added to Hetzner firewall fw-k3s
• If public IP is not in firewall, pods on that node cannot reach DNS or other pods
• Firewall must allow VXLAN traffic (UDP port 8472) between all node public IPs

Hetzner Firewall Configuration

Firewall Name: fw-k3s

Required Rules:

1. Allow public IPs of ALL cluster nodes for VXLAN (UDP 8472)
2. Allow external traffic: HTTP (80), HTTPS (443)
3. Allow mail traffic: SMTP (25, 587, 465), IMAP (143, 993)

When adding a new worker node:

# Get new node public IP first
hcloud server describe <server-name> -o json | jq -r '.public_net.ipv4.ip'

# Update firewall to include new public IP
hcloud firewall replace-rules fw-k3s \
  --rules-file fw-k3s-rules.json

Ingress Controller: Traefik

Configuration

• Type: LoadBalancer Service
• Controller: traefik.io/ingress-controller
• IngressClass: traefik (default)
• Namespace: kube-system
• Service Name: traefik

LoadBalancer IPs

Traefik LoadBalancer service exposes on:

10.42.1.2 (worker-1)
10.42.1.3 (worker-2)
10.42.1.5 (worker-3)
167.233.79.65 (update-runner)
178.105.17.239 (control-plane)

Ports:

• web: 80/TCP → 8000/TCP (NodePort: 31538)
• websecure: 443/TCP → 8443/TCP (NodePort: 32457)

k3s ServiceLB

k3s includes a built-in LoadBalancer controller (ServiceLB / Klipper-LB) that:

• Deploys DaemonSet pods on all nodes for each LoadBalancer service
• Advertises node IPs as LoadBalancer EXTERNAL-IPs
• Routes traffic to target pods via NodePort

Example DaemonSet: svclb-traefik-ee4e1a78 in kube-system namespace

Ingress Resources

All HTTP/HTTPS services use Traefik ingress with TLS:

Namespace	Ingress Name	Host(s)	TLS Certificate
basicstack-web	basicstack-web	basicstack.org, www.basicstack.org	basicstack-org-tls
bookstack	bookstack	books.basicstack.de	books-tls
container-registry	registry	registry.basicstack.de	registry-tls
directus	directus	directus.basicstack.de	directus-tls
forgejo	forgejo	forgejo.basicstack.de	forgejo-tls
monitoring	prometheus-ingress	prometheus.basicstack.de	prometheus-tls
monitoring	alertmanager-ingress	alertmanager.basicstack.de	alertmanager-tls
nextcloud	nextcloud	nextcloud.basicstack.de	nextcloud-tls
observability	grafana-ingress	grafana.basicstack.de	grafana-tls
opencloud	opencloud	opencloud.basicstack.de	opencloud-tls
paperclip	paperclip	paperclip.basicstack.de	paperclip-tls
passbolt	passbolt-ingress	passbolt.basicstack.de	passbolt-tls
planka	planka	planka.basicstack.de	planka-tls
pocket-id	pocket-id	auth.basicstack.de	pocket-id-tls
stalwart	stalwart-web	mail.basicstack.de	stalwart-tls
vaultwarden	vaultwarden	vaultwarden.basicstack.de	vaultwarden-tls

LoadBalancer Services

Traefik Ingress Controller

Service: kube-system/traefik
Type: LoadBalancer
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 167.233.79.65, 178.105.17.239
Ports:
  - 80/TCP (web)
  - 443/TCP (websecure)

Stalwart Mail Server

SMTP Service

Service: stalwart/stalwart-smtp
Type: LoadBalancer
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
Ports:
  - 25/TCP (SMTP)
  - 587/TCP (Submission)
  - 465/TCP (SMTPS)

IMAP Service

Service: stalwart/stalwart-imap
Type: LoadBalancer
External-IPs: 10.42.1.2, 10.42.1.3, 10.42.1.5, 178.105.17.239
Ports:
  - 143/TCP (IMAP)
  - 993/TCP (IMAPS)

DNS Configuration

Domain: basicstack.de

DNS Provider: Managed via Hetzner DNS

Required DNS Records

A Records (HTTP/HTTPS Services): All HTTPS services point to LoadBalancer IPs. Since we use multiple LoadBalancer IPs, DNS should point to the most stable/accessible IPs:

• Recommended: Point to worker node private IPs (10.42.1.2, 10.42.1.3, 10.42.1.5) OR
• Alternative: Point to control-plane public IP (178.105.17.239)

Wildcard Option:

*.basicstack.de → 178.105.17.239 (or round-robin to worker IPs)

Specific Records:

basicstack.org → 178.105.17.239
www.basicstack.org → 178.105.17.239
paperclip.basicstack.de → 178.105.17.239
mail.basicstack.de → 178.105.17.239
auth.basicstack.de → 178.105.17.239
(etc. for all services)

MX Records (Mail):

basicstack.de MX 10 mail.basicstack.de

SPF Record:

basicstack.de TXT "v=spf1 a:mail.basicstack.de ~all"

DKIM Record:

default._domainkey.basicstack.de TXT "v=DKIM1; k=rsa; p=<public-key>"

DMARC Record:

_dmarc.basicstack.de TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@basicstack.de"

Internal DNS: CoreDNS

Service: kube-system/kube-dns Cluster Domain: cluster.local

CoreDNS configuration:

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      fallthrough in-addr.arpa ip6.arpa
    }
    hosts /etc/coredns/NodeHosts {
      ttl 60
      reload 15s
      fallthrough
    }
    prometheus :9153
    cache 30
    loop
    reload
    loadbalance
    forward . /etc/resolv.conf
}

Services within the cluster are accessible via:

• <service-name>.<namespace>.svc.cluster.local
• <service-name>.<namespace>

TLS Certificate Management

cert-manager

Purpose: Automated TLS certificate provisioning and renewal via Let's Encrypt

Certificate Status:

All ingress resources have associated Certificate resources. Current status:

• Most certificates: Ready = True
• container-registry/registry-tls: Ready = False (requires troubleshooting)

Certificate Reloading: Stakater Reloader

Purpose: Automatically restarts pods when their ConfigMaps or Secrets change

Why: When cert-manager renews a certificate, pods need to reload the updated TLS certificate. Reloader watches for Secret changes and triggers rolling restarts.

Configuration: See infrastructure/networking/reloader.yaml

How to use: Add annotation to Deployment/StatefulSet:

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

Or watch specific secrets:

metadata:
  annotations:
    reloader.stakater.com/search: "true"

Network Policies

Current Policies

Network policies restrict pod-to-pod communication for security:

Namespace	Policy Name	Pod Selector	Purpose
kube-system	allow-all	(none)	Default allow-all (cluster-wide)
nextcloud	nextcloud-postgresql	PostgreSQL pods	Restrict DB access
nextcloud	nextcloud-redis	Redis pods	Restrict cache access
planka	planka-postgresql	PostgreSQL pods	Restrict DB access

Default Policy

The kube-system/allow-all policy allows all traffic by default. This is a permissive configuration suitable for trusted internal cluster traffic.

Service-Specific Policies

Database and cache services (PostgreSQL, Redis) have restrictive policies that only allow traffic from their respective application pods.

Service Types Decision Matrix

Service Type	When to Use	LoadBalancer IPs	External Access
ClusterIP	Internal services only	N/A	No direct access
NodePort	Direct node access needed (rare)	N/A	Via NodeIP:NodePort
LoadBalancer	External access needed (HTTP/HTTPS/SMTP/IMAP)	Advertised from all nodes	Yes, via LoadBalancer IP
Ingress	HTTP/HTTPS with routing	Via Traefik LoadBalancer	Yes, with TLS termination

Decision Tree

1. Is the service HTTP/HTTPS?

   • Yes → Use Ingress (Traefik handles routing + TLS)
   • No → Continue

2. Does the service need external access?

   • No → Use ClusterIP
   • Yes → Continue

3. Is it a standard protocol (SMTP, IMAP, etc.)?

   • Yes → Use LoadBalancer service
   • No → Consider NodePort or LoadBalancer

Current Service Distribution

• ClusterIP: Most internal services (databases, caches, APIs)
• LoadBalancer: Traefik (3x), Stalwart SMTP, Stalwart IMAP
• Ingress: All HTTP/HTTPS services (via Traefik)

Traffic Flows

Ingress Traffic (HTTP/HTTPS)

Internet → DNS → Hetzner Firewall (fw-k3s)
    → LoadBalancer IP (any worker node)
    → Traefik Ingress Controller (Pod on control-plane)
    → Backend Service (ClusterIP)
    → Application Pods

Direct LoadBalancer Traffic (SMTP/IMAP)

Internet → DNS → Hetzner Firewall (fw-k3s)
    → LoadBalancer IP (any worker or control-plane node)
    → Service DaemonSet Pod (svclb-stalwart-*)
    → Backend Service (ClusterIP)
    → Stalwart Pods

Internal Service-to-Service

Source Pod → ClusterIP Service → Target Pods
  (via Flannel VXLAN overlay network)

External API Calls (Egress)

Application Pod → Default Gateway → Node → Internet
  (No egress restrictions by default)

Network Troubleshooting

Common Issues and Solutions

1. Service Not Accessible Externally

Symptoms:

• Service times out or connection refused from external clients
• Works from within cluster

Diagnosis:

# Check service type and external IPs
kubectl get svc -n <namespace> <service-name>

# Check if LoadBalancer has assigned IPs
kubectl describe svc -n <namespace> <service-name> | grep "LoadBalancer Ingress"

# Check if Traefik ingress is configured
kubectl get ingress -n <namespace>

# Check Traefik logs
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik

Solutions:

• Verify DNS points to correct LoadBalancer IP
• Check Hetzner firewall allows required ports
• Verify ingress configuration and TLS certificate status
• Check that worker nodes have correct --node-ip configuration

2. DNS Resolution Failing

Symptoms:

• Pods cannot resolve external domains
• Inter-service DNS not working

Diagnosis:

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Test DNS from a pod
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup kubernetes.default

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

Solutions:

• Verify CoreDNS pods are running
• Check node's /etc/resolv.conf configuration
• Verify Flannel public IPs are in Hetzner firewall

3. Pod-to-Pod Communication Failing

Symptoms:

• Services cannot reach each other
• Random connection timeouts between pods

Diagnosis:

# Check Flannel annotations on nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.flannel\.alpha\.coreos\.com/public-ip}{"\n"}{end}'

# Check network policies
kubectl get networkpolicies -A

# Test connectivity from a pod
kubectl run -it --rm debug --image=busybox --restart=Never -- ping <other-pod-ip>

Solutions:

• Verify all node public IPs are in Hetzner firewall fw-k3s
• Check network policies aren't blocking traffic
• Verify Flannel VXLAN (UDP 8472) is allowed between all nodes

4. Certificate Issues

Symptoms:

• Browser shows certificate warnings
• Service returns 404 or connection refused on HTTPS

Diagnosis:

# Check certificate status
kubectl get certificate -n <namespace>

# Check certificate details
kubectl describe certificate -n <namespace> <cert-name>

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager

Solutions:

• Verify cert-manager is running and healthy
• Check that HTTP-01 challenge can reach the ingress
• Verify DNS points to correct LoadBalancer IP
• Manual certificate reissue: Delete certificate and let cert-manager recreate

5. LoadBalancer Service Not Getting IPs

Symptoms:

• LoadBalancer service shows <pending> for EXTERNAL-IP
• No IPs assigned after several minutes

Diagnosis:

# Check service status
kubectl describe svc -n <namespace> <service-name>

# Check ServiceLB DaemonSets
kubectl get ds -n kube-system | grep svclb

# Check ServiceLB controller logs (part of k3s)
kubectl logs -n kube-system -l svccontroller.k3s.cattle.io/svcname=<service-name>

Solutions:

• Verify worker nodes have --node-ip set to private IPs
• Check that nodes are in Ready state
• Restart k3s on affected nodes if needed

Diagnostic Commands

# Show all services with their types and IPs
kubectl get svc -A -o wide

# Show all ingress resources
kubectl get ingress -A

# Show all certificates
kubectl get certificate -A

# Show node details including IPs and annotations
kubectl get nodes -o json | jq -r '.items[] | {name: .metadata.name, internal_ip: (.status.addresses[] | select(.type=="InternalIP") | .address), external_ip: (.status.addresses[] | select(.type=="ExternalIP") | .address), flannel_public_ip: .metadata.annotations."flannel.alpha.coreos.com/public-ip"}'

# Check Flannel VXLAN interfaces on nodes (requires SSH)
ssh <node> ip -d link show flannel.1

# Check ServiceLB DaemonSets
kubectl get ds -n kube-system -l svccontroller.k3s.cattle.io/svcnamespace

# Test external connectivity from within cluster
kubectl run -it --rm curl --image=curlimages/curl --restart=Never -- curl -I https://<service>.<domain>

Maintenance Procedures

Adding a New Node

1. Provision node on Hetzner Cloud

2. Get node public IP:

   hcloud server describe <server-name> -o json | jq -r '.public_net.ipv4.ip'
   

3. Update Hetzner firewall:

   hcloud firewall replace-rules fw-k3s --rules-file fw-k3s-rules.json
   

   (Include new node's public IP)

4. Install k3s agent on new node:

   # For worker node - MUST include --node-ip
   curl -sfL https://get.k3s.io | K3S_URL=https://<control-plane>:6443 \
     K3S_TOKEN=<token> \
     INSTALL_K3S_EXEC="agent --node-ip=<private-ip>" sh -
   

5. Verify node joined:

   kubectl get nodes
   kubectl get nodes <node-name> -o jsonpath='{.metadata.annotations}'
   

Removing a Node

1. Drain node:

   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
   

2. Delete node from cluster:

   kubectl delete node <node-name>
   

3. Delete server on Hetzner Cloud

4. Update Hetzner firewall (remove node's public IP if no longer needed)

Updating DNS Records

Important: When updating DNS records for services, ensure:

1. DNS points to stable LoadBalancer IPs (control-plane or worker private IPs)
2. TTL is set appropriately (300-3600 seconds)
3. MX, SPF, DKIM, DMARC records are correct for mail services

Certificate Renewal

Certificates are automatically renewed by cert-manager 30 days before expiration. Reloader automatically restarts pods when certificates are updated.

Manual renewal (if needed):

# Delete certificate to trigger recreation
kubectl delete certificate -n <namespace> <cert-name>

# cert-manager will automatically recreate and issue
kubectl get certificate -n <namespace> <cert-name> -w

Security Considerations

Firewall Rules

• Only required ports are open (80, 443, 25, 587, 465, 143, 993)
• VXLAN traffic (UDP 8472) restricted to known node public IPs
• Regular review of firewall rules required

Network Policies

• Database and cache services have restrictive policies
• Consider implementing more restrictive default-deny policies for production
• Regularly audit network policy effectiveness

TLS/SSL

• All HTTP services must use HTTPS (enforced by Traefik)
• Certificate auto-renewal reduces manual intervention and expiration risk
• Regular monitoring of certificate status required

Node Access

• Worker nodes should use private IPs for --node-ip
• SSH access should be restricted to authorized IPs
• Regular security updates required

Related Documentation

• K3S Operations Guide
• Stalwart Mail Setup
• Certificate Renewal

Changelog

Date	Change	Author
2026-07-06	Initial comprehensive network architecture documentation	CTO (DEV-225)