fix(stalwart): co-locate stalwart-backup pod with stalwart-0 (DEV-468)

The stalwart-backup CronJob has failed nightly since 2026-08-13, all with
FailureTarget=DeadlineExceeded. Root cause: the backup pod had no
scheduling constraint and got placed on a node different from stalwart-0.
The hcloud CSI block volume is RWO and can only be attached to one node,
so the backup pod stayed in ContainerCreating with FailedAttachVolume /
Multi-Attach until the 600s active deadline killed it.

- Add podAffinity requiredDuringScheduling on
  app=stalwart,statefulset.kubernetes.io/pod-name=stalwart-0 with topology
  key kubernetes.io/hostname so the backup pod always lands on the same
  node. Same-node co-location lets both pods share the already-attached
  block volume; the in-container script then scales stalwart-0 down,
  backs up, and scales it back up as before.
- Raise activeDeadlineSeconds from 600s to 1800s as safety headroom
  (successful runs are ~86s; the extra budget covers prune growth).

Verified: manual run of the patched CronJob completed in 86s and wrote
restic snapshot f76f534e (2026-08-15 12:07:30) to
s3://basicstack-backup/stalwart. stalwart-0 is back to Ready.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-08-15 12:11:22 +00:00
parent 471b6a6109
commit c4551495e0

View file

@ -289,7 +289,7 @@ spec:
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 600 # 10 minute timeout
activeDeadlineSeconds: 1800 # 30 minute timeout (was 600s; retries + affinity settling can push past 10 min)
template:
metadata:
labels:
@ -297,6 +297,17 @@ spec:
spec:
serviceAccountName: stalwart-backup
restartPolicy: OnFailure
# Co-locate with stalwart-0 so both pods share the same hcloud block volume
# attachment (RWO). Without this the backup pod can be scheduled on a
# different node and hits FailedAttachVolume/Multi-Attach (DEV-468).
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: stalwart
statefulset.kubernetes.io/pod-name: stalwart-0
topologyKey: kubernetes.io/hostname
containers:
- name: backup
image: alpine:3.19