From c4551495e05cdb523daef432b5e70aad0172d38e Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sat, 15 Aug 2026 12:11:22 +0000 Subject: [PATCH] fix(stalwart): co-locate stalwart-backup pod with stalwart-0 (DEV-468) The stalwart-backup CronJob has failed nightly since 2026-08-13, all with FailureTarget=DeadlineExceeded. Root cause: the backup pod had no scheduling constraint and got placed on a node different from stalwart-0. The hcloud CSI block volume is RWO and can only be attached to one node, so the backup pod stayed in ContainerCreating with FailedAttachVolume / Multi-Attach until the 600s active deadline killed it. - Add podAffinity requiredDuringScheduling on app=stalwart,statefulset.kubernetes.io/pod-name=stalwart-0 with topology key kubernetes.io/hostname so the backup pod always lands on the same node. Same-node co-location lets both pods share the already-attached block volume; the in-container script then scales stalwart-0 down, backs up, and scales it back up as before. - Raise activeDeadlineSeconds from 600s to 1800s as safety headroom (successful runs are ~86s; the extra budget covers prune growth). Verified: manual run of the patched CronJob completed in 86s and wrote restic snapshot f76f534e (2026-08-15 12:07:30) to s3://basicstack-backup/stalwart. stalwart-0 is back to Ready. Co-Authored-By: Paperclip --- apps/stalwart/stalwart-fresh-deployment.yaml | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/apps/stalwart/stalwart-fresh-deployment.yaml b/apps/stalwart/stalwart-fresh-deployment.yaml index 220edfa..5201c47 100644 --- a/apps/stalwart/stalwart-fresh-deployment.yaml +++ b/apps/stalwart/stalwart-fresh-deployment.yaml @@ -289,7 +289,7 @@ spec: jobTemplate: spec: backoffLimit: 2 - activeDeadlineSeconds: 600 # 10 minute timeout + activeDeadlineSeconds: 1800 # 30 minute timeout (was 600s; retries + affinity settling can push past 10 min) template: metadata: labels: @@ -297,6 +297,17 @@ spec: spec: serviceAccountName: stalwart-backup restartPolicy: OnFailure + # Co-locate with stalwart-0 so both pods share the same hcloud block volume + # attachment (RWO). Without this the backup pod can be scheduled on a + # different node and hits FailedAttachVolume/Multi-Attach (DEV-468). + affinity: + podAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app: stalwart + statefulset.kubernetes.io/pod-name: stalwart-0 + topologyKey: kubernetes.io/hostname containers: - name: backup image: alpine:3.19