fix(stalwart): co-locate stalwart-backup pod with stalwart-0 (DEV-468)
The stalwart-backup CronJob has failed nightly since 2026-08-13, all with FailureTarget=DeadlineExceeded. Root cause: the backup pod had no scheduling constraint and got placed on a node different from stalwart-0. The hcloud CSI block volume is RWO and can only be attached to one node, so the backup pod stayed in ContainerCreating with FailedAttachVolume / Multi-Attach until the 600s active deadline killed it. - Add podAffinity requiredDuringScheduling on app=stalwart,statefulset.kubernetes.io/pod-name=stalwart-0 with topology key kubernetes.io/hostname so the backup pod always lands on the same node. Same-node co-location lets both pods share the already-attached block volume; the in-container script then scales stalwart-0 down, backs up, and scales it back up as before. - Raise activeDeadlineSeconds from 600s to 1800s as safety headroom (successful runs are ~86s; the extra budget covers prune growth). Verified: manual run of the patched CronJob completed in 86s and wrote restic snapshot f76f534e (2026-08-15 12:07:30) to s3://basicstack-backup/stalwart. stalwart-0 is back to Ready. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
parent
471b6a6109
commit
c4551495e0
1 changed files with 12 additions and 1 deletions
|
|
@ -289,7 +289,7 @@ spec:
|
|||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 600 # 10 minute timeout
|
||||
activeDeadlineSeconds: 1800 # 30 minute timeout (was 600s; retries + affinity settling can push past 10 min)
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
|
|
@ -297,6 +297,17 @@ spec:
|
|||
spec:
|
||||
serviceAccountName: stalwart-backup
|
||||
restartPolicy: OnFailure
|
||||
# Co-locate with stalwart-0 so both pods share the same hcloud block volume
|
||||
# attachment (RWO). Without this the backup pod can be scheduled on a
|
||||
# different node and hits FailedAttachVolume/Multi-Attach (DEV-468).
|
||||
affinity:
|
||||
podAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
- labelSelector:
|
||||
matchLabels:
|
||||
app: stalwart
|
||||
statefulset.kubernetes.io/pod-name: stalwart-0
|
||||
topologyKey: kubernetes.io/hostname
|
||||
containers:
|
||||
- name: backup
|
||||
image: alpine:3.19
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue