basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

chore(stalwart): retire RocksDB PVC + backup — Postgres backend authoritative (DEV-474) #10

Open
forgejoadmin wants to merge 1 commit from feat/DEV-474-rocksdb-teardown into main
pull from: feat/DEV-474-rocksdb-teardown
merge into: basicstack:main
Conversation 0 Commits 1 Files changed 2 +37 -224
forgejoadmin commented 2026-08-28 15:12:51 +00:00
Owner
Copy link

Post-cutover 14-day observation window on DEV-474 closed today (2026-08-28) with 14/14 healthy days. This PR retires every RocksDB-era artefact from the Stalwart deployment and updates the README to reflect the PostgreSQL backend as the default.

What this removes

  • PVC stalwart-data (20 Gi, hcloud-volumes-encrypted, RWO) — declaration deleted from the manifest.
  • StatefulSet stalwart:
    • fix-permissions init-container (only existed to chown the RocksDB PVC mount).
    • data volume replaced with emptyDir; /var/lib/stalwart remains as a plain scratch path (Stalwart state now lives in stalwart-postgres).
  • stalwart-backup CronJob + its stalwart-backup ServiceAccount / Role / RoleBinding — RocksDB snapshotting is obsolete.

What already ran (out-of-band, in the DEV-548 heartbeat)

  • restic forget --prune against tag=stalwart — all 11 RocksDB snapshots removed (16.165 MiB pruned). Log preserved on DEV-474. restic snapshots --tag stalwart now returns empty; only the 14 tag=stalwart-postgres snapshots remain in the shared repo.
  • stalwart-backup CronJob was pre-suspended in-cluster (suspend: true) to prevent tonight's 03:00 UTC fire from re-creating a stalwart-tagged snapshot before this PR merges. Merging this PR then removes the CronJob entirely.

Rollback path

Rollback authority is now the Postgres pg_dump restic chain (tag=stalwart-postgres, keep 14 daily / 8 weekly / 6 monthly, latest snapshot db127a39 from 2026-08-28 02:30 UTC).

If pre-merge rollback is needed for any reason: revert this branch, un-suspend the stalwart-backup CronJob (kubectl -n stalwart patch cronjob stalwart-backup -p '{"spec":{"suspend":false}}') — the CronJob would then reinitialise its restic tag from the live PVC on its next fire.

Server-side validation

kubectl apply --dry-run=server against the modified manifest reports the six remaining objects reconcile cleanly:

namespace/stalwart configured (server dry run)
service/stalwart-smtp configured (server dry run)
service/stalwart-imap configured (server dry run)
service/stalwart-http configured (server dry run)
statefulset.apps/stalwart configured (server dry run)
ingress.networking.k8s.io/stalwart-web configured (server dry run)

Note: on merge, ArgoCD/GitOps will delete PVC stalwart-data (20 Gi RocksDB volume) as a pruned resource. The volume is 55 days old and untouched by writes for 14 days.

Refs

Post-cutover 14-day observation window on **[DEV-474](/DEV/issues/DEV-474)** closed today (2026-08-28) with 14/14 healthy days. This PR retires every RocksDB-era artefact from the Stalwart deployment and updates the README to reflect the PostgreSQL backend as the default. ### What this removes - PVC `stalwart-data` (20 Gi, `hcloud-volumes-encrypted`, RWO) — declaration deleted from the manifest. - StatefulSet `stalwart`: - `fix-permissions` init-container (only existed to chown the RocksDB PVC mount). - `data` volume replaced with `emptyDir`; `/var/lib/stalwart` remains as a plain scratch path (Stalwart state now lives in `stalwart-postgres`). - `stalwart-backup` CronJob + its `stalwart-backup` ServiceAccount / Role / RoleBinding — RocksDB snapshotting is obsolete. ### What already ran (out-of-band, in the DEV-548 heartbeat) - `restic forget --prune` against `tag=stalwart` — all 11 RocksDB snapshots removed (16.165 MiB pruned). Log preserved on DEV-474. `restic snapshots --tag stalwart` now returns empty; only the 14 `tag=stalwart-postgres` snapshots remain in the shared repo. - `stalwart-backup` CronJob was pre-suspended in-cluster (`suspend: true`) to prevent tonight's 03:00 UTC fire from re-creating a stalwart-tagged snapshot before this PR merges. Merging this PR then removes the CronJob entirely. ### Rollback path Rollback authority is now the Postgres `pg_dump` restic chain (`tag=stalwart-postgres`, keep 14 daily / 8 weekly / 6 monthly, latest snapshot `db127a39` from 2026-08-28 02:30 UTC). If pre-merge rollback is needed for any reason: revert this branch, un-suspend the `stalwart-backup` CronJob (`kubectl -n stalwart patch cronjob stalwart-backup -p '{"spec":{"suspend":false}}'`) — the CronJob would then reinitialise its restic tag from the live PVC on its next fire. ### Server-side validation `kubectl apply --dry-run=server` against the modified manifest reports the six remaining objects reconcile cleanly: ``` namespace/stalwart configured (server dry run) service/stalwart-smtp configured (server dry run) service/stalwart-imap configured (server dry run) service/stalwart-http configured (server dry run) statefulset.apps/stalwart configured (server dry run) ingress.networking.k8s.io/stalwart-web configured (server dry run) ``` Note: on merge, ArgoCD/GitOps will delete PVC `stalwart-data` (20 Gi RocksDB volume) as a pruned resource. The volume is 55 days old and untouched by writes for 14 days. ### Refs - Cutover: [DEV-473](/DEV/issues/DEV-473) - Migration parent: [DEV-467](/DEV/issues/DEV-467) - This teardown: [DEV-474](/DEV/issues/DEV-474) - Day-14 execution issue: [DEV-548](/DEV/issues/DEV-548)
forgejoadmin added 1 commit 2026-08-28 15:12:51 +00:00 
Post-cutover 14-day observation window on DEV-474 closed today with 14/14
healthy days. This removes every RocksDB-era artefact from the manifest and
updates the README to reflect the PostgreSQL backend as the default:

- Delete PVC `stalwart-data` (was 20Gi hcloud-volumes-encrypted, RWO).
- Drop the `fix-permissions` init-container (existed only to chown the
  RocksDB PVC mount).
- Replace the StatefulSet's `data` volume with an `emptyDir` and keep the
  `/var/lib/stalwart` mount as a plain scratch path (Stalwart's state is
  now in the `stalwart-postgres` StatefulSet).
- Delete the `stalwart-backup` CronJob and its `stalwart-backup`
  ServiceAccount / Role / RoleBinding — RocksDB snapshotting is obsolete
  now that state lives in Postgres. Daily backup is via
  `stalwart-postgres-backup` (restic tag `stalwart-postgres`).
- README: update version, backend, storage, backup, files, and deployment
  sections; correct exposed-ports language to match SMTP 25/587 + IMAPS 993.

Executed alongside the merge:

- `restic forget --prune` removed all 11 snapshots on tag=stalwart (Job
  `stalwart-restic-forget-rocksdb` on 2026-08-28 15:09 UTC, log preserved
  on DEV-474). 16.165 MiB pruned. `restic snapshots --tag stalwart` now
  returns empty; only the 14 tag=`stalwart-postgres` snapshots remain.
- `stalwart-backup` CronJob was pre-suspended in-cluster to prevent
  tonight's 03:00 UTC fire from re-creating a stalwart-tagged snapshot
  before this PR merges.

Rollback (if needed before merge): revert this commit; the old CronJob
would then re-create RocksDB snapshots from live PVC contents. After
merge the rollback path is the Postgres `pg_dump` restic chain only.

Refs: DEV-467, DEV-473 (cutover), DEV-474 (this observation + teardown),
DEV-548 (day-14 execution issue).

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/DEV-474-rocksdb-teardown:feat/DEV-474-rocksdb-teardown
git checkout feat/DEV-474-rocksdb-teardown

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git checkout main
git merge --no-ff feat/DEV-474-rocksdb-teardown
git checkout feat/DEV-474-rocksdb-teardown
git rebase main
git checkout main
git merge --ff-only feat/DEV-474-rocksdb-teardown
git checkout feat/DEV-474-rocksdb-teardown
git rebase main
git checkout main
git merge --no-ff feat/DEV-474-rocksdb-teardown
git checkout main
git merge --squash feat/DEV-474-rocksdb-teardown
git checkout main
git merge --ff-only feat/DEV-474-rocksdb-teardown
git checkout main
git merge feat/DEV-474-rocksdb-teardown
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: basicstack/stack.basicstack.de#10
No description provided.
Delete branch "feat/DEV-474-rocksdb-teardown"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?