0c1c05fc50
The restic pipeline (loki/grafana/k8s-resources) is proven end-to-end after the DEV-488 restore drill, so the legacy rsync/tar pipeline into the 100 Gi local-path `backup-storage` PVC is removed. - Delete `apps/monitoring/backup-volumes-cronjob.yaml`. - Remove the DEV-483 bridge `nodeSelector: k3s-worker-2` from `apps/monitoring/loki-deployment.yaml`. Loki's data protection now runs via `backup-loki-restic`, which follows the pod via podAffinity regardless of which node the RWO CSI volume attaches on. The `Recreate` rollout strategy stays — it is unrelated (avoids the attach-deadlock during a rollout). Resolves the RWO/nodeSelector attach race that was blocking DEV-478 weekly OS updates. - Update `apps/monitoring/README.md` to drop the `backup-volumes` section, link the restic restore runbook, and record the pin removal. - Clean stale coexistence comments in the restic/prometheus CronJob manifests now that the legacy job is gone. Cluster-side (already applied out-of-band, since these manifests are `kubectl apply`-based, not Argo-managed): - `kubectl -n monitoring delete cronjob backup-volumes` -> NotFound. - `kubectl -n monitoring delete pvc backup-storage` -> gone; local-path PV `pvc-d0db0ba9-8f89-4f66-9e65-d573ebe1085a` reclaimed automatically (Delete policy). Two stale pre-DEV-487 `backup-k8s-resources` job pods that still referenced the PVC were deleted to release the `pvc-protection` finalizer. - `kubectl -n monitoring apply -f loki-deployment.yaml` -> Recreate rollout, new pod Ready in ~60s, no nodeSelector on the new spec. - No `VolumeAttachment` for the retired PV. - Restic CronJobs (`backup-loki-restic`, `backup-grafana-restic`, `backup-k8s-resources`, `prometheus-backup`) intact. Pre-delete snapshots retained on k3s-cp-1 under `/root/dev489-snapshots-20260816T155411Z/` for post-mortem. Co-Authored-By: Paperclip <noreply@paperclip.ing>
52 lines
1.3 KiB
YAML
52 lines
1.3 KiB
YAML
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: loki
|
|
namespace: monitoring
|
|
labels:
|
|
app: loki
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: loki
|
|
# loki-storage-encrypted is RWO on hcloud-volumes-encrypted, so a rolling
|
|
# update deadlocks (new pod cannot attach the PVC while the old pod holds
|
|
# it). Recreate drops the old pod first so the CSI detaches the volume
|
|
# before the new pod tries to attach it. Same pattern as harbor's RWO fix.
|
|
strategy:
|
|
type: Recreate
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: loki
|
|
spec:
|
|
containers:
|
|
- name: loki
|
|
image: grafana/loki:2.9.2
|
|
args:
|
|
- -config.file=/etc/loki/loki.yaml
|
|
ports:
|
|
- containerPort: 3100
|
|
name: http
|
|
- containerPort: 9096
|
|
name: grpc
|
|
resources:
|
|
requests:
|
|
cpu: 200m
|
|
memory: 512Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 1Gi
|
|
volumeMounts:
|
|
- name: loki-config
|
|
mountPath: /etc/loki
|
|
- name: loki-storage
|
|
mountPath: /loki
|
|
volumes:
|
|
- name: loki-config
|
|
configMap:
|
|
name: loki-config
|
|
- name: loki-storage
|
|
persistentVolumeClaim:
|
|
claimName: loki-storage-encrypted
|