From b874f3d184156ce4ed9156286ab2c23b2c745132 Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 12 Jul 2026 08:45:56 +0000 Subject: [PATCH] Add memory limits to Argo CD components to prevent OOM incidents Implements resource governance for all Argo CD components via kustomization overlay. This prevents unlimited memory consumption that led to the control plane resource exhaustion incident (DEV-281). Resource limits applied: - application-controller: 512Mi limit, 256Mi request - repo-server: 512Mi limit, 256Mi request - redis: 256Mi limit, 128Mi request - server: 256Mi limit, 128Mi request - notifications-controller: 128Mi limit, 64Mi request - applicationset-controller: 256Mi limit, 128Mi request The limits are based on observed usage patterns with headroom for growth while preventing runaway memory consumption. Usage: kubectl apply -k apps/argocd/ Co-Authored-By: Paperclip --- apps/argocd/README.md | 95 ++++++++++++++++++++++++++++++++++ apps/argocd/kustomization.yaml | 85 ++++++++++++++++++++++++++++++ 2 files changed, 180 insertions(+) create mode 100644 apps/argocd/README.md create mode 100644 apps/argocd/kustomization.yaml diff --git a/apps/argocd/README.md b/apps/argocd/README.md new file mode 100644 index 0000000..5b5a003 --- /dev/null +++ b/apps/argocd/README.md @@ -0,0 +1,95 @@ +# Argo CD Deployment + +This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster. + +## Files + +- `argocd-install.yaml` - Auto-generated Argo CD installation manifest (DO NOT EDIT DIRECTLY) +- `kustomization.yaml` - Kustomize overlay that adds resource limits and other customizations +- `argocd-ingress.yaml` - Ingress configuration for Argo CD UI +- `argocd-oidc-secret-sealed.yaml` - Sealed secret for OIDC integration +- `repo-*.yaml` - Sealed secrets for Git repository access + +## Resource Limits + +**IMPORTANT**: Resource limits were added after DEV-281 (resource exhaustion incident on 2026-07-12). + +All Argo CD components now have memory limits to prevent OOM incidents: + +| Component | Memory Limit | Memory Request | +|-----------|--------------|----------------| +| application-controller | 512Mi | 256Mi | +| repo-server | 512Mi | 256Mi | +| redis | 256Mi | 128Mi | +| server | 256Mi | 128Mi | +| notifications-controller | 128Mi | 64Mi | +| applicationset-controller | 256Mi | 128Mi | + +These limits are based on observed usage patterns and provide headroom while preventing unlimited memory consumption. + +## Deployment + +### Option 1: Apply with kustomize (RECOMMENDED) + +```bash +kubectl apply -k apps/argocd/ +``` + +This will apply the base manifests plus all patches defined in `kustomization.yaml`. + +### Option 2: Direct apply (not recommended) + +```bash +kubectl apply -f apps/argocd/argocd-install.yaml +kubectl apply -f apps/argocd/argocd-ingress.yaml +# etc. +``` + +**Note**: This skips the resource limit patches and is NOT recommended. + +## Updating Argo CD + +When updating to a new Argo CD version: + +1. Download the new install manifest: + ```bash + curl -sSL https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml > argocd-install.yaml + ``` + +2. Apply with kustomize (resource limits will be automatically applied): + ```bash + kubectl apply -k apps/argocd/ + ``` + +3. Verify resource limits are in place: + ```bash + kubectl get statefulset,deployment -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory' + ``` + +## Troubleshooting + +### Check resource usage + +```bash +kubectl top pods -n argocd +``` + +### Check if resource limits are applied + +```bash +kubectl get deployment,statefulset -n argocd -o json | jq '.items[] | {name: .metadata.name, limits: .spec.template.spec.containers[0].resources.limits}' +``` + +### Rollback if needed + +If there are issues after applying resource limits: + +```bash +# Remove limits from a specific component +kubectl patch deployment -n argocd argocd-server --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/resources"}]' +``` + +## History + +- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281) +- **2026-07-11**: Initial deployment diff --git a/apps/argocd/kustomization.yaml b/apps/argocd/kustomization.yaml new file mode 100644 index 0000000..8d75623 --- /dev/null +++ b/apps/argocd/kustomization.yaml @@ -0,0 +1,85 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: argocd + +resources: + - argocd-install.yaml + - argocd-ingress.yaml + - argocd-oidc-secret-sealed.yaml + - repo-basicstack-org-secret-sealed.yaml + - repo-stack-basicstack-de-secret-sealed.yaml + +patches: + # Add memory limits to prevent OOM incidents (DEV-281) + - target: + kind: StatefulSet + name: argocd-application-controller + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 512Mi + requests: + memory: 256Mi + + - target: + kind: Deployment + name: argocd-repo-server + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 512Mi + requests: + memory: 256Mi + + - target: + kind: Deployment + name: argocd-redis + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 256Mi + requests: + memory: 128Mi + + - target: + kind: Deployment + name: argocd-server + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 256Mi + requests: + memory: 128Mi + + - target: + kind: Deployment + name: argocd-notifications-controller + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 128Mi + requests: + memory: 64Mi + + - target: + kind: Deployment + name: argocd-applicationset-controller + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 256Mi + requests: + memory: 128Mi