diff --git a/apps/argocd/README.md b/apps/argocd/README.md new file mode 100644 index 0000000..5b5a003 --- /dev/null +++ b/apps/argocd/README.md @@ -0,0 +1,95 @@ +# Argo CD Deployment + +This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster. + +## Files + +- `argocd-install.yaml` - Auto-generated Argo CD installation manifest (DO NOT EDIT DIRECTLY) +- `kustomization.yaml` - Kustomize overlay that adds resource limits and other customizations +- `argocd-ingress.yaml` - Ingress configuration for Argo CD UI +- `argocd-oidc-secret-sealed.yaml` - Sealed secret for OIDC integration +- `repo-*.yaml` - Sealed secrets for Git repository access + +## Resource Limits + +**IMPORTANT**: Resource limits were added after DEV-281 (resource exhaustion incident on 2026-07-12). + +All Argo CD components now have memory limits to prevent OOM incidents: + +| Component | Memory Limit | Memory Request | +|-----------|--------------|----------------| +| application-controller | 512Mi | 256Mi | +| repo-server | 512Mi | 256Mi | +| redis | 256Mi | 128Mi | +| server | 256Mi | 128Mi | +| notifications-controller | 128Mi | 64Mi | +| applicationset-controller | 256Mi | 128Mi | + +These limits are based on observed usage patterns and provide headroom while preventing unlimited memory consumption. + +## Deployment + +### Option 1: Apply with kustomize (RECOMMENDED) + +```bash +kubectl apply -k apps/argocd/ +``` + +This will apply the base manifests plus all patches defined in `kustomization.yaml`. + +### Option 2: Direct apply (not recommended) + +```bash +kubectl apply -f apps/argocd/argocd-install.yaml +kubectl apply -f apps/argocd/argocd-ingress.yaml +# etc. +``` + +**Note**: This skips the resource limit patches and is NOT recommended. + +## Updating Argo CD + +When updating to a new Argo CD version: + +1. Download the new install manifest: + ```bash + curl -sSL https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml > argocd-install.yaml + ``` + +2. Apply with kustomize (resource limits will be automatically applied): + ```bash + kubectl apply -k apps/argocd/ + ``` + +3. Verify resource limits are in place: + ```bash + kubectl get statefulset,deployment -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory' + ``` + +## Troubleshooting + +### Check resource usage + +```bash +kubectl top pods -n argocd +``` + +### Check if resource limits are applied + +```bash +kubectl get deployment,statefulset -n argocd -o json | jq '.items[] | {name: .metadata.name, limits: .spec.template.spec.containers[0].resources.limits}' +``` + +### Rollback if needed + +If there are issues after applying resource limits: + +```bash +# Remove limits from a specific component +kubectl patch deployment -n argocd argocd-server --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/resources"}]' +``` + +## History + +- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281) +- **2026-07-11**: Initial deployment diff --git a/apps/argocd/kustomization.yaml b/apps/argocd/kustomization.yaml new file mode 100644 index 0000000..8d75623 --- /dev/null +++ b/apps/argocd/kustomization.yaml @@ -0,0 +1,85 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: argocd + +resources: + - argocd-install.yaml + - argocd-ingress.yaml + - argocd-oidc-secret-sealed.yaml + - repo-basicstack-org-secret-sealed.yaml + - repo-stack-basicstack-de-secret-sealed.yaml + +patches: + # Add memory limits to prevent OOM incidents (DEV-281) + - target: + kind: StatefulSet + name: argocd-application-controller + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 512Mi + requests: + memory: 256Mi + + - target: + kind: Deployment + name: argocd-repo-server + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 512Mi + requests: + memory: 256Mi + + - target: + kind: Deployment + name: argocd-redis + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 256Mi + requests: + memory: 128Mi + + - target: + kind: Deployment + name: argocd-server + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 256Mi + requests: + memory: 128Mi + + - target: + kind: Deployment + name: argocd-notifications-controller + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 128Mi + requests: + memory: 64Mi + + - target: + kind: Deployment + name: argocd-applicationset-controller + patch: |- + - op: add + path: /spec/template/spec/containers/0/resources + value: + limits: + memory: 256Mi + requests: + memory: 128Mi