Add memory limits to Argo CD components to prevent OOM incidents

Implements resource governance for all Argo CD components via kustomization
overlay. This prevents unlimited memory consumption that led to the control
plane resource exhaustion incident (DEV-281).

Resource limits applied:
- application-controller: 512Mi limit, 256Mi request
- repo-server: 512Mi limit, 256Mi request
- redis: 256Mi limit, 128Mi request
- server: 256Mi limit, 128Mi request
- notifications-controller: 128Mi limit, 64Mi request
- applicationset-controller: 256Mi limit, 128Mi request

The limits are based on observed usage patterns with headroom for growth
while preventing runaway memory consumption.

Usage: kubectl apply -k apps/argocd/

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-07-12 08:45:56 +00:00
parent a95e638f6b
commit b874f3d184
2 changed files with 180 additions and 0 deletions

95
apps/argocd/README.md Normal file
View file

@ -0,0 +1,95 @@
# Argo CD Deployment
This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster.
## Files
- `argocd-install.yaml` - Auto-generated Argo CD installation manifest (DO NOT EDIT DIRECTLY)
- `kustomization.yaml` - Kustomize overlay that adds resource limits and other customizations
- `argocd-ingress.yaml` - Ingress configuration for Argo CD UI
- `argocd-oidc-secret-sealed.yaml` - Sealed secret for OIDC integration
- `repo-*.yaml` - Sealed secrets for Git repository access
## Resource Limits
**IMPORTANT**: Resource limits were added after DEV-281 (resource exhaustion incident on 2026-07-12).
All Argo CD components now have memory limits to prevent OOM incidents:
| Component | Memory Limit | Memory Request |
|-----------|--------------|----------------|
| application-controller | 512Mi | 256Mi |
| repo-server | 512Mi | 256Mi |
| redis | 256Mi | 128Mi |
| server | 256Mi | 128Mi |
| notifications-controller | 128Mi | 64Mi |
| applicationset-controller | 256Mi | 128Mi |
These limits are based on observed usage patterns and provide headroom while preventing unlimited memory consumption.
## Deployment
### Option 1: Apply with kustomize (RECOMMENDED)
```bash
kubectl apply -k apps/argocd/
```
This will apply the base manifests plus all patches defined in `kustomization.yaml`.
### Option 2: Direct apply (not recommended)
```bash
kubectl apply -f apps/argocd/argocd-install.yaml
kubectl apply -f apps/argocd/argocd-ingress.yaml
# etc.
```
**Note**: This skips the resource limit patches and is NOT recommended.
## Updating Argo CD
When updating to a new Argo CD version:
1. Download the new install manifest:
```bash
curl -sSL https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml > argocd-install.yaml
```
2. Apply with kustomize (resource limits will be automatically applied):
```bash
kubectl apply -k apps/argocd/
```
3. Verify resource limits are in place:
```bash
kubectl get statefulset,deployment -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'
```
## Troubleshooting
### Check resource usage
```bash
kubectl top pods -n argocd
```
### Check if resource limits are applied
```bash
kubectl get deployment,statefulset -n argocd -o json | jq '.items[] | {name: .metadata.name, limits: .spec.template.spec.containers[0].resources.limits}'
```
### Rollback if needed
If there are issues after applying resource limits:
```bash
# Remove limits from a specific component
kubectl patch deployment -n argocd argocd-server --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/resources"}]'
```
## History
- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281)
- **2026-07-11**: Initial deployment

View file

@ -0,0 +1,85 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
resources:
- argocd-install.yaml
- argocd-ingress.yaml
- argocd-oidc-secret-sealed.yaml
- repo-basicstack-org-secret-sealed.yaml
- repo-stack-basicstack-de-secret-sealed.yaml
patches:
# Add memory limits to prevent OOM incidents (DEV-281)
- target:
kind: StatefulSet
name: argocd-application-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 512Mi
requests:
memory: 256Mi
- target:
kind: Deployment
name: argocd-repo-server
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 512Mi
requests:
memory: 256Mi
- target:
kind: Deployment
name: argocd-redis
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 256Mi
requests:
memory: 128Mi
- target:
kind: Deployment
name: argocd-server
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 256Mi
requests:
memory: 128Mi
- target:
kind: Deployment
name: argocd-notifications-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 128Mi
requests:
memory: 64Mi
- target:
kind: Deployment
name: argocd-applicationset-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 256Mi
requests:
memory: 128Mi