b874f3d184
Implements resource governance for all Argo CD components via kustomization overlay. This prevents unlimited memory consumption that led to the control plane resource exhaustion incident (DEV-281). Resource limits applied: - application-controller: 512Mi limit, 256Mi request - repo-server: 512Mi limit, 256Mi request - redis: 256Mi limit, 128Mi request - server: 256Mi limit, 128Mi request - notifications-controller: 128Mi limit, 64Mi request - applicationset-controller: 256Mi limit, 128Mi request The limits are based on observed usage patterns with headroom for growth while preventing runaway memory consumption. Usage: kubectl apply -k apps/argocd/ Co-Authored-By: Paperclip <noreply@paperclip.ing>
95 lines
2.8 KiB
Markdown
95 lines
2.8 KiB
Markdown
# Argo CD Deployment
|
|
|
|
This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster.
|
|
|
|
## Files
|
|
|
|
- `argocd-install.yaml` - Auto-generated Argo CD installation manifest (DO NOT EDIT DIRECTLY)
|
|
- `kustomization.yaml` - Kustomize overlay that adds resource limits and other customizations
|
|
- `argocd-ingress.yaml` - Ingress configuration for Argo CD UI
|
|
- `argocd-oidc-secret-sealed.yaml` - Sealed secret for OIDC integration
|
|
- `repo-*.yaml` - Sealed secrets for Git repository access
|
|
|
|
## Resource Limits
|
|
|
|
**IMPORTANT**: Resource limits were added after DEV-281 (resource exhaustion incident on 2026-07-12).
|
|
|
|
All Argo CD components now have memory limits to prevent OOM incidents:
|
|
|
|
| Component | Memory Limit | Memory Request |
|
|
|-----------|--------------|----------------|
|
|
| application-controller | 512Mi | 256Mi |
|
|
| repo-server | 512Mi | 256Mi |
|
|
| redis | 256Mi | 128Mi |
|
|
| server | 256Mi | 128Mi |
|
|
| notifications-controller | 128Mi | 64Mi |
|
|
| applicationset-controller | 256Mi | 128Mi |
|
|
|
|
These limits are based on observed usage patterns and provide headroom while preventing unlimited memory consumption.
|
|
|
|
## Deployment
|
|
|
|
### Option 1: Apply with kustomize (RECOMMENDED)
|
|
|
|
```bash
|
|
kubectl apply -k apps/argocd/
|
|
```
|
|
|
|
This will apply the base manifests plus all patches defined in `kustomization.yaml`.
|
|
|
|
### Option 2: Direct apply (not recommended)
|
|
|
|
```bash
|
|
kubectl apply -f apps/argocd/argocd-install.yaml
|
|
kubectl apply -f apps/argocd/argocd-ingress.yaml
|
|
# etc.
|
|
```
|
|
|
|
**Note**: This skips the resource limit patches and is NOT recommended.
|
|
|
|
## Updating Argo CD
|
|
|
|
When updating to a new Argo CD version:
|
|
|
|
1. Download the new install manifest:
|
|
```bash
|
|
curl -sSL https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml > argocd-install.yaml
|
|
```
|
|
|
|
2. Apply with kustomize (resource limits will be automatically applied):
|
|
```bash
|
|
kubectl apply -k apps/argocd/
|
|
```
|
|
|
|
3. Verify resource limits are in place:
|
|
```bash
|
|
kubectl get statefulset,deployment -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Check resource usage
|
|
|
|
```bash
|
|
kubectl top pods -n argocd
|
|
```
|
|
|
|
### Check if resource limits are applied
|
|
|
|
```bash
|
|
kubectl get deployment,statefulset -n argocd -o json | jq '.items[] | {name: .metadata.name, limits: .spec.template.spec.containers[0].resources.limits}'
|
|
```
|
|
|
|
### Rollback if needed
|
|
|
|
If there are issues after applying resource limits:
|
|
|
|
```bash
|
|
# Remove limits from a specific component
|
|
kubectl patch deployment -n argocd argocd-server --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/resources"}]'
|
|
```
|
|
|
|
## History
|
|
|
|
- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281)
|
|
- **2026-07-11**: Initial deployment
|