stack.basicstack.de/apps/argocd
CTO ab60ef78f9 fix(argocd): bump application-controller memory 512Mi→1Gi (OOMKilled 1919× in 7d, DEV-596 preflight unblocker)
Container was repeatedly OOMKilled (exit 137) every ~5 s of live time since
2026-10-11 (pod age 6d23h, 1919 restarts). Live steady-state memory is now
~321 MiB with 1Gi/512Mi settings, so comfortable headroom. Patched the live
StatefulSet first to recover the control plane; this commit lets GitOps
retain the fix on the next reconcile.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-18 01:07:26 +00:00
..
argocd-ingress.yaml Add Argo CD installation manifests with Pocket ID SSO 2026-07-12 09:19:28 +00:00
argocd-oidc-secret-sealed.yaml fix(argocd): label argocd-oidc-secret so Argo CD reads its keys (DEV-523) 2026-08-23 10:37:01 +00:00
kustomization.yaml feat(argocd): switch to community Helm chart (DEV-519) 2026-08-23 08:32:23 +00:00
README.md fix(argocd): bump application-controller memory 512Mi→1Gi (OOMKilled 1919× in 7d, DEV-596 preflight unblocker) 2026-10-18 01:07:26 +00:00
repo-basicstack-org-secret-sealed.yaml Add managed label to argocd sealed secrets 2026-07-12 16:32:34 +00:00
repo-stack-basicstack-de-secret-sealed.yaml Add managed label to argocd sealed secrets 2026-07-12 16:32:34 +00:00
values.yaml fix(argocd): bump application-controller memory 512Mi→1Gi (OOMKilled 1919× in 7d, DEV-596 preflight unblocker) 2026-10-18 01:07:26 +00:00

Argo CD Deployment

This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster.

Argo CD itself is installed from the community Helm chart (argoproj/argo-helm, chart argo-cd). This directory holds the chart values file plus a small kustomize wrapper for the ingress and sealed secrets that stay in git.

Files

  • values.yaml - Helm values for the argo-cd chart (image tag, OIDC/RBAC config, resource limits, ingress disabled).
  • kustomization.yaml - Kustomize wrapper for the ingress + sealed secrets (does NOT install Argo CD itself).
  • argocd-ingress.yaml - Ingress configuration for the Argo CD UI (Traefik + cert-manager letsencrypt-prod).
  • argocd-oidc-secret-sealed.yaml - Sealed secret for Pocket ID OIDC integration.
  • repo-*.yaml - Sealed secrets for Git repository access.

How the install is wired

The root ../app-argocd.yaml is an Argo CD Application with two sources:

  1. The public Helm chart at https://argoproj.github.io/argo-helm, chart argo-cd, targetRevision pinned in git.
  2. This repo (ref: values) providing the values.yaml used by source (1) AND the ingress/sealed secrets applied via kustomize.

Once bootstrapped, Argo CD manages its own install by syncing this Application.

Resource limits

Memory limits were added after DEV-281 (resource exhaustion incident on 2026-07-12) and are now driven by values.yaml:

Component Memory Limit Memory Request
application-controller 1Gi 512Mi
repo-server 512Mi 256Mi
redis 256Mi 128Mi
server 256Mi 128Mi
notifications-controller 128Mi 64Mi
applicationset-controller 256Mi 128Mi

Deployment

Steady state (managed by Argo CD)

Once the cluster is bootstrapped, changes to this directory are picked up by the root argocd Application on the next sync. No manual kubectl apply is required.

First-time / disaster-recovery bootstrap

Argo CD cannot install itself while it is gone. Bootstrap with helm, then hand ownership back:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd \
  --version 10.4.0 \
  --namespace argocd --create-namespace \
  --values apps/argocd/values.yaml \
  --wait --timeout 10m

kubectl apply -k apps/argocd/          # ingress + sealed secrets
kubectl apply -f apps/app-argocd.yaml  # hand ownership back to GitOps

Updating Argo CD

Bump the chart and the image tag in a single PR:

  1. Refresh the local helm repo cache and check what's available:

    helm repo update
    helm search repo argo/argo-cd --versions | head
    
  2. Bump both fields together, keeping them in sync with the chart's appVersion:

    • apps/app-argocd.yaml -> spec.sources[0].targetRevision (chart version, e.g. 10.4.0)
    • apps/argocd/values.yaml -> global.image.tag (app version, e.g. v3.5.1)
  3. Optionally render locally to sanity-check the output before opening the PR:

    helm template argocd argo/argo-cd \
      --version <new-chart-version> \
      -f apps/argocd/values.yaml -n argocd | less
    
  4. Open the PR. After merge, Argo CD syncs itself onto the new version.

Troubleshooting

Check resource usage

kubectl top pods -n argocd

Check applied resource limits

kubectl get deployment,statefulset -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'

Rollback if a chart upgrade misbehaves

helm -n argocd history argocd
helm -n argocd rollback argocd <previous-revision>

History

  • 2026-08-23: Switched to the community Helm chart, bumped to v3.5.1 / chart 10.4.0 (DEV-519).
  • 2026-07-12: Added resource limits via kustomization to prevent OOM incidents (DEV-281).
  • 2026-07-11: Initial deployment (vendored install.yaml).