Compare commits
No commits in common. "82de4507a89495c77fc8f3638962a5c1c150722e" and "e848d225cd37cc7d6d5c56b67da02486ff905a18" have entirely different histories.
82de4507a8
...
e848d225cd
5 changed files with 33487 additions and 152 deletions
|
|
@ -3,27 +3,16 @@ kind: Application
|
||||||
metadata:
|
metadata:
|
||||||
name: argocd
|
name: argocd
|
||||||
namespace: argocd
|
namespace: argocd
|
||||||
finalizers:
|
|
||||||
- resources-finalizer.argocd.argoproj.io
|
|
||||||
spec:
|
spec:
|
||||||
project: default
|
project: default
|
||||||
|
source:
|
||||||
|
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
|
||||||
|
targetRevision: main
|
||||||
|
path: apps/argocd
|
||||||
destination:
|
destination:
|
||||||
server: https://kubernetes.default.svc
|
server: https://kubernetes.default.svc
|
||||||
namespace: argocd
|
namespace: argocd
|
||||||
sources:
|
|
||||||
- repoURL: https://argoproj.github.io/argo-helm
|
|
||||||
chart: argo-cd
|
|
||||||
targetRevision: 10.4.0
|
|
||||||
helm:
|
|
||||||
releaseName: argocd
|
|
||||||
valueFiles:
|
|
||||||
- $values/apps/argocd/values.yaml
|
|
||||||
- repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
|
|
||||||
targetRevision: main
|
|
||||||
path: apps/argocd
|
|
||||||
ref: values
|
|
||||||
syncPolicy:
|
syncPolicy:
|
||||||
syncOptions:
|
syncOptions:
|
||||||
- CreateNamespace=true
|
- CreateNamespace=true
|
||||||
- ServerSideApply=true
|
- ServerSideApply=true
|
||||||
- ApplyOutOfSyncOnly=true
|
|
||||||
|
|
|
||||||
|
|
@ -2,28 +2,19 @@
|
||||||
|
|
||||||
This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster.
|
This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster.
|
||||||
|
|
||||||
Argo CD itself is installed from the community Helm chart (`argoproj/argo-helm`, chart `argo-cd`). This directory holds the chart values file plus a small kustomize wrapper for the ingress and sealed secrets that stay in git.
|
|
||||||
|
|
||||||
## Files
|
## Files
|
||||||
|
|
||||||
- `values.yaml` - Helm values for the `argo-cd` chart (image tag, OIDC/RBAC config, resource limits, ingress disabled).
|
- `argocd-install.yaml` - Auto-generated Argo CD installation manifest (DO NOT EDIT DIRECTLY)
|
||||||
- `kustomization.yaml` - Kustomize wrapper for the ingress + sealed secrets (does NOT install Argo CD itself).
|
- `kustomization.yaml` - Kustomize overlay that adds resource limits and other customizations
|
||||||
- `argocd-ingress.yaml` - Ingress configuration for the Argo CD UI (Traefik + cert-manager `letsencrypt-prod`).
|
- `argocd-ingress.yaml` - Ingress configuration for Argo CD UI
|
||||||
- `argocd-oidc-secret-sealed.yaml` - Sealed secret for Pocket ID OIDC integration.
|
- `argocd-oidc-secret-sealed.yaml` - Sealed secret for OIDC integration
|
||||||
- `repo-*.yaml` - Sealed secrets for Git repository access.
|
- `repo-*.yaml` - Sealed secrets for Git repository access
|
||||||
|
|
||||||
## How the install is wired
|
## Resource Limits
|
||||||
|
|
||||||
The root [`../app-argocd.yaml`](../app-argocd.yaml) is an Argo CD `Application` with two sources:
|
**IMPORTANT**: Resource limits were added after DEV-281 (resource exhaustion incident on 2026-07-12).
|
||||||
|
|
||||||
1. The public Helm chart at `https://argoproj.github.io/argo-helm`, chart `argo-cd`, `targetRevision` pinned in git.
|
All Argo CD components now have memory limits to prevent OOM incidents:
|
||||||
2. This repo (`ref: values`) providing the `values.yaml` used by source (1) AND the ingress/sealed secrets applied via `kustomize`.
|
|
||||||
|
|
||||||
Once bootstrapped, Argo CD manages its own install by syncing this Application.
|
|
||||||
|
|
||||||
## Resource limits
|
|
||||||
|
|
||||||
Memory limits were added after DEV-281 (resource exhaustion incident on 2026-07-12) and are now driven by `values.yaml`:
|
|
||||||
|
|
||||||
| Component | Memory Limit | Memory Request |
|
| Component | Memory Limit | Memory Request |
|
||||||
|-----------|--------------|----------------|
|
|-----------|--------------|----------------|
|
||||||
|
|
@ -34,51 +25,46 @@ Memory limits were added after DEV-281 (resource exhaustion incident on 2026-07-
|
||||||
| notifications-controller | 128Mi | 64Mi |
|
| notifications-controller | 128Mi | 64Mi |
|
||||||
| applicationset-controller | 256Mi | 128Mi |
|
| applicationset-controller | 256Mi | 128Mi |
|
||||||
|
|
||||||
|
These limits are based on observed usage patterns and provide headroom while preventing unlimited memory consumption.
|
||||||
|
|
||||||
## Deployment
|
## Deployment
|
||||||
|
|
||||||
### Steady state (managed by Argo CD)
|
### Option 1: Apply with kustomize (RECOMMENDED)
|
||||||
|
|
||||||
Once the cluster is bootstrapped, changes to this directory are picked up by the root `argocd` Application on the next sync. No manual `kubectl apply` is required.
|
|
||||||
|
|
||||||
### First-time / disaster-recovery bootstrap
|
|
||||||
|
|
||||||
Argo CD cannot install itself while it is gone. Bootstrap with helm, then hand ownership back:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
helm repo add argo https://argoproj.github.io/argo-helm
|
kubectl apply -k apps/argocd/
|
||||||
helm repo update
|
|
||||||
helm install argocd argo/argo-cd \
|
|
||||||
--version 10.4.0 \
|
|
||||||
--namespace argocd --create-namespace \
|
|
||||||
--values apps/argocd/values.yaml \
|
|
||||||
--wait --timeout 10m
|
|
||||||
|
|
||||||
kubectl apply -k apps/argocd/ # ingress + sealed secrets
|
|
||||||
kubectl apply -f apps/app-argocd.yaml # hand ownership back to GitOps
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This will apply the base manifests plus all patches defined in `kustomization.yaml`.
|
||||||
|
|
||||||
|
### Option 2: Direct apply (not recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl apply -f apps/argocd/argocd-install.yaml
|
||||||
|
kubectl apply -f apps/argocd/argocd-ingress.yaml
|
||||||
|
# etc.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note**: This skips the resource limit patches and is NOT recommended.
|
||||||
|
|
||||||
## Updating Argo CD
|
## Updating Argo CD
|
||||||
|
|
||||||
Bump the chart and the image tag in a single PR:
|
When updating to a new Argo CD version:
|
||||||
|
|
||||||
1. Refresh the local helm repo cache and check what's available:
|
1. Download the new install manifest:
|
||||||
```bash
|
```bash
|
||||||
helm repo update
|
curl -sSL https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml > argocd-install.yaml
|
||||||
helm search repo argo/argo-cd --versions | head
|
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Bump both fields together, keeping them in sync with the chart's `appVersion`:
|
2. Apply with kustomize (resource limits will be automatically applied):
|
||||||
- `apps/app-argocd.yaml` -> `spec.sources[0].targetRevision` (chart version, e.g. `10.4.0`)
|
|
||||||
- `apps/argocd/values.yaml` -> `global.image.tag` (app version, e.g. `v3.5.1`)
|
|
||||||
|
|
||||||
3. Optionally render locally to sanity-check the output before opening the PR:
|
|
||||||
```bash
|
```bash
|
||||||
helm template argocd argo/argo-cd \
|
kubectl apply -k apps/argocd/
|
||||||
--version <new-chart-version> \
|
|
||||||
-f apps/argocd/values.yaml -n argocd | less
|
|
||||||
```
|
```
|
||||||
|
|
||||||
4. Open the PR. After merge, Argo CD syncs itself onto the new version.
|
3. Verify resource limits are in place:
|
||||||
|
```bash
|
||||||
|
kubectl get statefulset,deployment -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'
|
||||||
|
```
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
|
|
@ -88,21 +74,22 @@ Bump the chart and the image tag in a single PR:
|
||||||
kubectl top pods -n argocd
|
kubectl top pods -n argocd
|
||||||
```
|
```
|
||||||
|
|
||||||
### Check applied resource limits
|
### Check if resource limits are applied
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kubectl get deployment,statefulset -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'
|
kubectl get deployment,statefulset -n argocd -o json | jq '.items[] | {name: .metadata.name, limits: .spec.template.spec.containers[0].resources.limits}'
|
||||||
```
|
```
|
||||||
|
|
||||||
### Rollback if a chart upgrade misbehaves
|
### Rollback if needed
|
||||||
|
|
||||||
|
If there are issues after applying resource limits:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
helm -n argocd history argocd
|
# Remove limits from a specific component
|
||||||
helm -n argocd rollback argocd <previous-revision>
|
kubectl patch deployment -n argocd argocd-server --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/resources"}]'
|
||||||
```
|
```
|
||||||
|
|
||||||
## History
|
## History
|
||||||
|
|
||||||
- **2026-08-23**: Switched to the community Helm chart, bumped to `v3.5.1` / chart `10.4.0` (DEV-519).
|
- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281)
|
||||||
- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281).
|
- **2026-07-11**: Initial deployment
|
||||||
- **2026-07-11**: Initial deployment (vendored `install.yaml`).
|
|
||||||
|
|
|
||||||
33364
apps/argocd/argocd-install.yaml
Normal file
33364
apps/argocd/argocd-install.yaml
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -4,7 +4,82 @@ kind: Kustomization
|
||||||
namespace: argocd
|
namespace: argocd
|
||||||
|
|
||||||
resources:
|
resources:
|
||||||
|
- argocd-install.yaml
|
||||||
- argocd-ingress.yaml
|
- argocd-ingress.yaml
|
||||||
- argocd-oidc-secret-sealed.yaml
|
- argocd-oidc-secret-sealed.yaml
|
||||||
- repo-basicstack-org-secret-sealed.yaml
|
- repo-basicstack-org-secret-sealed.yaml
|
||||||
- repo-stack-basicstack-de-secret-sealed.yaml
|
- repo-stack-basicstack-de-secret-sealed.yaml
|
||||||
|
|
||||||
|
patches:
|
||||||
|
# Add memory limits to prevent OOM incidents (DEV-281)
|
||||||
|
- target:
|
||||||
|
kind: StatefulSet
|
||||||
|
name: argocd-application-controller
|
||||||
|
patch: |-
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/containers/0/resources
|
||||||
|
value:
|
||||||
|
limits:
|
||||||
|
memory: 512Mi
|
||||||
|
requests:
|
||||||
|
memory: 256Mi
|
||||||
|
|
||||||
|
- target:
|
||||||
|
kind: Deployment
|
||||||
|
name: argocd-repo-server
|
||||||
|
patch: |-
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/containers/0/resources
|
||||||
|
value:
|
||||||
|
limits:
|
||||||
|
memory: 512Mi
|
||||||
|
requests:
|
||||||
|
memory: 256Mi
|
||||||
|
|
||||||
|
- target:
|
||||||
|
kind: Deployment
|
||||||
|
name: argocd-redis
|
||||||
|
patch: |-
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/containers/0/resources
|
||||||
|
value:
|
||||||
|
limits:
|
||||||
|
memory: 256Mi
|
||||||
|
requests:
|
||||||
|
memory: 128Mi
|
||||||
|
|
||||||
|
- target:
|
||||||
|
kind: Deployment
|
||||||
|
name: argocd-server
|
||||||
|
patch: |-
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/containers/0/resources
|
||||||
|
value:
|
||||||
|
limits:
|
||||||
|
memory: 256Mi
|
||||||
|
requests:
|
||||||
|
memory: 128Mi
|
||||||
|
|
||||||
|
- target:
|
||||||
|
kind: Deployment
|
||||||
|
name: argocd-notifications-controller
|
||||||
|
patch: |-
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/containers/0/resources
|
||||||
|
value:
|
||||||
|
limits:
|
||||||
|
memory: 128Mi
|
||||||
|
requests:
|
||||||
|
memory: 64Mi
|
||||||
|
|
||||||
|
- target:
|
||||||
|
kind: Deployment
|
||||||
|
name: argocd-applicationset-controller
|
||||||
|
patch: |-
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/containers/0/resources
|
||||||
|
value:
|
||||||
|
limits:
|
||||||
|
memory: 256Mi
|
||||||
|
requests:
|
||||||
|
memory: 128Mi
|
||||||
|
|
|
||||||
|
|
@ -1,80 +0,0 @@
|
||||||
global:
|
|
||||||
image:
|
|
||||||
tag: v3.5.1
|
|
||||||
|
|
||||||
configs:
|
|
||||||
cm:
|
|
||||||
url: https://argo.basicstack.de
|
|
||||||
application.instanceLabelKey: argocd.argoproj.io/instance
|
|
||||||
resource.exclusions: |
|
|
||||||
- apiGroups: [cilium.io]
|
|
||||||
kinds: [CiliumIdentity, CiliumEndpoint, CiliumEndpointSlice]
|
|
||||||
- apiGroups: [kyverno.io, reports.kyverno.io, wgpolicyk8s.io]
|
|
||||||
kinds: [PolicyReport, ClusterPolicyReport, EphemeralReport,
|
|
||||||
ClusterEphemeralReport, AdmissionReport, ClusterAdmissionReport,
|
|
||||||
BackgroundScanReport, ClusterBackgroundScanReport, UpdateRequest]
|
|
||||||
oidc.config: |
|
|
||||||
name: Pocket ID
|
|
||||||
issuer: https://auth.basicstack.de
|
|
||||||
clientID: $oidc.pocketid.clientId
|
|
||||||
clientSecret: $oidc.pocketid.clientSecret
|
|
||||||
requestedScopes: [openid, profile, email, groups]
|
|
||||||
requestedIDTokenClaims:
|
|
||||||
groups: {essential: true}
|
|
||||||
|
|
||||||
rbac:
|
|
||||||
policy.default: role:readonly
|
|
||||||
policy.csv: |
|
|
||||||
g, argo_admins, role:admin
|
|
||||||
p, role:admin, applications, *, */*, allow
|
|
||||||
p, role:admin, clusters, *, *, allow
|
|
||||||
p, role:admin, repositories, *, *, allow
|
|
||||||
p, role:admin, projects, *, *, allow
|
|
||||||
p, role:admin, accounts, *, *, allow
|
|
||||||
p, role:admin, gpgkeys, *, *, allow
|
|
||||||
p, role:admin, certificates, *, *, allow
|
|
||||||
p, role:admin, exec, *, *, allow
|
|
||||||
|
|
||||||
controller:
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
memory: 256Mi
|
|
||||||
limits:
|
|
||||||
memory: 512Mi
|
|
||||||
|
|
||||||
repoServer:
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
memory: 256Mi
|
|
||||||
limits:
|
|
||||||
memory: 512Mi
|
|
||||||
|
|
||||||
server:
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
memory: 128Mi
|
|
||||||
limits:
|
|
||||||
memory: 256Mi
|
|
||||||
ingress:
|
|
||||||
enabled: false
|
|
||||||
|
|
||||||
redis:
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
memory: 128Mi
|
|
||||||
limits:
|
|
||||||
memory: 256Mi
|
|
||||||
|
|
||||||
notifications:
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
memory: 64Mi
|
|
||||||
limits:
|
|
||||||
memory: 128Mi
|
|
||||||
|
|
||||||
applicationSet:
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
memory: 128Mi
|
|
||||||
limits:
|
|
||||||
memory: 256Mi
|
|
||||||
Loading…
Add table
Reference in a new issue