Compare commits

..

No commits in common. "4f979d657e715603eb13fbf18f61cfcd2d999efb" and "5e02c64dc455543aada05d47d851110766ec20cb" have entirely different histories.

10 changed files with 6 additions and 33684 deletions

View file

@ -1,95 +0,0 @@
# Argo CD Deployment
This directory contains the Argo CD deployment configuration for the basicstack.de k3s cluster.
## Files
- `argocd-install.yaml` - Auto-generated Argo CD installation manifest (DO NOT EDIT DIRECTLY)
- `kustomization.yaml` - Kustomize overlay that adds resource limits and other customizations
- `argocd-ingress.yaml` - Ingress configuration for Argo CD UI
- `argocd-oidc-secret-sealed.yaml` - Sealed secret for OIDC integration
- `repo-*.yaml` - Sealed secrets for Git repository access
## Resource Limits
**IMPORTANT**: Resource limits were added after DEV-281 (resource exhaustion incident on 2026-07-12).
All Argo CD components now have memory limits to prevent OOM incidents:
| Component | Memory Limit | Memory Request |
|-----------|--------------|----------------|
| application-controller | 512Mi | 256Mi |
| repo-server | 512Mi | 256Mi |
| redis | 256Mi | 128Mi |
| server | 256Mi | 128Mi |
| notifications-controller | 128Mi | 64Mi |
| applicationset-controller | 256Mi | 128Mi |
These limits are based on observed usage patterns and provide headroom while preventing unlimited memory consumption.
## Deployment
### Option 1: Apply with kustomize (RECOMMENDED)
```bash
kubectl apply -k apps/argocd/
```
This will apply the base manifests plus all patches defined in `kustomization.yaml`.
### Option 2: Direct apply (not recommended)
```bash
kubectl apply -f apps/argocd/argocd-install.yaml
kubectl apply -f apps/argocd/argocd-ingress.yaml
# etc.
```
**Note**: This skips the resource limit patches and is NOT recommended.
## Updating Argo CD
When updating to a new Argo CD version:
1. Download the new install manifest:
```bash
curl -sSL https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml > argocd-install.yaml
```
2. Apply with kustomize (resource limits will be automatically applied):
```bash
kubectl apply -k apps/argocd/
```
3. Verify resource limits are in place:
```bash
kubectl get statefulset,deployment -n argocd -o custom-columns='NAME:.metadata.name,MEMORY_LIMIT:.spec.template.spec.containers[0].resources.limits.memory'
```
## Troubleshooting
### Check resource usage
```bash
kubectl top pods -n argocd
```
### Check if resource limits are applied
```bash
kubectl get deployment,statefulset -n argocd -o json | jq '.items[] | {name: .metadata.name, limits: .spec.template.spec.containers[0].resources.limits}'
```
### Rollback if needed
If there are issues after applying resource limits:
```bash
# Remove limits from a specific component
kubectl patch deployment -n argocd argocd-server --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/resources"}]'
```
## History
- **2026-07-12**: Added resource limits via kustomization to prevent OOM incidents (DEV-281)
- **2026-07-11**: Initial deployment

View file

@ -1,20 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: basicstack-org
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/basicstack.org.git
targetRevision: main
path: k8s
destination:
server: https://kubernetes.default.svc
namespace: basicstack-web
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

View file

@ -1,23 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: stack-basicstack-de
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps
directory:
recurse: true
exclude: |
**/argocd/**
destination:
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: false # Manual approval for deletions
selfHeal: true
syncOptions:
- CreateNamespace=true

View file

@ -1,27 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd-server
namespace: argocd
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
traefik.ingress.kubernetes.io/router.entrypoints: websecure
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
ingressClassName: traefik
tls:
- hosts:
- argo.basicstack.de
secretName: argocd-server-tls
rules:
- host: argo.basicstack.de
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: argocd-server
port:
number: 80

File diff suppressed because it is too large Load diff

View file

@ -1,16 +0,0 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: argocd-oidc-secret
namespace: paperclip
spec:
encryptedData:
oidc.pocketid.clientId: AgCji/I1a9Xm0/cMakoFJj9IygsvzJhl/NclwDQfB1uz+0PlMr0PWPlSwZyyAZE3SilkzWMLGEfHIT78vlUZkzF9Y937xtshw79f0EIDqGSazvRbSg3Sjcl79pbYJBq8JXOTsDe38mV6RjJ5bTZ47l14YZfIx6crtofy8zjF4I8Wdgr/Zjr2HBZm7BTFrdciKEf5OKYrOOV2Ypm3LM1W248sT/upw/CSZuLwNjJp7GVvHQF3oajWL4rvOZBeeLsyjRCdeUaViVFRQ2qDHgTEFZzryxSsagUu6QGE93HC3DJLFwveVhKt/7xcslEfxR18UL1EB0H9NJo5vjb0TH5ZeuszKbTOoFJDMFcP6q2cLu8R3vWM6W2lPHJZFs2Pvc/0FNOpNDJqUdjQYbKq8InRvXqCOzhHDcsvra0FhRgY+2ww7MLZoJ+vlljtIsvL33pUawgcCauL2GXBYHImvbm2f7Max3dsbHlA0HJZVrn3Iwq7u1mpHvD+Zch+8Of6KjZAbvy+5FQF3SqblQJV2UEM0vy9Vg3fT0V5fmXNaGtSEFmI6QvFTSLA+1UjjPHvMNpk+PL2ahRv5O3uwRoUv9dFtE7DsY2raLrc/RxxeZiqLAwUT7afP1pPaBkAUXqjjPa/VpTRszJ3y1DWxqanOkDu1KZdBHRebUvtj2/IxcpexunlGgTwAsGQ8/5uMkb98r7I9r3pfPDvLwhY9GlVvizl5P/5WzwzbAPo7sRaKetNfqRzNTXzXFc=
oidc.pocketid.clientSecret: AgBXdRcZuIY5IprCwxBeJ55Gxv1xSE7qAppNBXAqHwImUI2LLRoDciZeGz4w/a0Fn6jW5hBTUI9QQiG7BuznZzP3UtQfNE4YyDvCO539UKjCNcvDAMnoBo4KJ1vdJCLYwCxx5FCv9K95UJmbmQsKQB2+xnFWhEpv+aharJn+9rNh5Z6lz0kKLKclV7Bmh88WbfEJLPjowsEpDqB9Ti3HMwIOjdUbdKfL4wctHgI42yul449OJJqq8SeMQDvGX3nX7oo2wHad/AFXsAf6tvx7wthMhDjdo8RwP6hqO8s6CDcMf79RA2Wx9Y/TZavTpXQlX0M4h9YAtixz3etZIVw/y4n8R7xTZWcNKoyL/xt2GbcrV/UAexEfVEJMaoxBzdz4N6Tk3gmmsLlku7zuCylXRxaNwxuhlHLhdv5TxRo93Ut/RVdZKO2V8euXcsJ7i/5w96fmZFnJcfo+HRIh+pyzg3IZ0iPq/ajstTRpGAd0mof35NNEASSIWWckoZjXljO1IiMtBpYpQA8dCOwjSr7LDR6cEedxj2hpXGkjRzJWbQeH1IFpnM13QVClv3eK6rrpZMGfQrLT+CP5HlWFCmjE73b9jODqGcI0cHSk4XHA0ZFH5b1aT51Icv5qQIzUEJvloJOgIkSp8/yI8Xe7zeZ6uLa7UtZ/fq9/SmvhLDpp7l0R/ScHKMUfQDAgaipCJZjR814BCNBKbJk7CfSiHnfePJO90717VynnsL30oufj7Tk/hw==
template:
metadata:
creationTimestamp: null
name: argocd-oidc-secret
namespace: paperclip

View file

@ -1,85 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
resources:
- argocd-install.yaml
- argocd-ingress.yaml
- argocd-oidc-secret-sealed.yaml
- repo-basicstack-org-secret-sealed.yaml
- repo-stack-basicstack-de-secret-sealed.yaml
patches:
# Add memory limits to prevent OOM incidents (DEV-281)
- target:
kind: StatefulSet
name: argocd-application-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 512Mi
requests:
memory: 256Mi
- target:
kind: Deployment
name: argocd-repo-server
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 512Mi
requests:
memory: 256Mi
- target:
kind: Deployment
name: argocd-redis
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 256Mi
requests:
memory: 128Mi
- target:
kind: Deployment
name: argocd-server
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 256Mi
requests:
memory: 128Mi
- target:
kind: Deployment
name: argocd-notifications-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 128Mi
requests:
memory: 64Mi
- target:
kind: Deployment
name: argocd-applicationset-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value:
limits:
memory: 256Mi
requests:
memory: 128Mi

View file

@ -1,18 +0,0 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: repo-basicstack-org
namespace: paperclip
spec:
encryptedData:
sshPrivateKey: AgAWGlmIoVGXXMN94YxrAOrFfNxbN3b/BHK4yevOlukxGWbB0qkuynt6E73soqp78q5rWDnU6GrOvSScEqjpomUMhV/kPeY7jiX81Ef5VjzUTVYnf1Ioo/xwFiIZQ859WepnnVmZFK1p3kPaMzk6b7jfynd8yjzbNJZpP43Ss5KK3ENsvgysc+P/r/qBQsi8B4oosuUucwrnsdb8j3GPcDzNT5DZpi4Pb/b4EiPryTr/L7z9khh0VZW1+FbQS8fcYbqAJxq7Mm0HjgkEZmledmQEkqBLnvW2rg0UqvqjMC+mkTXk29jWS3uC8arPk1BoBcqA3Obs8nV3Y/opcPyGhDve9mUumBJpsjQSenF9EguWgoXalaBEl8vsD4Ymllorc55b8mMxXVW9LTvNDv8d+CdFBaT4fLyN3FoMNZeWkM9WahZzvM+pNPUVFKqe7+JiagBAuGNY6d3WbXlZElox8VqMxO19hjzVdDW6abmRN4BDQ+U8k50ISfwnAyB7ssnNkx0aRdb6/TEs0YQeweYPdXDH1iSKzn7CBDXWj8djgRdAbDYoll2CFuO6fj0EHru/9TWrBp5RInqE7UTkhaFpQgDlr9Grbva6pFMK/yFYiVC4dhNj8ZXls/miyqQqLguzKMSnNLiXymB7waMRwTil7SA7JaqqGq6/pO+nYiQYrwLKZGvyWVAr/rO89De5SImxOeVfMY3Be4tobDD/U+mEbVPHPtijkB5c4O5MJgLwnVD66n+bD2fCgorXENcVRxcPBeHE9fTe5cJ3+OJvlJZRJFRin7GBSA5EEaWjARyQAbngcFe2KGux7ShaJEMVC8vrN7HV0XqBIUFNCpQNMK++xBlhxg/BmX0A4KaSs6LqkqeyiGE7IYccZCxkgb51cKHzVRxRhLOEV4yUJ11EFHdSgjC4lCZ2f0DVnSjIb0j9W3CX03MrahtRYIszEfW9g1e6bx5KB83z6apQu1lvPAsDqEvEu95CbYWd0xxF43tUAUVyVJG0D01r3xVPujwDXmvwqJrzlKvwH14j3g9TvPJS2a6nyLxZq9NNrrWX3qRbpOvi6bV0IMb1/KSyKWYsp0nu+RDrysuz3SHepIG9B2itw81xDWLIeHwRik9dvQns69yLj80VwXUuwpbDG/d+XPwiSBBQTGrGbjc5caXpoCFqpakJvk0yi5+iCP5iTv096SsJu5XjLkI3ETEdjxjSILq8CE6PmYZUCHu24PHA3zHhDAPCWQhdjf7lyXEM9yk=
url: AgCJFEYWCE/HlGzqksnY2LrDUwITDvmH3twoMSuDUpysfw3L4nDGq4v//fitpRQlQKp+7SZYFpPmgevWY0j/HpKsB0TLvT25WgYZCyLXOk7lW2LXkrQnVDpOrGI8YCpodtqLnxk1abQ1eZWqL+etEQl4lhUQ2uBiydQG4+dazCrj179c9j2X04yLrt3iJ9dnufpcl6oaHG3cho+6gvDkFbsMkYKWmubZ8kgsXmditmK37q7Dqj7Ct5xhdl8YH1kdLrBoUjiCfAx81BtAixCoHukHxDnVQ3oKDDkwxhk6/quyVqdrthslyb0fOmoyZOz1KdMHD2TyFhAv51XFoBZAE4rNemwHSTmrbbh56QlMtL1pl6rPBarZeucuEGKgq5UR4ve7qc+zw6QG8uHHyW1Y41aYF/Z4nzCSgoPG9luiDiTMrD/VxkWk+7ry6epU9mM3d2cVJa/68Kvgx6TVR4ifMpKJTA7UZETC5vntuxz0M960wbYajMyC++ojyBZVBJlHp4K7xluLBNN8BVJcYjQMkoHBWDmeoJAmTZ70c6RZS5vcO+eAMyNjTrSsawMDaPTejLkHk2l0YEb6rJKMZKotaOtYTZjCrPwKpYFrZovxss3eujCwRunL1Jub7PDOY68kIRBxh+haRjEIdPygHuM/A4ZRgsJSEZo9Wlk806DU1N8f/1OEAb252Fi9EbWbauVDOLK8dXxEN4GuhF8BUWy2nyzRFPBUX4vFy2cYjKMIBjRHKvl1yEzA6g1vsZNGs6iMPEZ9Iq3VXem2
template:
metadata:
creationTimestamp: null
labels:
argocd.argoproj.io/secret-type: repository
name: repo-basicstack-org
namespace: paperclip

View file

@ -1,18 +0,0 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: repo-stack-basicstack-de
namespace: paperclip
spec:
encryptedData:
sshPrivateKey: AgAmDGi8MznJXDao2L33/HQGrl3Aaj0asrnRi22HgDN4re7SbveNs3JKi0DGPEOW/GyI40mvrnxY1S8Zor0XntnR/MroW5emLp6aPp+X8qBCbPaYlQeHn++eUY6dRDfUk1vigSnmAvo02ONTpp0oK8D3F2853ZA+AtoTaVtWvVAuHZjo92X7fhYCYcp69jbBq4Cj2ni7St7DV7nV0B86VZIIcprWyWVTxY1Jq66t0aO25jrDPjvswHhkASWd+7B/KmWZelzVehZ36hkbA0Cm2BtwBK0dbPbNEtighScQIbPfwQCIQvVa4WHdUAUPi2iyV84trgd5pWteNYtGG/C9sYrO6y5/xuH1Li7p43UQwJWuNmU4IPcuDYxx+BIsbUP3rqSDvJuWp2uEaPfiuAECN6toVRrEFTt+pU5chu1ar98PDftnDlj+rGlla8wwQVQuZA0zjTUihwHFeXxD4C0tb/EdSSyEtUQVWKDvTfNFldWk/0pMecVLlKqEUIYGEKABVs1flHHdkOklpz4/qZZcs7A6Vo/nGVq69BqDvqTS0g50sMRIcevtXXVwIR4qfm72WDSxrs6zOwVMsVDNFVMcFX7z0+MFVWevHyH3uGy9s8qBn2tOEpzo+gzstYZkDQDBOoIKqCS8tM7ySp+OZSpfFR4g2P0PJ2COMzkbhV2Vl33JvxxDY74aWwq1/Xvnq4BmPV7XqmnAoXGXs+IszHvfg2M9s6f3wdIhr+hDsA4JxXsx9sZLNAgxjDubBPvgfkPiXLDr6S0Z7ZQuLQuSAoJjvy9UAy65Ck0UhDptrtaK0tQdlARVru5aVbfEY/4kTJOfgn3iUXJWvT/jBXH6OKAU3fapz3NgL4gTrhmuR+jjrtf+sHA/RO31BEC/ILc6qmj0FlUhI6QSaxPNXyn3G2pNZhiri4AvbLJsrp9IV3SixG9KQIV8zLIQ6ebs5s3QQvZKcltreSiF1+S0TdbLYMUw65rYk70aJp9Iar64Q7sWrUZeVVJySPMnO3P/8Nz2Bsh0KCOagmtN5C10OliaMR7PRZlYQnt0nmIPrv6xeOWCwFcFqlCVGF4XouUhO7ufmeAcPGsIllBkTW7kHh3ycojyY3ZsDBbQUkCvksJUNDyd5Sg+h6HBtOVNfhA/sVV7nS4QboEmNghNYPMxgyET1CE4MT6mtdY+XcyQWtK3bWDLsWTrBLtpp8p4gNd4XcJOL/TSRriBvBy0TlF9ouW3SXnB6q0=
url: AgAtK+bVTonn9QHBYZ+39N5vx/6TNFcr6zvk0yQmjn0QoIlYHwwMYaP+G9IEXchhMDT715t8fouB6lKM45D4KyZSKJQmdOXS+7/DuXQoKeK2qKb9GCj/m19VKAN0Yq/e0/l2gMvCCGdo3/uXNG+9mYyTWi/wRSLPjlfATtinWcoUAbATra3CAKH30b/31brdWI76eM2x2XyDa5G7kQ9t0PoU0uZVveocGaqRV+DRIj2dQjOBCShuZaAgwLC9ezXpsL0p/A7d8wOfbILzxA2Dzj1GbdNsVwondeLP0Iyyh1YUBvhbnVjpJLCHZJUgslyYu8w3GzB/xOYneFZbZ3pHBIPSKde5apWQmu4Xsq9Ivbcy/NQvSFLDWGmkE2R3C82CPsWn0xTDBK2ibtyPJ6sMWTXyZ+Ba4RiNj+LT+WcNnGcFXyUZAXMWFD965t30Vy2+rcaUAnVh5tzv9gamXAuCj5jYY/O+jh3dXidCxoLXaFuKocNyBZMnR5d9/XtXYUlC4YDmQCu4A2UtQsKEMPrbPENNrsUt96AqozcZpYop55yOI9LTsZDGc4G1kGDKBoXfsVyRn7SLORIaV4D1xx2koOi0NPswz0gHtoEFBVZr9F/o16AHnyRAxu0iNLV0kyJrUZCEWIDYkgMbfbnoMkn3M7qdEl3Qn/R+FuRahXE1SNA5Dyoos9YfHA3+97LJQIabRUHKW15A8FVZdt28kQX6RYzwFLKQfj/AJ/gPK2PELiPbh7LiC7H7ULJAaNcc1iOJSTXgHnP7QyixwXA9FX8=
template:
metadata:
creationTimestamp: null
labels:
argocd.argoproj.io/secret-type: repository
name: repo-stack-basicstack-de
namespace: paperclip

View file

@ -15,17 +15,6 @@ spec:
template:
spec:
restartPolicy: OnFailure
# Run on same node as OpenCloud to allow volume attachment
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- opencloud
topologyKey: kubernetes.io/hostname
containers:
- name: backup
image: rclone/rclone:latest
@ -38,21 +27,20 @@ spec:
echo "Starting OpenCloud backup at $DATE"
# Configure rclone for Hetzner S3
mkdir -p /root/.config/rclone
cat > /root/.config/rclone/rclone.conf <<EOC
[hetzner-s3]
type = s3
provider = Other
access_key_id = ${S3_ACCESS_KEY}
secret_access_key = ${S3_SECRET_KEY}
endpoint = https://hel1.your-objectstorage.com
region = hel1
endpoint = https://fsn1.your-objectstorage.com
region = fsn1
acl = private
EOC
# Backup OpenCloud data directory
echo "Backing up data directory..."
rclone sync /data hetzner-s3:basicstack-backup/opencloud/data-${DATE} \
rclone sync /data hetzner-s3:opencloud-backup/data-${DATE} \
--progress \
--transfers 8 \
--checkers 8 \
@ -60,12 +48,12 @@ spec:
# Backup configuration
echo "Backing up configuration..."
rclone copy /config hetzner-s3:basicstack-backup/opencloud/config-${DATE} \
rclone copy /config hetzner-s3:opencloud-backup/config-${DATE} \
--progress
# Keep only last 7 days of backups
echo "Cleaning old backups (keeping last 7 days)..."
rclone delete hetzner-s3:basicstack-backup/opencloud \
rclone delete hetzner-s3:opencloud-backup \
--min-age 7d \
--rmdirs