stack.basicstack.de/docs/argocd.md

Argo CD - GitOps Platform

Overview

Argo CD is the GitOps deployment platform for the basicstack.de Kubernetes cluster. It continuously monitors Git repositories and automatically syncs application state to the cluster.

• UI URL: https://argo.basicstack.de
• Namespace: argocd
• Authentication: Pocket ID SSO (OIDC)
• Managed Repositories:
  • stack.basicstack.de - Infrastructure and platform applications
  • basicstack.org - Frontend web application

Authentication & Access

SSO Login Flow

1. Navigate to https://argo.basicstack.de
2. Click "LOGIN VIA POCKET ID"
3. You will be redirected to https://auth.basicstack.de (Pocket ID)
4. Enter your credentials and complete 2FA if prompted
5. After successful authentication, you'll be redirected back to Argo CD

Authorization

Access is controlled via Pocket ID groups mapped to Argo CD roles:

• argo_admins group → role:admin (full access to all resources)
• Default users → role:readonly (read-only access)

To grant admin access to a user:

1. Log in to Pocket ID admin UI at https://auth.basicstack.de
2. Navigate to the user's profile
3. Add them to the argo_admins group

Emergency Access

IMPORTANT: Local admin account is disabled by policy. If SSO is down:

1. Do NOT reset to recovery admin - this destroys the working OIDC configuration
2. Restore SSO service first (see Pocket ID documentation)
3. If SSO cannot be restored, escalate to CTO for manual intervention

Recovery from SSO lockout should follow this priority:

1. Fix Pocket ID service (check pods, ingress, database)
2. Verify OIDC secret is valid in argocd namespace
3. Only as absolute last resort: temporary re-enable local admin with explicit approval

Adding New Applications

Method 1: Via Git (Recommended)

Applications are defined as YAML manifests in the apps/ directory of the stack.basicstack.de repository.

1. Create a new application manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
    targetRevision: main
    path: apps/my-app  # Directory containing K8s manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: my-namespace
  syncPolicy:
    automated:
      prune: false  # Manual approval for deletions (recommended)
      selfHeal: true  # Auto-sync on Git changes
    syncOptions:
    - CreateNamespace=true

2. Commit the file to stack.basicstack.de repo:

git add apps/app-my-app.yaml
git commit -m "Add Argo CD application for my-app"
git push origin main

3. Argo CD will automatically detect and sync the new application (via the stack-basicstack-de self-managing application)

Method 2: Via UI

1. Log in to https://argo.basicstack.de
2. Click "+ NEW APP"
3. Fill in application details:
   • Application Name: Unique identifier
   • Project: default
   • Sync Policy: Choose automated or manual
   • Repository URL: Select from existing or add new
   • Path: Directory containing manifests
   • Cluster: https://kubernetes.default.svc (in-cluster)
   • Namespace: Target namespace
4. Click "CREATE"

Note: Applications created via UI are not persisted in Git. Use Method 1 for production applications.

Repository Credential Rotation

Both repositories use SSH key authentication with Forgejo running in-cluster.

Current Repositories

• git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
• git@forgejo.forgejo.svc.cluster.local:basicstack/basicstack.org.git

Rotation Procedure

1. Generate new SSH key pair on a secure workstation:

ssh-keygen -t ed25519 -C "argocd@basicstack.de" -f argocd-forgejo-key

2. Add the public key to Forgejo:

   • Log in to https://forgejo.basicstack.de
   • Navigate to Organization Settings → Deploy Keys
   • Add the new public key with read-only access
   • Give it a descriptive name (e.g., "ArgoCD Deploy Key - 2026-07")

3. Update the sealed secrets in the repository:

# Seal the new credentials
kubectl create secret generic repo-stack-basicstack-de \
  --namespace=argocd \
  --from-literal=type=git \
  --from-file=sshPrivateKey=argocd-forgejo-key \
  --dry-run=client -o yaml | \
  kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \
  --format=yaml > apps/argocd/repo-stack-basicstack-de-secret-sealed.yaml

kubectl create secret generic repo-basicstack-org \
  --namespace=argocd \
  --from-literal=type=git \
  --from-file=sshPrivateKey=argocd-forgejo-key \
  --dry-run=client -o yaml | \
  kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \
  --format=yaml > apps/argocd/repo-basicstack-org-secret-sealed.yaml

4. Commit and push the updated sealed secrets:

git add apps/argocd/repo-*-secret-sealed.yaml
git commit -m "Rotate Argo CD repository credentials"
git push origin main

5. Wait for Argo CD to sync (or manually sync the argocd application)

6. Verify repository connections in the UI:

   • Settings → Repositories
   • Both repos should show "Successful" connection status

7. Remove the old public key from Forgejo

8. Securely delete the private key file:

shred -u argocd-forgejo-key argocd-forgejo-key.pub

Self-Management

Argo CD manages itself via the argocd Application, which monitors apps/argocd/ in the stack.basicstack.de repository.

Key Self-Management Settings

spec:
  syncPolicy:
    automated:
      prune: false  # Prevent accidental deletion of Argo CD resources
      selfHeal: true  # Auto-apply Git changes to Argo CD config

Updating Argo CD Configuration

Changes to Argo CD itself (RBAC, OIDC, ingress, etc.) are made via Git:

1. Edit files in apps/argocd/ directory
2. Commit and push to main
3. Argo CD will self-apply the changes

Critical files:

• argocd-install.yaml - Argo CD installation manifests (CRDs, deployments)
• argocd-ingress.yaml - TLS ingress configuration
• argocd-rbac-cm.yaml - RBAC policies and group mappings (if customized)
• argocd-oidc-secret-sealed.yaml - Pocket ID OAuth credentials (sealed)
• repo-*-secret-sealed.yaml - Repository SSH keys (sealed)

Upgrading Argo CD

To upgrade to a new version:

1. Download the new installation manifest:

curl -L https://raw.githubusercontent.com/argoproj/argo-cd/v2.x.x/manifests/install.yaml -o apps/argocd/argocd-install.yaml

2. Review the diff carefully, especially:

   • Breaking changes in the release notes
   • Custom patches that may need reapplication
   • ConfigMap/Secret schema changes

3. Commit the updated manifest:

git add apps/argocd/argocd-install.yaml
git commit -m "Upgrade Argo CD to v2.x.x"
git push origin main

4. Monitor the upgrade in the UI or via CLI:

kubectl get pods -n argocd -w

5. Verify all applications remain healthy after the upgrade

Monitoring & Maintenance

Health Checks

Daily Checks (automated via Paperclip or manual):

# Check Argo CD pods are running
kubectl get pods -n argocd

# Check application sync status
kubectl get applications -n argocd

# Check for out-of-sync or unhealthy apps
kubectl get applications -n argocd -o json | \
  jq -r '.items[] | select(.status.sync.status != "Synced" or .status.health.status != "Healthy") | .metadata.name'

UI Health Dashboard:

• Navigate to https://argo.basicstack.de
• Applications with issues are highlighted in red/yellow
• Click into an application to see detailed sync/health status

Common Issues

Application Stuck in "Syncing"

# Check sync operation status
kubectl describe application <app-name> -n argocd

# Cancel stuck sync
kubectl patch application <app-name> -n argocd --type json \
  -p='[{"op": "remove", "path": "/operation"}]'

Repository Connection Failed

# Check repository secret exists
kubectl get secret repo-<repo-name> -n argocd

# View Argo CD repo server logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server

Out of Sync Applications

• Manual intervention: Application has syncPolicy: manual or automated sync is disabled
• Failed sync: Check sync logs in the UI or via kubectl describe application
• Resource drift: External changes made directly to cluster (not via Git)
  • Solution: Either revert cluster changes or commit them to Git

Log Access

# Argo CD server logs (API/UI)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server

# Application controller logs (sync operations)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller

# Repo server logs (Git operations)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server

Backup & Disaster Recovery

Argo CD state is derived from Git, but the following should be backed up:

Critical Data:

1. Sealed secrets source keys - sealed-secrets namespace
2. OIDC credentials - argocd-oidc-secret (sealed in Git)
3. Repository SSH keys - repo-* secrets (sealed in Git)

Recovery Procedure (cluster rebuild):

1. Restore k3s cluster and sealed-secrets controller
2. Apply Argo CD installation:

kubectl create namespace argocd
kubectl apply -f apps/argocd/argocd-install.yaml

3. Wait for Argo CD pods to be ready:

kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

4. Apply sealed secrets (will be unsealed by controller):

kubectl apply -f apps/argocd/argocd-oidc-secret-sealed.yaml
kubectl apply -f apps/argocd/repo-stack-basicstack-de-secret-sealed.yaml
kubectl apply -f apps/argocd/repo-basicstack-org-secret-sealed.yaml

5. Apply Argo CD application manifests:

kubectl apply -f apps/app-argocd.yaml
kubectl apply -f apps/app-stack-basicstack-de.yaml
kubectl apply -f apps/app-basicstack-org.yaml

6. Verify sync in UI: https://argo.basicstack.de

Architecture

Application Hierarchy

argocd (self-managing)
  └── apps/argocd/
      ├── argocd-install.yaml       # CRDs, deployments, services
      ├── argocd-ingress.yaml       # TLS ingress
      ├── argocd-oidc-secret-sealed.yaml  # Pocket ID credentials
      └── repo-*-secret-sealed.yaml       # Git credentials

stack-basicstack-de (infrastructure)
  └── apps/ (recursive, excludes apps/argocd/)
      ├── app-basicstack-org.yaml   # Declares basicstack-org app
      ├── app-stack-basicstack-de.yaml  # Declares self
      ├── bookstack/
      ├── directus/
      ├── forgejo/
      ├── paperclip/
      └── ... (other platform services)

basicstack-org (frontend application)
  └── k8s/
      ├── deployment.yaml
      ├── service.yaml
      └── ingress.yaml

Sync Policies

Application	Prune	Self-Heal	Rationale
argocd	false	true	Manual approval for deletions prevents accidental Argo CD breakage
stack-basicstack-de	false	true	Manual approval for platform service deletions
basicstack-org	true	true	Fully automated for stateless frontend

CLI Usage

Install the Argo CD CLI:

# Linux/WSL
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
chmod +x argocd
sudo mv argocd /usr/local/bin/

Login via SSO:

argocd login argo.basicstack.de --sso

Common commands:

# List applications
argocd app list

# Get application details
argocd app get <app-name>

# Sync application
argocd app sync <app-name>

# Watch sync progress
argocd app wait <app-name>

# View application logs
argocd app logs <app-name>

# Set application sync policy
argocd app set <app-name> --sync-policy automated --auto-prune --self-heal

Troubleshooting Checklist

When Argo CD has issues, work through this checklist:

1. Argo CD Pods Not Running

kubectl get pods -n argocd
kubectl describe pod <failing-pod> -n argocd
kubectl logs <failing-pod> -n argocd

2. SSO Login Fails

• Pocket ID is accessible: curl -I https://auth.basicstack.de
• OIDC secret exists: kubectl get secret argocd-oidc-secret -n argocd
• OIDC config in argocd-cm: kubectl get cm argocd-cm -n argocd -o yaml | grep -A 10 oidc.config
• Check Argo CD server logs for OIDC errors

3. Repository Connection Failed

• Forgejo is accessible: kubectl get pods -n forgejo
• Repository secret exists: kubectl get secret repo-<name> -n argocd
• SSH key is valid in Forgejo deploy keys
• Check repo-server logs: kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server

4. Application Sync Fails

• Check application status: kubectl describe application <name> -n argocd
• Review manifests in Git for syntax errors
• Check destination namespace exists
• Verify RBAC permissions if deploying to specific namespaces
• Review sync operation in UI for detailed error messages

5. Ingress/TLS Issues

• Ingress object exists: kubectl get ingress argocd-server -n argocd
• TLS certificate is valid: kubectl get certificate argocd-server-tls -n argocd
• Traefik is routing correctly: kubectl logs -n kube-system -l app.kubernetes.io/name=traefik
• DNS resolves: dig argo.basicstack.de

Security Considerations

Sealed Secrets

All sensitive data (OIDC credentials, SSH keys) are encrypted using sealed-secrets before being committed to Git. The sealing key is stored in the sealed-secrets namespace and should be backed up securely.

Never commit unsealed Secret manifests to Git.

RBAC

The default policy is role:readonly - all users have read-only access unless explicitly granted admin via the argo_admins group.

Admin permissions include:

• Syncing applications
• Creating/deleting applications
• Managing repositories and clusters
• Modifying project settings
• Executing into containers

Network Security

Argo CD uses in-cluster Forgejo for Git operations:

• Repository URLs use cluster DNS: git@forgejo.forgejo.svc.cluster.local
• No external Git credentials or tokens are exposed
• SSH keys are scoped to specific repositories as deploy keys (read-only)

References

• Argo CD Documentation
• Argo CD Best Practices
• Pocket ID (SSO Provider)
• Sealed Secrets