From 18bdb78429a659715f7ce639a6a0a965a297fb1e Mon Sep 17 00:00:00 2001 From: CTO Agent Date: Sun, 12 Jul 2026 10:04:51 +0000 Subject: [PATCH] Add comprehensive Argo CD documentation Created detailed documentation covering: - Authentication and SSO login flow - Adding new applications (Git and UI methods) - Repository credential rotation procedure - Emergency recovery procedures (with critical safety warnings) - Self-management architecture - Monitoring, maintenance, and troubleshooting - Backup and disaster recovery Updated README.md with Argo CD section and links to docs. Updated docs/README.md to index the new Argo CD documentation. Co-Authored-By: Paperclip --- README.md | 13 ++ docs/README.md | 10 +- docs/argocd.md | 465 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 485 insertions(+), 3 deletions(-) create mode 100644 docs/argocd.md diff --git a/README.md b/README.md index d5c115b..176edd8 100644 --- a/README.md +++ b/README.md @@ -42,12 +42,25 @@ General documentation including: - Troubleshooting guides - Best practices +## GitOps with Argo CD + +This repository is managed via **Argo CD**, the GitOps deployment platform for the cluster. + +- **Argo CD UI**: https://argo.basicstack.de +- **Authentication**: Pocket ID SSO (https://auth.basicstack.de) +- **Documentation**: [docs/argocd.md](docs/argocd.md) + +All changes pushed to the `main` branch are automatically synchronized to the cluster. Applications are defined in `apps/app-*.yaml` files and reference subdirectories for their manifests. + +For details on managing applications, repository credentials, troubleshooting, and emergency procedures, see the [Argo CD documentation](docs/argocd.md). + ## Getting Started 1. Clone this repository 2. Review the example Stalwart deployment in `apps/stalwart/` 3. Follow the pattern for new application deployments 4. Ensure all manifests are tested before committing +5. Argo CD will automatically sync changes to the cluster (or use manual sync for critical changes) ## Contributing diff --git a/docs/README.md b/docs/README.md index 6b4b5bf..5085e13 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,11 +2,15 @@ This directory contains general documentation for the basicstack.de cluster and deployment processes. -## Suggested Content +## Available Documentation + +- **[Argo CD - GitOps Platform](argocd.md)**: Comprehensive guide to the Argo CD deployment, including SSO login, managing applications, repository credential rotation, emergency recovery, and troubleshooting + +## Suggested Future Content - **Architecture**: Overall cluster architecture and design decisions -- **Deployment Guides**: Step-by-step deployment procedures -- **Troubleshooting**: Common issues and solutions +- **Deployment Guides**: Step-by-step deployment procedures for specific scenarios +- **Troubleshooting**: Common cluster-wide issues and solutions - **Best Practices**: Standards and conventions for this cluster - **Operations Runbooks**: Incident response and operational procedures - **Disaster Recovery**: Backup and restore procedures at the cluster level diff --git a/docs/argocd.md b/docs/argocd.md new file mode 100644 index 0000000..62d2ecb --- /dev/null +++ b/docs/argocd.md @@ -0,0 +1,465 @@ +# Argo CD - GitOps Platform + +## Overview + +Argo CD is the GitOps deployment platform for the basicstack.de Kubernetes cluster. It continuously monitors Git repositories and automatically syncs application state to the cluster. + +- **UI URL**: https://argo.basicstack.de +- **Namespace**: `argocd` +- **Authentication**: Pocket ID SSO (OIDC) +- **Managed Repositories**: + - `stack.basicstack.de` - Infrastructure and platform applications + - `basicstack.org` - Frontend web application + +## Authentication & Access + +### SSO Login Flow + +1. Navigate to https://argo.basicstack.de +2. Click "LOGIN VIA POCKET ID" +3. You will be redirected to https://auth.basicstack.de (Pocket ID) +4. Enter your credentials and complete 2FA if prompted +5. After successful authentication, you'll be redirected back to Argo CD + +### Authorization + +Access is controlled via Pocket ID groups mapped to Argo CD roles: + +- **`argo_admins` group** → `role:admin` (full access to all resources) +- **Default users** → `role:readonly` (read-only access) + +To grant admin access to a user: +1. Log in to Pocket ID admin UI at https://auth.basicstack.de +2. Navigate to the user's profile +3. Add them to the `argo_admins` group + +### Emergency Access + +**IMPORTANT**: Local admin account is disabled by policy. If SSO is down: + +1. Do NOT reset to recovery admin - this destroys the working OIDC configuration +2. Restore SSO service first (see Pocket ID documentation) +3. If SSO cannot be restored, escalate to CTO for manual intervention + +Recovery from SSO lockout should follow this priority: +1. Fix Pocket ID service (check pods, ingress, database) +2. Verify OIDC secret is valid in `argocd` namespace +3. Only as absolute last resort: temporary re-enable local admin with explicit approval + +## Adding New Applications + +### Method 1: Via Git (Recommended) + +Applications are defined as YAML manifests in the `apps/` directory of the `stack.basicstack.de` repository. + +1. Create a new application manifest: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: my-app + namespace: argocd +spec: + project: default + source: + repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git + targetRevision: main + path: apps/my-app # Directory containing K8s manifests + destination: + server: https://kubernetes.default.svc + namespace: my-namespace + syncPolicy: + automated: + prune: false # Manual approval for deletions (recommended) + selfHeal: true # Auto-sync on Git changes + syncOptions: + - CreateNamespace=true +``` + +2. Commit the file to `stack.basicstack.de` repo: +```bash +git add apps/app-my-app.yaml +git commit -m "Add Argo CD application for my-app" +git push origin main +``` + +3. Argo CD will automatically detect and sync the new application (via the `stack-basicstack-de` self-managing application) + +### Method 2: Via UI + +1. Log in to https://argo.basicstack.de +2. Click "+ NEW APP" +3. Fill in application details: + - **Application Name**: Unique identifier + - **Project**: `default` + - **Sync Policy**: Choose automated or manual + - **Repository URL**: Select from existing or add new + - **Path**: Directory containing manifests + - **Cluster**: `https://kubernetes.default.svc` (in-cluster) + - **Namespace**: Target namespace +4. Click "CREATE" + +**Note**: Applications created via UI are not persisted in Git. Use Method 1 for production applications. + +## Repository Credential Rotation + +Both repositories use SSH key authentication with Forgejo running in-cluster. + +### Current Repositories +- `git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git` +- `git@forgejo.forgejo.svc.cluster.local:basicstack/basicstack.org.git` + +### Rotation Procedure + +1. Generate new SSH key pair on a secure workstation: +```bash +ssh-keygen -t ed25519 -C "argocd@basicstack.de" -f argocd-forgejo-key +``` + +2. Add the public key to Forgejo: + - Log in to https://forgejo.basicstack.de + - Navigate to Organization Settings → Deploy Keys + - Add the new public key with read-only access + - Give it a descriptive name (e.g., "ArgoCD Deploy Key - 2026-07") + +3. Update the sealed secrets in the repository: + +```bash +# Seal the new credentials +kubectl create secret generic repo-stack-basicstack-de \ + --namespace=argocd \ + --from-literal=type=git \ + --from-file=sshPrivateKey=argocd-forgejo-key \ + --dry-run=client -o yaml | \ + kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \ + --format=yaml > apps/argocd/repo-stack-basicstack-de-secret-sealed.yaml + +kubectl create secret generic repo-basicstack-org \ + --namespace=argocd \ + --from-literal=type=git \ + --from-file=sshPrivateKey=argocd-forgejo-key \ + --dry-run=client -o yaml | \ + kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \ + --format=yaml > apps/argocd/repo-basicstack-org-secret-sealed.yaml +``` + +4. Commit and push the updated sealed secrets: +```bash +git add apps/argocd/repo-*-secret-sealed.yaml +git commit -m "Rotate Argo CD repository credentials" +git push origin main +``` + +5. Wait for Argo CD to sync (or manually sync the `argocd` application) + +6. Verify repository connections in the UI: + - Settings → Repositories + - Both repos should show "Successful" connection status + +7. Remove the old public key from Forgejo + +8. Securely delete the private key file: +```bash +shred -u argocd-forgejo-key argocd-forgejo-key.pub +``` + +## Self-Management + +Argo CD manages itself via the `argocd` Application, which monitors `apps/argocd/` in the `stack.basicstack.de` repository. + +### Key Self-Management Settings + +```yaml +spec: + syncPolicy: + automated: + prune: false # Prevent accidental deletion of Argo CD resources + selfHeal: true # Auto-apply Git changes to Argo CD config +``` + +### Updating Argo CD Configuration + +Changes to Argo CD itself (RBAC, OIDC, ingress, etc.) are made via Git: + +1. Edit files in `apps/argocd/` directory +2. Commit and push to `main` +3. Argo CD will self-apply the changes + +**Critical files**: +- `argocd-install.yaml` - Argo CD installation manifests (CRDs, deployments) +- `argocd-ingress.yaml` - TLS ingress configuration +- `argocd-rbac-cm.yaml` - RBAC policies and group mappings (if customized) +- `argocd-oidc-secret-sealed.yaml` - Pocket ID OAuth credentials (sealed) +- `repo-*-secret-sealed.yaml` - Repository SSH keys (sealed) + +### Upgrading Argo CD + +To upgrade to a new version: + +1. Download the new installation manifest: +```bash +curl -L https://raw.githubusercontent.com/argoproj/argo-cd/v2.x.x/manifests/install.yaml -o apps/argocd/argocd-install.yaml +``` + +2. Review the diff carefully, especially: + - Breaking changes in the release notes + - Custom patches that may need reapplication + - ConfigMap/Secret schema changes + +3. Commit the updated manifest: +```bash +git add apps/argocd/argocd-install.yaml +git commit -m "Upgrade Argo CD to v2.x.x" +git push origin main +``` + +4. Monitor the upgrade in the UI or via CLI: +```bash +kubectl get pods -n argocd -w +``` + +5. Verify all applications remain healthy after the upgrade + +## Monitoring & Maintenance + +### Health Checks + +**Daily Checks** (automated via Paperclip or manual): +```bash +# Check Argo CD pods are running +kubectl get pods -n argocd + +# Check application sync status +kubectl get applications -n argocd + +# Check for out-of-sync or unhealthy apps +kubectl get applications -n argocd -o json | \ + jq -r '.items[] | select(.status.sync.status != "Synced" or .status.health.status != "Healthy") | .metadata.name' +``` + +**UI Health Dashboard**: +- Navigate to https://argo.basicstack.de +- Applications with issues are highlighted in red/yellow +- Click into an application to see detailed sync/health status + +### Common Issues + +#### Application Stuck in "Syncing" +```bash +# Check sync operation status +kubectl describe application -n argocd + +# Cancel stuck sync +kubectl patch application -n argocd --type json \ + -p='[{"op": "remove", "path": "/operation"}]' +``` + +#### Repository Connection Failed +```bash +# Check repository secret exists +kubectl get secret repo- -n argocd + +# View Argo CD repo server logs +kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server +``` + +#### Out of Sync Applications +- **Manual intervention**: Application has `syncPolicy: manual` or automated sync is disabled +- **Failed sync**: Check sync logs in the UI or via `kubectl describe application` +- **Resource drift**: External changes made directly to cluster (not via Git) + - Solution: Either revert cluster changes or commit them to Git + +### Log Access + +```bash +# Argo CD server logs (API/UI) +kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server + +# Application controller logs (sync operations) +kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller + +# Repo server logs (Git operations) +kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server +``` + +### Backup & Disaster Recovery + +Argo CD state is derived from Git, but the following should be backed up: + +**Critical Data**: +1. **Sealed secrets source keys** - `sealed-secrets` namespace +2. **OIDC credentials** - `argocd-oidc-secret` (sealed in Git) +3. **Repository SSH keys** - `repo-*` secrets (sealed in Git) + +**Recovery Procedure** (cluster rebuild): + +1. Restore k3s cluster and sealed-secrets controller +2. Apply Argo CD installation: +```bash +kubectl create namespace argocd +kubectl apply -f apps/argocd/argocd-install.yaml +``` + +3. Wait for Argo CD pods to be ready: +```bash +kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s +``` + +4. Apply sealed secrets (will be unsealed by controller): +```bash +kubectl apply -f apps/argocd/argocd-oidc-secret-sealed.yaml +kubectl apply -f apps/argocd/repo-stack-basicstack-de-secret-sealed.yaml +kubectl apply -f apps/argocd/repo-basicstack-org-secret-sealed.yaml +``` + +5. Apply Argo CD application manifests: +```bash +kubectl apply -f apps/app-argocd.yaml +kubectl apply -f apps/app-stack-basicstack-de.yaml +kubectl apply -f apps/app-basicstack-org.yaml +``` + +6. Verify sync in UI: https://argo.basicstack.de + +## Architecture + +### Application Hierarchy + +``` +argocd (self-managing) + └── apps/argocd/ + ├── argocd-install.yaml # CRDs, deployments, services + ├── argocd-ingress.yaml # TLS ingress + ├── argocd-oidc-secret-sealed.yaml # Pocket ID credentials + └── repo-*-secret-sealed.yaml # Git credentials + +stack-basicstack-de (infrastructure) + └── apps/ (recursive, excludes apps/argocd/) + ├── app-basicstack-org.yaml # Declares basicstack-org app + ├── app-stack-basicstack-de.yaml # Declares self + ├── bookstack/ + ├── directus/ + ├── forgejo/ + ├── paperclip/ + └── ... (other platform services) + +basicstack-org (frontend application) + └── k8s/ + ├── deployment.yaml + ├── service.yaml + └── ingress.yaml +``` + +### Sync Policies + +| Application | Prune | Self-Heal | Rationale | +|------------|-------|-----------|-----------| +| `argocd` | `false` | `true` | Manual approval for deletions prevents accidental Argo CD breakage | +| `stack-basicstack-de` | `false` | `true` | Manual approval for platform service deletions | +| `basicstack-org` | `true` | `true` | Fully automated for stateless frontend | + +## CLI Usage + +Install the Argo CD CLI: +```bash +# Linux/WSL +curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64 +chmod +x argocd +sudo mv argocd /usr/local/bin/ +``` + +Login via SSO: +```bash +argocd login argo.basicstack.de --sso +``` + +Common commands: +```bash +# List applications +argocd app list + +# Get application details +argocd app get + +# Sync application +argocd app sync + +# Watch sync progress +argocd app wait + +# View application logs +argocd app logs + +# Set application sync policy +argocd app set --sync-policy automated --auto-prune --self-heal +``` + +## Troubleshooting Checklist + +When Argo CD has issues, work through this checklist: + +### 1. Argo CD Pods Not Running +```bash +kubectl get pods -n argocd +kubectl describe pod -n argocd +kubectl logs -n argocd +``` + +### 2. SSO Login Fails +- [ ] Pocket ID is accessible: `curl -I https://auth.basicstack.de` +- [ ] OIDC secret exists: `kubectl get secret argocd-oidc-secret -n argocd` +- [ ] OIDC config in argocd-cm: `kubectl get cm argocd-cm -n argocd -o yaml | grep -A 10 oidc.config` +- [ ] Check Argo CD server logs for OIDC errors + +### 3. Repository Connection Failed +- [ ] Forgejo is accessible: `kubectl get pods -n forgejo` +- [ ] Repository secret exists: `kubectl get secret repo- -n argocd` +- [ ] SSH key is valid in Forgejo deploy keys +- [ ] Check repo-server logs: `kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server` + +### 4. Application Sync Fails +- [ ] Check application status: `kubectl describe application -n argocd` +- [ ] Review manifests in Git for syntax errors +- [ ] Check destination namespace exists +- [ ] Verify RBAC permissions if deploying to specific namespaces +- [ ] Review sync operation in UI for detailed error messages + +### 5. Ingress/TLS Issues +- [ ] Ingress object exists: `kubectl get ingress argocd-server -n argocd` +- [ ] TLS certificate is valid: `kubectl get certificate argocd-server-tls -n argocd` +- [ ] Traefik is routing correctly: `kubectl logs -n kube-system -l app.kubernetes.io/name=traefik` +- [ ] DNS resolves: `dig argo.basicstack.de` + +## Security Considerations + +### Sealed Secrets + +All sensitive data (OIDC credentials, SSH keys) are encrypted using sealed-secrets before being committed to Git. The sealing key is stored in the `sealed-secrets` namespace and should be backed up securely. + +Never commit unsealed Secret manifests to Git. + +### RBAC + +The default policy is `role:readonly` - all users have read-only access unless explicitly granted admin via the `argo_admins` group. + +Admin permissions include: +- Syncing applications +- Creating/deleting applications +- Managing repositories and clusters +- Modifying project settings +- Executing into containers + +### Network Security + +Argo CD uses in-cluster Forgejo for Git operations: +- Repository URLs use cluster DNS: `git@forgejo.forgejo.svc.cluster.local` +- No external Git credentials or tokens are exposed +- SSH keys are scoped to specific repositories as deploy keys (read-only) + +## References + +- [Argo CD Documentation](https://argo-cd.readthedocs.io/) +- [Argo CD Best Practices](https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/) +- [Pocket ID (SSO Provider)](https://github.com/stonith404/pocket-id) +- [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)