stack.basicstack.de/docs/argocd.md

# Argo CD - GitOps Platform

## Overview

Argo CD is the GitOps deployment platform for the basicstack.de Kubernetes cluster. It continuously monitors Git repositories and automatically syncs application state to the cluster.

- **UI URL**: https://argo.basicstack.de
- **Namespace**: `argocd`
- **Authentication**: Pocket ID SSO (OIDC)
- **Managed Repositories**:
  - `stack.basicstack.de` - Infrastructure and platform applications
  - `basicstack.org` - Frontend web application

## Authentication & Access

### SSO Login Flow

1. Navigate to https://argo.basicstack.de
2. Click "LOGIN VIA POCKET ID"
3. You will be redirected to https://auth.basicstack.de (Pocket ID)
4. Enter your credentials and complete 2FA if prompted
5. After successful authentication, you'll be redirected back to Argo CD

### Authorization

Access is controlled via Pocket ID groups mapped to Argo CD roles:

- **`argo_admins` group** → `role:admin` (full access to all resources)
- **Default users** → `role:readonly` (read-only access)

To grant admin access to a user:
1. Log in to Pocket ID admin UI at https://auth.basicstack.de
2. Navigate to the user's profile
3. Add them to the `argo_admins` group

### Emergency Access

**IMPORTANT**: Local admin account is disabled by policy. If SSO is down:

1. Do NOT reset to recovery admin - this destroys the working OIDC configuration
2. Restore SSO service first (see Pocket ID documentation)
3. If SSO cannot be restored, escalate to CTO for manual intervention

Recovery from SSO lockout should follow this priority:
1. Fix Pocket ID service (check pods, ingress, database)
2. Verify OIDC secret is valid in `argocd` namespace
3. Only as absolute last resort: temporary re-enable local admin with explicit approval

## Adding New Applications

### Method 1: Via Git (Recommended)

Applications are defined as YAML manifests in the `apps/` directory of the `stack.basicstack.de` repository.

1. Create a new application manifest:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
    targetRevision: main
    path: apps/my-app  # Directory containing K8s manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: my-namespace
  syncPolicy:
    automated:
      prune: false  # Manual approval for deletions (recommended)
      selfHeal: true  # Auto-sync on Git changes
    syncOptions:
    - CreateNamespace=true
```

2. Commit the file to `stack.basicstack.de` repo:
```bash
git add apps/app-my-app.yaml
git commit -m "Add Argo CD application for my-app"
git push origin main
```

3. Argo CD will automatically detect and sync the new application (via the `stack-basicstack-de` self-managing application)

### Method 2: Via UI

1. Log in to https://argo.basicstack.de
2. Click "+ NEW APP"
3. Fill in application details:
   - **Application Name**: Unique identifier
   - **Project**: `default`
   - **Sync Policy**: Choose automated or manual
   - **Repository URL**: Select from existing or add new
   - **Path**: Directory containing manifests
   - **Cluster**: `https://kubernetes.default.svc` (in-cluster)
   - **Namespace**: Target namespace
4. Click "CREATE"

**Note**: Applications created via UI are not persisted in Git. Use Method 1 for production applications.

## Repository Credential Rotation

Both repositories use SSH key authentication with Forgejo running in-cluster.

### Current Repositories
- `git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git`
- `git@forgejo.forgejo.svc.cluster.local:basicstack/basicstack.org.git`

### Rotation Procedure

1. Generate new SSH key pair on a secure workstation:
```bash
ssh-keygen -t ed25519 -C "argocd@basicstack.de" -f argocd-forgejo-key
```

2. Add the public key to Forgejo:
   - Log in to https://forgejo.basicstack.de
   - Navigate to Organization Settings → Deploy Keys
   - Add the new public key with read-only access
   - Give it a descriptive name (e.g., "ArgoCD Deploy Key - 2026-07")

3. Update the sealed secrets in the repository:

```bash
# Seal the new credentials
kubectl create secret generic repo-stack-basicstack-de \
  --namespace=argocd \
  --from-literal=type=git \
  --from-file=sshPrivateKey=argocd-forgejo-key \
  --dry-run=client -o yaml | \
  kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \
  --format=yaml > apps/argocd/repo-stack-basicstack-de-secret-sealed.yaml

kubectl create secret generic repo-basicstack-org \
  --namespace=argocd \
  --from-literal=type=git \
  --from-file=sshPrivateKey=argocd-forgejo-key \
  --dry-run=client -o yaml | \
  kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \
  --format=yaml > apps/argocd/repo-basicstack-org-secret-sealed.yaml
```

4. Commit and push the updated sealed secrets:
```bash
git add apps/argocd/repo-*-secret-sealed.yaml
git commit -m "Rotate Argo CD repository credentials"
git push origin main
```

5. Wait for Argo CD to sync (or manually sync the `argocd` application)

6. Verify repository connections in the UI:
   - Settings → Repositories
   - Both repos should show "Successful" connection status

7. Remove the old public key from Forgejo

8. Securely delete the private key file:
```bash
shred -u argocd-forgejo-key argocd-forgejo-key.pub
```

## Self-Management

Argo CD manages itself via the `argocd` Application, which monitors `apps/argocd/` in the `stack.basicstack.de` repository.

### Key Self-Management Settings

```yaml
spec:
  syncPolicy:
    automated:
      prune: false  # Prevent accidental deletion of Argo CD resources
      selfHeal: true  # Auto-apply Git changes to Argo CD config
```

### Updating Argo CD Configuration

Changes to Argo CD itself (RBAC, OIDC, ingress, etc.) are made via Git:

1. Edit files in `apps/argocd/` directory
2. Commit and push to `main`
3. Argo CD will self-apply the changes

**Critical files**:
- `argocd-install.yaml` - Argo CD installation manifests (CRDs, deployments)
- `argocd-ingress.yaml` - TLS ingress configuration
- `argocd-rbac-cm.yaml` - RBAC policies and group mappings (if customized)
- `argocd-oidc-secret-sealed.yaml` - Pocket ID OAuth credentials (sealed)
- `repo-*-secret-sealed.yaml` - Repository SSH keys (sealed)

### Upgrading Argo CD

To upgrade to a new version:

1. Download the new installation manifest:
```bash
curl -L https://raw.githubusercontent.com/argoproj/argo-cd/v2.x.x/manifests/install.yaml -o apps/argocd/argocd-install.yaml
```

2. Review the diff carefully, especially:
   - Breaking changes in the release notes
   - Custom patches that may need reapplication
   - ConfigMap/Secret schema changes

3. Commit the updated manifest:
```bash
git add apps/argocd/argocd-install.yaml
git commit -m "Upgrade Argo CD to v2.x.x"
git push origin main
```

4. Monitor the upgrade in the UI or via CLI:
```bash
kubectl get pods -n argocd -w
```

5. Verify all applications remain healthy after the upgrade

## Monitoring & Maintenance

### Health Checks

**Daily Checks** (automated via Paperclip or manual):
```bash
# Check Argo CD pods are running
kubectl get pods -n argocd

# Check application sync status
kubectl get applications -n argocd

# Check for out-of-sync or unhealthy apps
kubectl get applications -n argocd -o json | \
  jq -r '.items[] | select(.status.sync.status != "Synced" or .status.health.status != "Healthy") | .metadata.name'
```

**UI Health Dashboard**:
- Navigate to https://argo.basicstack.de
- Applications with issues are highlighted in red/yellow
- Click into an application to see detailed sync/health status

### Common Issues

#### Application Stuck in "Syncing"
```bash
# Check sync operation status
kubectl describe application <app-name> -n argocd

# Cancel stuck sync
kubectl patch application <app-name> -n argocd --type json \
  -p='[{"op": "remove", "path": "/operation"}]'
```

#### Repository Connection Failed
```bash
# Check repository secret exists
kubectl get secret repo-<repo-name> -n argocd

# View Argo CD repo server logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server
```

#### Out of Sync Applications
- **Manual intervention**: Application has `syncPolicy: manual` or automated sync is disabled
- **Failed sync**: Check sync logs in the UI or via `kubectl describe application`
- **Resource drift**: External changes made directly to cluster (not via Git)
  - Solution: Either revert cluster changes or commit them to Git

### Log Access

```bash
# Argo CD server logs (API/UI)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server

# Application controller logs (sync operations)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller

# Repo server logs (Git operations)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server
```

### Backup & Disaster Recovery

Argo CD state is derived from Git, but the following should be backed up:

**Critical Data**:
1. **Sealed secrets source keys** - `sealed-secrets` namespace
2. **OIDC credentials** - `argocd-oidc-secret` (sealed in Git)
3. **Repository SSH keys** - `repo-*` secrets (sealed in Git)

**Recovery Procedure** (cluster rebuild):

1. Restore k3s cluster and sealed-secrets controller
2. Apply Argo CD installation:
```bash
kubectl create namespace argocd
kubectl apply -f apps/argocd/argocd-install.yaml
```

3. Wait for Argo CD pods to be ready:
```bash
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s
```

4. Apply sealed secrets (will be unsealed by controller):
```bash
kubectl apply -f apps/argocd/argocd-oidc-secret-sealed.yaml
kubectl apply -f apps/argocd/repo-stack-basicstack-de-secret-sealed.yaml
kubectl apply -f apps/argocd/repo-basicstack-org-secret-sealed.yaml
```

5. Apply Argo CD application manifests:
```bash
kubectl apply -f apps/app-argocd.yaml
kubectl apply -f apps/app-stack-basicstack-de.yaml
kubectl apply -f apps/app-basicstack-org.yaml
```

6. Verify sync in UI: https://argo.basicstack.de

## Architecture

### Application Hierarchy

```
argocd (self-managing)
  └── apps/argocd/
      ├── argocd-install.yaml       # CRDs, deployments, services
      ├── argocd-ingress.yaml       # TLS ingress
      ├── argocd-oidc-secret-sealed.yaml  # Pocket ID credentials
      └── repo-*-secret-sealed.yaml       # Git credentials

stack-basicstack-de (infrastructure)
  └── apps/ (recursive, excludes apps/argocd/)
      ├── app-basicstack-org.yaml   # Declares basicstack-org app
      ├── app-stack-basicstack-de.yaml  # Declares self
      ├── bookstack/
      ├── directus/
      ├── forgejo/
      ├── paperclip/
      └── ... (other platform services)

basicstack-org (frontend application)
  └── k8s/
      ├── deployment.yaml
      ├── service.yaml
      └── ingress.yaml
```

### Sync Policies

| Application | Prune | Self-Heal | Rationale |
|------------|-------|-----------|-----------|
| `argocd` | `false` | `true` | Manual approval for deletions prevents accidental Argo CD breakage |
| `stack-basicstack-de` | `false` | `true` | Manual approval for platform service deletions |
| `basicstack-org` | `true` | `true` | Fully automated for stateless frontend |

## CLI Usage

Install the Argo CD CLI:
```bash
# Linux/WSL
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
chmod +x argocd
sudo mv argocd /usr/local/bin/
```

Login via SSO:
```bash
argocd login argo.basicstack.de --sso
```

Common commands:
```bash
# List applications
argocd app list

# Get application details
argocd app get <app-name>

# Sync application
argocd app sync <app-name>

# Watch sync progress
argocd app wait <app-name>

# View application logs
argocd app logs <app-name>

# Set application sync policy
argocd app set <app-name> --sync-policy automated --auto-prune --self-heal
```

## Troubleshooting Checklist

When Argo CD has issues, work through this checklist:

### 1. Argo CD Pods Not Running
```bash
kubectl get pods -n argocd
kubectl describe pod <failing-pod> -n argocd
kubectl logs <failing-pod> -n argocd
```

### 2. SSO Login Fails
- [ ] Pocket ID is accessible: `curl -I https://auth.basicstack.de`
- [ ] OIDC secret exists: `kubectl get secret argocd-oidc-secret -n argocd`
- [ ] OIDC config in argocd-cm: `kubectl get cm argocd-cm -n argocd -o yaml | grep -A 10 oidc.config`
- [ ] Check Argo CD server logs for OIDC errors

### 3. Repository Connection Failed
- [ ] Forgejo is accessible: `kubectl get pods -n forgejo`
- [ ] Repository secret exists: `kubectl get secret repo-<name> -n argocd`
- [ ] SSH key is valid in Forgejo deploy keys
- [ ] Check repo-server logs: `kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server`

### 4. Application Sync Fails
- [ ] Check application status: `kubectl describe application <name> -n argocd`
- [ ] Review manifests in Git for syntax errors
- [ ] Check destination namespace exists
- [ ] Verify RBAC permissions if deploying to specific namespaces
- [ ] Review sync operation in UI for detailed error messages

### 5. Ingress/TLS Issues
- [ ] Ingress object exists: `kubectl get ingress argocd-server -n argocd`
- [ ] TLS certificate is valid: `kubectl get certificate argocd-server-tls -n argocd`
- [ ] Traefik is routing correctly: `kubectl logs -n kube-system -l app.kubernetes.io/name=traefik`
- [ ] DNS resolves: `dig argo.basicstack.de`

## Security Considerations

### Sealed Secrets

All sensitive data (OIDC credentials, SSH keys) are encrypted using sealed-secrets before being committed to Git. The sealing key is stored in the `sealed-secrets` namespace and should be backed up securely.

Never commit unsealed Secret manifests to Git.

### RBAC

The default policy is `role:readonly` - all users have read-only access unless explicitly granted admin via the `argo_admins` group.

Admin permissions include:
- Syncing applications
- Creating/deleting applications
- Managing repositories and clusters
- Modifying project settings
- Executing into containers

### Network Security

Argo CD uses in-cluster Forgejo for Git operations:
- Repository URLs use cluster DNS: `git@forgejo.forgejo.svc.cluster.local`
- No external Git credentials or tokens are exposed
- SSH keys are scoped to specific repositories as deploy keys (read-only)

## References

- [Argo CD Documentation](https://argo-cd.readthedocs.io/)
- [Argo CD Best Practices](https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/)
- [Pocket ID (SSO Provider)](https://github.com/stonith404/pocket-id)
- [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)