basicstack/stack.basicstack.de
2
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Add comprehensive Argo CD documentation

Browse source 
Created detailed documentation covering:
- Authentication and SSO login flow
- Adding new applications (Git and UI methods)
- Repository credential rotation procedure
- Emergency recovery procedures (with critical safety warnings)
- Self-management architecture
- Monitoring, maintenance, and troubleshooting
- Backup and disaster recovery

Updated README.md with Argo CD section and links to docs.
Updated docs/README.md to index the new Argo CD documentation.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
CTO Agent 2026-07-12 10:04:51 +00:00
parent 03da09370d
commit 18bdb78429
3 changed files with 485 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
README.md
View file

@ -42,12 +42,25 @@ General documentation including:
- Troubleshooting guides
- Best practices
## GitOps with Argo CD
This repository is managed via **Argo CD**, the GitOps deployment platform for the cluster.
- **Argo CD UI**: https://argo.basicstack.de
- **Authentication**: Pocket ID SSO (https://auth.basicstack.de)
- **Documentation**: [docs/argocd.md](docs/argocd.md)
All changes pushed to the `main` branch are automatically synchronized to the cluster. Applications are defined in `apps/app-*.yaml` files and reference subdirectories for their manifests.
For details on managing applications, repository credentials, troubleshooting, and emergency procedures, see the [Argo CD documentation](docs/argocd.md).
## Getting Started
1. Clone this repository
2. Review the example Stalwart deployment in `apps/stalwart/`
3. Follow the pattern for new application deployments
4. Ensure all manifests are tested before committing
5. Argo CD will automatically sync changes to the cluster (or use manual sync for critical changes)
## Contributing

10
docs/README.md
View file

@ -2,11 +2,15 @@
This directory contains general documentation for the basicstack.de cluster and deployment processes.
## Suggested Content
## Available Documentation
- **[Argo CD - GitOps Platform](argocd.md)**: Comprehensive guide to the Argo CD deployment, including SSO login, managing applications, repository credential rotation, emergency recovery, and troubleshooting
## Suggested Future Content
- **Architecture**: Overall cluster architecture and design decisions
- **Deployment Guides**: Step-by-step deployment procedures
- **Troubleshooting**: Common issues and solutions
- **Deployment Guides**: Step-by-step deployment procedures for specific scenarios
- **Troubleshooting**: Common cluster-wide issues and solutions
- **Best Practices**: Standards and conventions for this cluster
- **Operations Runbooks**: Incident response and operational procedures
- **Disaster Recovery**: Backup and restore procedures at the cluster level

465
docs/argocd.md Normal file
View file

@ -0,0 +1,465 @@
# Argo CD - GitOps Platform
## Overview
Argo CD is the GitOps deployment platform for the basicstack.de Kubernetes cluster. It continuously monitors Git repositories and automatically syncs application state to the cluster.
- **UI URL**: https://argo.basicstack.de
- **Namespace**: `argocd`
- **Authentication**: Pocket ID SSO (OIDC)
- **Managed Repositories**:
- `stack.basicstack.de` - Infrastructure and platform applications
- `basicstack.org` - Frontend web application
## Authentication & Access
### SSO Login Flow
1. Navigate to https://argo.basicstack.de
2. Click "LOGIN VIA POCKET ID"
3. You will be redirected to https://auth.basicstack.de (Pocket ID)
4. Enter your credentials and complete 2FA if prompted
5. After successful authentication, you'll be redirected back to Argo CD
### Authorization
Access is controlled via Pocket ID groups mapped to Argo CD roles:
- **`argo_admins` group** → `role:admin` (full access to all resources)
- **Default users** → `role:readonly` (read-only access)
To grant admin access to a user:
1. Log in to Pocket ID admin UI at https://auth.basicstack.de
2. Navigate to the user's profile
3. Add them to the `argo_admins` group
### Emergency Access
**IMPORTANT**: Local admin account is disabled by policy. If SSO is down:
1. Do NOT reset to recovery admin - this destroys the working OIDC configuration
2. Restore SSO service first (see Pocket ID documentation)
3. If SSO cannot be restored, escalate to CTO for manual intervention
Recovery from SSO lockout should follow this priority:
1. Fix Pocket ID service (check pods, ingress, database)
2. Verify OIDC secret is valid in `argocd` namespace
3. Only as absolute last resort: temporary re-enable local admin with explicit approval
## Adding New Applications
### Method 1: Via Git (Recommended)
Applications are defined as YAML manifests in the `apps/` directory of the `stack.basicstack.de` repository.
1. Create a new application manifest:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: default
source:
repoURL: git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git
targetRevision: main
path: apps/my-app # Directory containing K8s manifests
destination:
server: https://kubernetes.default.svc
namespace: my-namespace
syncPolicy:
automated:
prune: false # Manual approval for deletions (recommended)
selfHeal: true # Auto-sync on Git changes
syncOptions:
- CreateNamespace=true
```
2. Commit the file to `stack.basicstack.de` repo:
```bash
git add apps/app-my-app.yaml
git commit -m "Add Argo CD application for my-app"
git push origin main
```
3. Argo CD will automatically detect and sync the new application (via the `stack-basicstack-de` self-managing application)
### Method 2: Via UI
1. Log in to https://argo.basicstack.de
2. Click "+ NEW APP"
3. Fill in application details:
- **Application Name**: Unique identifier
- **Project**: `default`
- **Sync Policy**: Choose automated or manual
- **Repository URL**: Select from existing or add new
- **Path**: Directory containing manifests
- **Cluster**: `https://kubernetes.default.svc` (in-cluster)
- **Namespace**: Target namespace
4. Click "CREATE"
**Note**: Applications created via UI are not persisted in Git. Use Method 1 for production applications.
## Repository Credential Rotation
Both repositories use SSH key authentication with Forgejo running in-cluster.
### Current Repositories
- `git@forgejo.forgejo.svc.cluster.local:basicstack/stack.basicstack.de.git`
- `git@forgejo.forgejo.svc.cluster.local:basicstack/basicstack.org.git`
### Rotation Procedure
1. Generate new SSH key pair on a secure workstation:
```bash
ssh-keygen -t ed25519 -C "argocd@basicstack.de" -f argocd-forgejo-key
```
2. Add the public key to Forgejo:
- Log in to https://forgejo.basicstack.de
- Navigate to Organization Settings → Deploy Keys
- Add the new public key with read-only access
- Give it a descriptive name (e.g., "ArgoCD Deploy Key - 2026-07")
3. Update the sealed secrets in the repository:
```bash
# Seal the new credentials
kubectl create secret generic repo-stack-basicstack-de \
--namespace=argocd \
--from-literal=type=git \
--from-file=sshPrivateKey=argocd-forgejo-key \
--dry-run=client -o yaml | \
kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \
--format=yaml > apps/argocd/repo-stack-basicstack-de-secret-sealed.yaml
kubectl create secret generic repo-basicstack-org \
--namespace=argocd \
--from-literal=type=git \
--from-file=sshPrivateKey=argocd-forgejo-key \
--dry-run=client -o yaml | \
kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \
--format=yaml > apps/argocd/repo-basicstack-org-secret-sealed.yaml
```
4. Commit and push the updated sealed secrets:
```bash
git add apps/argocd/repo-*-secret-sealed.yaml
git commit -m "Rotate Argo CD repository credentials"
git push origin main
```
5. Wait for Argo CD to sync (or manually sync the `argocd` application)
6. Verify repository connections in the UI:
- Settings → Repositories
- Both repos should show "Successful" connection status
7. Remove the old public key from Forgejo
8. Securely delete the private key file:
```bash
shred -u argocd-forgejo-key argocd-forgejo-key.pub
```
## Self-Management
Argo CD manages itself via the `argocd` Application, which monitors `apps/argocd/` in the `stack.basicstack.de` repository.
### Key Self-Management Settings
```yaml
spec:
syncPolicy:
automated:
prune: false # Prevent accidental deletion of Argo CD resources
selfHeal: true # Auto-apply Git changes to Argo CD config
```
### Updating Argo CD Configuration
Changes to Argo CD itself (RBAC, OIDC, ingress, etc.) are made via Git:
1. Edit files in `apps/argocd/` directory
2. Commit and push to `main`
3. Argo CD will self-apply the changes
**Critical files**:
- `argocd-install.yaml` - Argo CD installation manifests (CRDs, deployments)
- `argocd-ingress.yaml` - TLS ingress configuration
- `argocd-rbac-cm.yaml` - RBAC policies and group mappings (if customized)
- `argocd-oidc-secret-sealed.yaml` - Pocket ID OAuth credentials (sealed)
- `repo-*-secret-sealed.yaml` - Repository SSH keys (sealed)
### Upgrading Argo CD
To upgrade to a new version:
1. Download the new installation manifest:
```bash
curl -L https://raw.githubusercontent.com/argoproj/argo-cd/v2.x.x/manifests/install.yaml -o apps/argocd/argocd-install.yaml
```
2. Review the diff carefully, especially:
- Breaking changes in the release notes
- Custom patches that may need reapplication
- ConfigMap/Secret schema changes
3. Commit the updated manifest:
```bash
git add apps/argocd/argocd-install.yaml
git commit -m "Upgrade Argo CD to v2.x.x"
git push origin main
```
4. Monitor the upgrade in the UI or via CLI:
```bash
kubectl get pods -n argocd -w
```
5. Verify all applications remain healthy after the upgrade
## Monitoring & Maintenance
### Health Checks
**Daily Checks** (automated via Paperclip or manual):
```bash
# Check Argo CD pods are running
kubectl get pods -n argocd
# Check application sync status
kubectl get applications -n argocd
# Check for out-of-sync or unhealthy apps
kubectl get applications -n argocd -o json | \
jq -r '.items[] | select(.status.sync.status != "Synced" or .status.health.status != "Healthy") | .metadata.name'
```
**UI Health Dashboard**:
- Navigate to https://argo.basicstack.de
- Applications with issues are highlighted in red/yellow
- Click into an application to see detailed sync/health status
### Common Issues
#### Application Stuck in "Syncing"
```bash
# Check sync operation status
kubectl describe application <app-name> -n argocd
# Cancel stuck sync
kubectl patch application <app-name> -n argocd --type json \
-p='[{"op": "remove", "path": "/operation"}]'
```
#### Repository Connection Failed
```bash
# Check repository secret exists
kubectl get secret repo-<repo-name> -n argocd
# View Argo CD repo server logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server
```
#### Out of Sync Applications
- **Manual intervention**: Application has `syncPolicy: manual` or automated sync is disabled
- **Failed sync**: Check sync logs in the UI or via `kubectl describe application`
- **Resource drift**: External changes made directly to cluster (not via Git)
- Solution: Either revert cluster changes or commit them to Git
### Log Access
```bash
# Argo CD server logs (API/UI)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server
# Application controller logs (sync operations)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller
# Repo server logs (Git operations)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server
```
### Backup & Disaster Recovery
Argo CD state is derived from Git, but the following should be backed up:
**Critical Data**:
1. **Sealed secrets source keys** - `sealed-secrets` namespace
2. **OIDC credentials** - `argocd-oidc-secret` (sealed in Git)
3. **Repository SSH keys** - `repo-*` secrets (sealed in Git)
**Recovery Procedure** (cluster rebuild):
1. Restore k3s cluster and sealed-secrets controller
2. Apply Argo CD installation:
```bash
kubectl create namespace argocd
kubectl apply -f apps/argocd/argocd-install.yaml
```
3. Wait for Argo CD pods to be ready:
```bash
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s
```
4. Apply sealed secrets (will be unsealed by controller):
```bash
kubectl apply -f apps/argocd/argocd-oidc-secret-sealed.yaml
kubectl apply -f apps/argocd/repo-stack-basicstack-de-secret-sealed.yaml
kubectl apply -f apps/argocd/repo-basicstack-org-secret-sealed.yaml
```
5. Apply Argo CD application manifests:
```bash
kubectl apply -f apps/app-argocd.yaml
kubectl apply -f apps/app-stack-basicstack-de.yaml
kubectl apply -f apps/app-basicstack-org.yaml
```
6. Verify sync in UI: https://argo.basicstack.de
## Architecture
### Application Hierarchy
```
argocd (self-managing)
└── apps/argocd/
├── argocd-install.yaml # CRDs, deployments, services
├── argocd-ingress.yaml # TLS ingress
├── argocd-oidc-secret-sealed.yaml # Pocket ID credentials
└── repo-*-secret-sealed.yaml # Git credentials
stack-basicstack-de (infrastructure)
└── apps/ (recursive, excludes apps/argocd/)
├── app-basicstack-org.yaml # Declares basicstack-org app
├── app-stack-basicstack-de.yaml # Declares self
├── bookstack/
├── directus/
├── forgejo/
├── paperclip/
└── ... (other platform services)
basicstack-org (frontend application)
└── k8s/
├── deployment.yaml
├── service.yaml
└── ingress.yaml
```
### Sync Policies
| Application | Prune | Self-Heal | Rationale |
|------------|-------|-----------|-----------|
| `argocd` | `false` | `true` | Manual approval for deletions prevents accidental Argo CD breakage |
| `stack-basicstack-de` | `false` | `true` | Manual approval for platform service deletions |
| `basicstack-org` | `true` | `true` | Fully automated for stateless frontend |
## CLI Usage
Install the Argo CD CLI:
```bash
# Linux/WSL
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
chmod +x argocd
sudo mv argocd /usr/local/bin/
```
Login via SSO:
```bash
argocd login argo.basicstack.de --sso
```
Common commands:
```bash
# List applications
argocd app list
# Get application details
argocd app get <app-name>
# Sync application
argocd app sync <app-name>
# Watch sync progress
argocd app wait <app-name>
# View application logs
argocd app logs <app-name>
# Set application sync policy
argocd app set <app-name> --sync-policy automated --auto-prune --self-heal
```
## Troubleshooting Checklist
When Argo CD has issues, work through this checklist:
### 1. Argo CD Pods Not Running
```bash
kubectl get pods -n argocd
kubectl describe pod <failing-pod> -n argocd
kubectl logs <failing-pod> -n argocd
```
### 2. SSO Login Fails
- [ ] Pocket ID is accessible: `curl -I https://auth.basicstack.de`
- [ ] OIDC secret exists: `kubectl get secret argocd-oidc-secret -n argocd`
- [ ] OIDC config in argocd-cm: `kubectl get cm argocd-cm -n argocd -o yaml | grep -A 10 oidc.config`
- [ ] Check Argo CD server logs for OIDC errors
### 3. Repository Connection Failed
- [ ] Forgejo is accessible: `kubectl get pods -n forgejo`
- [ ] Repository secret exists: `kubectl get secret repo-<name> -n argocd`
- [ ] SSH key is valid in Forgejo deploy keys
- [ ] Check repo-server logs: `kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server`
### 4. Application Sync Fails
- [ ] Check application status: `kubectl describe application <name> -n argocd`
- [ ] Review manifests in Git for syntax errors
- [ ] Check destination namespace exists
- [ ] Verify RBAC permissions if deploying to specific namespaces
- [ ] Review sync operation in UI for detailed error messages
### 5. Ingress/TLS Issues
- [ ] Ingress object exists: `kubectl get ingress argocd-server -n argocd`
- [ ] TLS certificate is valid: `kubectl get certificate argocd-server-tls -n argocd`
- [ ] Traefik is routing correctly: `kubectl logs -n kube-system -l app.kubernetes.io/name=traefik`
- [ ] DNS resolves: `dig argo.basicstack.de`
## Security Considerations
### Sealed Secrets
All sensitive data (OIDC credentials, SSH keys) are encrypted using sealed-secrets before being committed to Git. The sealing key is stored in the `sealed-secrets` namespace and should be backed up securely.
Never commit unsealed Secret manifests to Git.
### RBAC
The default policy is `role:readonly` - all users have read-only access unless explicitly granted admin via the `argo_admins` group.
Admin permissions include:
- Syncing applications
- Creating/deleting applications
- Managing repositories and clusters
- Modifying project settings
- Executing into containers
### Network Security
Argo CD uses in-cluster Forgejo for Git operations:
- Repository URLs use cluster DNS: `git@forgejo.forgejo.svc.cluster.local`
- No external Git credentials or tokens are exposed
- SSH keys are scoped to specific repositories as deploy keys (read-only)
## References
- [Argo CD Documentation](https://argo-cd.readthedocs.io/)
- [Argo CD Best Practices](https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/)
- [Pocket ID (SSO Provider)](https://github.com/stonith404/pocket-id)
- [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)