Development Workflow (Coder Workspace)
graph TB
A[AI Agent Picks GitHub Issue] --> B[AI Requests Dev Environment]
B --> C[Conductor Receives Request]
C --> D[Trigger Secret Sync Workflow]
D --> E[ARC Runner Syncs Secrets]
E --> F[Create K8s Namespace]
F --> G[Provision Coder Workspace]
G --> H[Deploy Dev Pod with Pre-built Image]
H --> I[Attach Monitoring: Grafana + Promtail]
I --> J[Attach Logging: Loki]
J --> K[Mount Secrets to Workspace]
K --> L[AI Works in Workspace]
L --> M[AI Writes Code & Tests]
M --> N[AI Runs Unit Tests]
N --> O[AI Commits to Feature Branch]
O --> P[AI Tags: run-integration-tests]
P --> Q[Push Tag to GitHub]
style A fill:#e1f5ff
style L fill:#fff3e0
style M fill:#fff3e0
style N fill:#fff3e0
style O fill:#fff3e0
style P fill:#c8e6c9
style Q fill:#c8e6c9
Key Components
- Coder Workspace: Full IDE environment in Kubernetes
- Dev Pod: Pre-built development image with tools
- Secret Sync: GitHub Secrets → ARC → K8s Secrets
- Monitoring: Grafana dashboards for workspace metrics
- Logging: Promtail → Loki for centralized logs
Integration Test Workflow (Tag-Based)
graph TB
A[Git Tag: run-integration-tests] --> B[GitHub Webhook to Conductor]
B --> C[Conductor Detects Tag Push]
C --> D[Clone Repo at Tag]
D --> E[Read .conductor.yaml]
E --> F[Create Test Namespace]
F --> G[Build Test Image with Kaniko]
G --> H[Push to Harbor Registry]
H --> I[Deploy QA Stack: docker-compose.qa.yml]
I --> J[Convert to K8s Manifests]
J --> K[Deploy Redis]
J --> L[Deploy Backend]
J --> M[Deploy Mailhog]
J --> N[Deploy Frontend]
K --> O[Create K8s Services]
L --> O
M --> O
N --> O
O --> P[Create Test Job]
P --> Q[Run: yarn test:integration:nc]
Q --> R[Tests Connect to Backend via K8s DNS]
R --> S[Tests Connect to Redis via K8s DNS]
S --> T[Tests Verify Emails via Mailhog]
T --> U{Tests Pass?}
U -->|Yes| V[Capture Logs & Metrics]
U -->|No| V
V --> W[Store Results in Prometheus]
W --> X[Create Grafana Dashboard]
X --> Y[Post Results to GitHub Issue]
Y --> Z[Keep QA Stack Running --no-cleanup]
style A fill:#c8e6c9
style G fill:#fff3e0
style H fill:#fff3e0
style Q fill:#e1f5ff
style U fill:#ffecb3
style V fill:#f3e5f5
style Y fill:#c8e6c9
style Z fill:#ffcdd2
Test Execution Details
- Trigger: Git tag
run-integration-tests
- Build: Kaniko builds test image in K8s (no Docker daemon)
- Registry: Harbor (self-hosted) or Docker Hub
- QA Stack: Converted from docker-compose.qa.yml
- Networking: K8s DNS (backend:3001, redis:6379)
- No Cleanup: Stack stays running for inspection
- Results: Posted to GitHub issue + Grafana dashboard
Registry Architecture (Harbor + Docker Hub)
graph LR
A[Conductor] --> B{Registry Type?}
B -->|Harbor| C[Harbor Registry]
B -->|Docker Hub| D[Docker Hub]
C --> E[harbor.conductor.digitalservic.es/conductor/image:tag]
D --> F[w3evolutionsllc/image:tag]
G[Kaniko Build Job] --> H[Read Registry Config]
H --> I[Get Image URL]
I --> J[Build & Push]
J --> C
J --> D
K[Test Job] --> L[Pull Image]
L --> C
L --> D
C --> M[Web UI: https://harbor.conductor.digitalservic.es]
C --> N[Trivy Vulnerability Scanning]
C --> O[Prometheus Metrics]
style A fill:#e1f5ff
style B fill:#ffecb3
style C fill:#c8e6c9
style D fill:#bbdefb
style M fill:#f3e5f5
style N fill:#fff3e0
style O fill:#ffe0b2
Registry Switching
- Environment Variable:
REGISTRY_TYPE=harbor or dockerhub
- Harbor: Self-hosted, vulnerability scanning, RBAC
- Docker Hub: External, simple, w3evolutionsllc org
- Abstraction:
registry_config.py handles all differences
- Switch Time: Update secret + restart Conductor (30 seconds)
Secret Management Flow (ARC-Based)
graph TB
A[GitHub Secrets] --> B[.conductor.yaml defines needed secrets]
B --> C[Conductor Triggers Workflow]
C --> D[GitHub Actions Workflow Dispatch]
D --> E[ARC Self-Hosted Runner in K8s]
E --> F[Runner Has kubectl Access]
F --> G[Read .conductor.yaml]
G --> H[Resolve Secrets from GitHub]
H --> I{Environment?}
I -->|dev| J[Use _DEV suffix secrets]
I -->|qa| K[Use _QA suffix secrets]
I -->|prod| L[Use _PROD suffix secrets]
J --> M[Create K8s Secret]
K --> M
L --> M
M --> N[Mount to Coder Workspace]
M --> O[Mount to Test Job]
M --> P[Mount to QA Stack Pods]
style A fill:#c8e6c9
style E fill:#e1f5ff
style F fill:#fff3e0
style I fill:#ffecb3
style M fill:#f3e5f5
Secret Resolution
- Source of Truth: GitHub Secrets
- Environment-Specific: OPENAI_API_KEY_DEV, OPENAI_API_KEY_QA, etc.
- Conductor Never Handles Secrets: Only orchestrates workflow
- ARC Runner: Runs in K8s, has kubectl access
- Workflow: .github/workflows/conductor-sync-secrets.yml
Complete End-to-End Flow
graph TB
subgraph "Phase 1: Development"
A1[Issue Created] --> A2[AI Requests Dev Env]
A2 --> A3[Conductor Provisions Coder]
A3 --> A4[AI Develops & Tests]
A4 --> A5[AI Tags: run-integration-tests]
end
subgraph "Phase 2: Build"
A5 --> B1[Webhook to Conductor]
B1 --> B2[Clone Repo at Tag]
B2 --> B3[Kaniko Builds Test Image]
B3 --> B4[Push to Harbor]
end
subgraph "Phase 3: Deploy QA"
B4 --> C1[Convert docker-compose.qa.yml]
C1 --> C2[Deploy to K8s]
C2 --> C3[Redis + Backend + Mailhog + Frontend]
end
subgraph "Phase 4: Test"
C3 --> D1[Create Test Job]
D1 --> D2[Run Integration Tests]
D2 --> D3[Tests Use K8s DNS]
D3 --> D4[Capture Results]
end
subgraph "Phase 5: Report"
D4 --> E1[Store in Prometheus]
E1 --> E2[Create Grafana Dashboard]
E2 --> E3[Post to GitHub Issue]
E3 --> E4[Keep Stack Running]
end
style A1 fill:#e1f5ff
style A5 fill:#c8e6c9
style B3 fill:#fff3e0
style C2 fill:#e1f5ff
style D2 fill:#ffecb3
style E3 fill:#c8e6c9
style E4 fill:#ffcdd2