🎯 Conductor Workflow Diagrams

AI-Driven Development & Integration Test Orchestration

Development Workflow (Coder Workspace)

graph TB A[AI Agent Picks GitHub Issue] --> B[AI Requests Dev Environment] B --> C[Conductor Receives Request] C --> D[Trigger Secret Sync Workflow] D --> E[ARC Runner Syncs Secrets] E --> F[Create K8s Namespace] F --> G[Provision Coder Workspace] G --> H[Deploy Dev Pod with Pre-built Image] H --> I[Attach Monitoring: Grafana + Promtail] I --> J[Attach Logging: Loki] J --> K[Mount Secrets to Workspace] K --> L[AI Works in Workspace] L --> M[AI Writes Code & Tests] M --> N[AI Runs Unit Tests] N --> O[AI Commits to Feature Branch] O --> P[AI Tags: run-integration-tests] P --> Q[Push Tag to GitHub] style A fill:#e1f5ff style L fill:#fff3e0 style M fill:#fff3e0 style N fill:#fff3e0 style O fill:#fff3e0 style P fill:#c8e6c9 style Q fill:#c8e6c9

Key Components

  • Coder Workspace: Full IDE environment in Kubernetes
  • Dev Pod: Pre-built development image with tools
  • Secret Sync: GitHub Secrets → ARC → K8s Secrets
  • Monitoring: Grafana dashboards for workspace metrics
  • Logging: Promtail → Loki for centralized logs

Integration Test Workflow (Tag-Based)

graph TB A[Git Tag: run-integration-tests] --> B[GitHub Webhook to Conductor] B --> C[Conductor Detects Tag Push] C --> D[Clone Repo at Tag] D --> E[Read .conductor.yaml] E --> F[Create Test Namespace] F --> G[Build Test Image with Kaniko] G --> H[Push to Harbor Registry] H --> I[Deploy QA Stack: docker-compose.qa.yml] I --> J[Convert to K8s Manifests] J --> K[Deploy Redis] J --> L[Deploy Backend] J --> M[Deploy Mailhog] J --> N[Deploy Frontend] K --> O[Create K8s Services] L --> O M --> O N --> O O --> P[Create Test Job] P --> Q[Run: yarn test:integration:nc] Q --> R[Tests Connect to Backend via K8s DNS] R --> S[Tests Connect to Redis via K8s DNS] S --> T[Tests Verify Emails via Mailhog] T --> U{Tests Pass?} U -->|Yes| V[Capture Logs & Metrics] U -->|No| V V --> W[Store Results in Prometheus] W --> X[Create Grafana Dashboard] X --> Y[Post Results to GitHub Issue] Y --> Z[Keep QA Stack Running --no-cleanup] style A fill:#c8e6c9 style G fill:#fff3e0 style H fill:#fff3e0 style Q fill:#e1f5ff style U fill:#ffecb3 style V fill:#f3e5f5 style Y fill:#c8e6c9 style Z fill:#ffcdd2

Test Execution Details

  • Trigger: Git tag run-integration-tests
  • Build: Kaniko builds test image in K8s (no Docker daemon)
  • Registry: Harbor (self-hosted) or Docker Hub
  • QA Stack: Converted from docker-compose.qa.yml
  • Networking: K8s DNS (backend:3001, redis:6379)
  • No Cleanup: Stack stays running for inspection
  • Results: Posted to GitHub issue + Grafana dashboard

Registry Architecture (Harbor + Docker Hub)

graph LR A[Conductor] --> B{Registry Type?} B -->|Harbor| C[Harbor Registry] B -->|Docker Hub| D[Docker Hub] C --> E[harbor.conductor.digitalservic.es/conductor/image:tag] D --> F[w3evolutionsllc/image:tag] G[Kaniko Build Job] --> H[Read Registry Config] H --> I[Get Image URL] I --> J[Build & Push] J --> C J --> D K[Test Job] --> L[Pull Image] L --> C L --> D C --> M[Web UI: https://harbor.conductor.digitalservic.es] C --> N[Trivy Vulnerability Scanning] C --> O[Prometheus Metrics] style A fill:#e1f5ff style B fill:#ffecb3 style C fill:#c8e6c9 style D fill:#bbdefb style M fill:#f3e5f5 style N fill:#fff3e0 style O fill:#ffe0b2

Registry Switching

  • Environment Variable: REGISTRY_TYPE=harbor or dockerhub
  • Harbor: Self-hosted, vulnerability scanning, RBAC
  • Docker Hub: External, simple, w3evolutionsllc org
  • Abstraction: registry_config.py handles all differences
  • Switch Time: Update secret + restart Conductor (30 seconds)

Secret Management Flow (ARC-Based)

graph TB A[GitHub Secrets] --> B[.conductor.yaml defines needed secrets] B --> C[Conductor Triggers Workflow] C --> D[GitHub Actions Workflow Dispatch] D --> E[ARC Self-Hosted Runner in K8s] E --> F[Runner Has kubectl Access] F --> G[Read .conductor.yaml] G --> H[Resolve Secrets from GitHub] H --> I{Environment?} I -->|dev| J[Use _DEV suffix secrets] I -->|qa| K[Use _QA suffix secrets] I -->|prod| L[Use _PROD suffix secrets] J --> M[Create K8s Secret] K --> M L --> M M --> N[Mount to Coder Workspace] M --> O[Mount to Test Job] M --> P[Mount to QA Stack Pods] style A fill:#c8e6c9 style E fill:#e1f5ff style F fill:#fff3e0 style I fill:#ffecb3 style M fill:#f3e5f5

Secret Resolution

  • Source of Truth: GitHub Secrets
  • Environment-Specific: OPENAI_API_KEY_DEV, OPENAI_API_KEY_QA, etc.
  • Conductor Never Handles Secrets: Only orchestrates workflow
  • ARC Runner: Runs in K8s, has kubectl access
  • Workflow: .github/workflows/conductor-sync-secrets.yml

Complete End-to-End Flow

graph TB subgraph "Phase 1: Development" A1[Issue Created] --> A2[AI Requests Dev Env] A2 --> A3[Conductor Provisions Coder] A3 --> A4[AI Develops & Tests] A4 --> A5[AI Tags: run-integration-tests] end subgraph "Phase 2: Build" A5 --> B1[Webhook to Conductor] B1 --> B2[Clone Repo at Tag] B2 --> B3[Kaniko Builds Test Image] B3 --> B4[Push to Harbor] end subgraph "Phase 3: Deploy QA" B4 --> C1[Convert docker-compose.qa.yml] C1 --> C2[Deploy to K8s] C2 --> C3[Redis + Backend + Mailhog + Frontend] end subgraph "Phase 4: Test" C3 --> D1[Create Test Job] D1 --> D2[Run Integration Tests] D2 --> D3[Tests Use K8s DNS] D3 --> D4[Capture Results] end subgraph "Phase 5: Report" D4 --> E1[Store in Prometheus] E1 --> E2[Create Grafana Dashboard] E2 --> E3[Post to GitHub Issue] E3 --> E4[Keep Stack Running] end style A1 fill:#e1f5ff style A5 fill:#c8e6c9 style B3 fill:#fff3e0 style C2 fill:#e1f5ff style D2 fill:#ffecb3 style E3 fill:#c8e6c9 style E4 fill:#ffcdd2