🧪 Automated Model Benchmarking Pipeline SuperAdmin A/B Testing for Voice Agent Models

Priority: High | Type: Feature | Estimated Effort: 8-12 days

Problem

We manually test different STT/TTS/LLM model combinations by changing agent config, making a call, listening to it, and judging quality ourselves. This is:

Slow testing 5 STT × 5 TTS × 3 LLM = 75 combinations manually is impractical
Subjective no consistent scoring across tests
Not tracked no historical data to compare cost vs quality vs latency across combinations
Not repeatable can't re-run the same test scenario to compare fairly

We need an automated benchmarking pipeline that the SuperAdmin can run from the UI to systematically test model combinations on real phone calls, collect metrics, and rank them.

How Competitors / Industry Does This

Before building, understand how the voice AI industry approaches model evaluation:

1. A/B Testing on Live Traffic (Vapi, Bland.ai, Retell)

Platforms like Vapi route a percentage of live calls through different model configs and compare metrics. We can't do this yet (too risky for customer campaigns), but the benchmark pipeline is the first step toward it.

2. Synthetic Evaluation (Hamming.ai, Vocode)

They call a test phone number where another AI answers (simulating a real person), then evaluate the conversation. This removes the need for a real human to pick up every test call. We should support this.

3. LLM-as-Judge (Industry Standard)

After a call, an LLM (GPT-4o, Claude) scores the conversation on multiple dimensions: naturalness, task completion, latency perception, Hindi accuracy, etc. This is how OpenAI, Google, and Anthropic evaluate their own models. We already have an Analysis pipeline we extend it.

4. Metric Collection (Everyone)

Every serious platform tracks: Time-to-First-Token (TTFT), end-to-end latency, STT word error rate, TTS naturalness score, LLM response relevance, cost per minute, and call completion rate.

Our Approach: Combine All Four

We build a "Benchmark Campaign" that the SuperAdmin creates from their UI. It's like a regular campaign but:

It tests multiple model combinations (a "test matrix")
Each combination calls the same phone number(s) with the same prompt
After all calls complete, an LLM judge scores every conversation
Results are stored in a dedicated collection and displayed in a comparison dashboard

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    SuperAdmin UI                             │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  Benchmark Creator                                   │    │
│  │  - Test phone numbers (real or synthetic)            │    │
│  │  - Agent prompt + context                            │    │
│  │  - Model matrix (STT × TTS × LLM combinations)      │    │
│  │  - Evaluation criteria                               │    │
│  │  - [Start Benchmark]                                 │    │
│  └─────────────────────────────────────────────────────┘    │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  Benchmark Results Dashboard                         │    │
│  │  - Comparison table: latency, cost, quality scores   │    │
│  │  - Listen to recordings side-by-side                 │    │
│  │  - LLM judge scores per dimension                    │    │
│  │  - Winner recommendation                             │    │
│  └─────────────────────────────────────────────────────┘    │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                   Backend (client_backend)                    │
│                                                              │
│  POST /api/superadmin/benchmarks         → Create benchmark  │
│  POST /api/superadmin/benchmarks/:id/run → Start execution   │
│  GET  /api/superadmin/benchmarks         → List benchmarks   │
│  GET  /api/superadmin/benchmarks/:id     → Get results       │
│  POST /api/superadmin/benchmarks/:id/judge → Trigger LLM eval│
│                                                              │
│  BenchmarkService:                                           │
│  - Expands model matrix into individual test runs            │
│  - Creates temporary Agent docs per combination              │
│  - Dispatches calls sequentially (one combo at a time)       │
│  - Collects metrics from Langfuse + webhooks                 │
│  - Triggers LLM judge after all calls complete               │
└──────────────────────────┬──────────────────────────────────┘
                           │
              ┌────────────┼────────────────┐
              ▼            ▼                ▼
     ┌──────────────┐ ┌────────┐  ┌──────────────────┐
     │ LiveKit Agent │ │Langfuse│  │ Analysis Service  │
     │ (uses temp    │ │(metrics│  │ (LLM Judge mode)  │
     │  agent config)│ │ traces)│  │                    │
     └──────────────┘ └────────┘  └──────────────────┘

Data Model

New Collection: `Benchmark`

typescript

interface IBenchmark {
  benchmarkUID: string;           // unique ID (nanoid)
  name: string;                   // "Hindi TTS Comparison March 2026"
  status: "draft" | "running" | "completed" | "failed";
  createdBy: string;              // superadmin userUID
  
  // Test Configuration
  testConfig: {
    phoneNumbers: string[];       // numbers to call (1-5 numbers)
    prompt: string;               // agent system prompt
    context: string;              // agent context
    firstMessage: string;         // what agent says first
    maxCallDuration: number;      // auto-hangup (seconds), default 60
    callsPerCombination: number;  // how many times to call each combo (1-3 for statistical significance)
    language: string;             // "hi", "en"
  };

  // Model Matrix each array item is tested in combination
  modelMatrix: {
    stt: Array<{ provider: string; model: string; label?: string }>;
    tts: Array<{ provider: string; model: string; voice: string; label?: string }>;
    llm: Array<{ provider: string; model: string; temperature?: number; label?: string }>;
  };

  // Evaluation Config
  evaluation: {
    useLLMJudge: boolean;
    judgeModel: string;           // "gpt-4o" or "gemini-2.5-flash"
    criteria: string[];           // ["naturalness", "hindi_accuracy", "task_completion", "response_relevance", "latency_perception"]
    customCriteria?: string;      // free-text additional evaluation instructions
  };

  // Computed
  totalCombinations: number;      // stt.length × tts.length × llm.length
  totalCalls: number;             // totalCombinations × callsPerCombination × phoneNumbers.length
  
  // Results (populated after completion)
  summary?: {
    winner: {
      stt: { provider: string; model: string };
      tts: { provider: string; model: string };
      llm: { provider: string; model: string };
    };
    rankedCombinations: Array<{
      rank: number;
      combinationId: string;
      overallScore: number;
      costPerMin: number;
      avgLatency: number;
    }>;
  };

  startedAt?: Date;
  completedAt?: Date;
  createdAt: Date;
  updatedAt: Date;
}

New Collection: `BenchmarkRun`

Each individual call in the benchmark:

typescript

interface IBenchmarkRun {
  runUID: string;                 // unique ID
  benchmarkUID: string;           // parent benchmark
  combinationId: string;          // hash of stt+tts+llm combo for grouping
  combinationIndex: number;       // 0, 1, 2... for ordering
  callIndex: number;              // which repeat (0, 1, 2 if callsPerCombination=3)
  
  // Model Config for this run
  models: {
    stt: { provider: string; model: string };
    tts: { provider: string; model: string; voice: string };
    llm: { provider: string; model: string; temperature?: number };
  };

  // Call Data (populated during/after call)
  phoneNumber: string;
  contactUID: string;             // links to Contact doc
  roomId: string;
  status: "pending" | "calling" | "completed" | "failed";
  
  // Metrics (populated from Langfuse + webhooks)
  metrics: {
    callDuration?: number;        // seconds
    billingDuration?: number;     // seconds (for cost calc)
    
    // Latency metrics (from Langfuse OTel spans)
    sttLatency?: number;          // ms time from audio to transcript
    llmTimeToFirstToken?: number; // ms TTFT
    llmTotalLatency?: number;     // ms full response
    ttsLatency?: number;          // ms time from text to first audio
    endToEndLatency?: number;     // ms user stops speaking → agent starts speaking
    
    // Usage metrics
    llmPromptTokens?: number;
    llmCompletionTokens?: number;
    ttsCharacters?: number;
    sttAudioDuration?: number;    // seconds
    
    // Cost (calculated from provider pricing)
    estimatedCost?: number;       // USD
    costPerMin?: number;          // USD/min
  };

  // Transcription (from post-call sync)
  transcription?: any;
  recordingUrl?: string;

  // LLM Judge Scores (populated after judge runs)
  judgeScores?: {
    overall: number;              // 1-10
    dimensions: Record<string, number>;  // { naturalness: 8, hindi_accuracy: 7, ... }
    reasoning: string;            // judge's explanation
    judgeModel: string;           // which model judged
  };

  startedAt?: Date;
  completedAt?: Date;
  error?: string;
}

Implementation Plan

Phase 1: Backend Benchmark CRUD & Execution Engine (4-5 days)

1.1 Models & Routes

Create:

client_backend/src/models/Benchmark.ts Mongoose schema for IBenchmark
client_backend/src/models/BenchmarkRun.ts Mongoose schema for IBenchmarkRun
client_backend/src/routes/benchmark.routes.ts SuperAdmin-only routes
client_backend/src/controllers/benchmark.controller.ts
client_backend/src/services/benchmark.service.ts Core orchestration logic

Routes (all behind requireRole("superadmin")):

POST   /api/superadmin/benchmarks              → createBenchmark
GET    /api/superadmin/benchmarks              → listBenchmarks
GET    /api/superadmin/benchmarks/:id          → getBenchmark (with runs)
POST   /api/superadmin/benchmarks/:id/start    → startBenchmark
POST   /api/superadmin/benchmarks/:id/stop     → stopBenchmark
POST   /api/superadmin/benchmarks/:id/judge    → triggerLLMJudge
DELETE /api/superadmin/benchmarks/:id          → deleteBenchmark

1.2 Benchmark Execution Service

This is the core. It:

Expands the matrix Given stt: [A, B], tts: [X, Y], llm: [P, Q], generates 8 combinations: (A,X,P), (A,X,Q), (A,Y,P), (A,Y,Q), (B,X,P), (B,X,Q), (B,Y,P), (B,Y,Q)
Creates temporary Agent docs For each combination, creates a temporary Agent document with the specific STT/TTS/LLM config. These are marked with isBenchmark: true so they don't pollute the real agent list.
Runs calls sequentially Unlike regular campaigns that run concurrently, benchmark calls run one combination at a time (or one at a time per phone number). This ensures fair comparison you don't want network congestion from parallel calls affecting latency measurements.
Reuses existing infrastructure Each benchmark call goes through the same createDispatch() → LiveKit Agent → SIP call → webhook pipeline. The Contact documents are created with isTestCall: true (this field already exists in your Contact model).
Collects metrics After each call completes (webhook fires), the service pulls metrics from:
- Contact document (callDuration, billingDuration, transcription)
- Langfuse API (latency spans STT/LLM/TTS timing)
- Calculates cost based on provider pricing
Triggers LLM judge After ALL calls complete, sends each transcription to the judge LLM.

Key design decision: Run calls sequentially, not in parallel. Parallel calls would share network/CPU resources and make latency comparisons unfair. Sequential execution is slower but produces reliable data.

typescript

// Pseudocode for the execution loop
async function executeBenchmark(benchmarkUID: string) {
  const benchmark = await Benchmark.findOne({ benchmarkUID });
  const combinations = expandMatrix(benchmark.modelMatrix);
  
  for (const combo of combinations) {
    for (let callIdx = 0; callIdx < benchmark.testConfig.callsPerCombination; callIdx++) {
      for (const phone of benchmark.testConfig.phoneNumbers) {
        
        // 1. Create temp agent with this combo's config
        const tempAgent = await createTempAgent(benchmark, combo);
        
        // 2. Create Contact doc (isTestCall: true)
        const contact = await createBenchmarkContact(benchmark, phone, tempAgent);
        
        // 3. Create BenchmarkRun doc
        const run = await BenchmarkRun.create({ ... });
        
        // 4. Dispatch call (reuse existing infrastructure)
        await createDispatch({
          mode: "call",
          phone_number: phone,
          agent_id: tempAgent.agent_id,
          // ... standard dispatch metadata
        });
        
        // 5. Wait for call to complete (poll Contact status or use webhook callback)
        await waitForCallCompletion(contact.contactUID, benchmark.testConfig.maxCallDuration + 30);
        
        // 6. Collect metrics
        await collectRunMetrics(run.runUID, contact.contactUID);
      }
    }
  }
  
  // 7. Run LLM judge on all completed runs
  if (benchmark.evaluation.useLLMJudge) {
    await runLLMJudge(benchmarkUID);
  }
  
  // 8. Compute rankings and summary
  await computeBenchmarkSummary(benchmarkUID);
}

1.3 Waiting for Call Completion

The benchmark service needs to know when a call finishes. Two approaches:

Option A (Recommended): Polling Poll the Contact document every 5 seconds until status changes from in_progress to done or failed. Simple, reliable, works with existing webhook pipeline.

typescript

async function waitForCallCompletion(contactUID: string, timeoutSec: number): Promise<void> {
  const deadline = Date.now() + timeoutSec * 1000;
  while (Date.now() < deadline) {
    const contact = await Contact.findOne({ contactUID }).lean();
    if (contact?.status === "done" || contact?.status === "failed") return;
    await sleep(5000);
  }
  throw new Error(`Call timed out after ${timeoutSec}s`);
}

Option B: Redis Pub/Sub Webhook handler publishes to a Redis channel when a benchmark call completes. More efficient but more complex. Use Option A first, optimize later if needed.

1.4 Metric Collection from Langfuse

After a call completes, pull latency metrics from Langfuse using their API:

typescript

async function collectLatencyMetrics(roomName: string): Promise<LatencyMetrics> {
  // Langfuse API: GET /api/public/traces?sessionId={roomName}
  // Each trace contains spans for STT, LLM, TTS with duration_ms
  const traces = await langfuseClient.getTraces({ sessionId: roomName });
  
  // Extract timing from OTel spans
  // LiveKit auto-emits spans named: "stt.recognize", "llm.chat", "tts.synthesize"
  // Each span has start_time, end_time, duration
}

Alternatively, extend the existing post_call.py to write latency metrics directly to the Contact or BenchmarkRun document. This avoids an extra Langfuse API call.

Phase 2: LLM Judge (2 days)

2.1 Judge Prompt Design

The LLM judge evaluates each conversation transcript on configurable dimensions. This is a well-established technique in AI evaluation.

typescript

const JUDGE_SYSTEM_PROMPT = `You are an expert evaluator of AI voice agent conversations. 
You will be given a transcript of a phone call between an AI agent and a human.
Score the conversation on each dimension from 1-10 and provide brief reasoning.

Evaluation dimensions:
{{dimensions}}

Additional context:
- The agent's intended behavior: {{agentPrompt}}
- Language: {{language}}

Respond in JSON format:
{
  "overall": <1-10>,
  "dimensions": { "<dimension_name>": <1-10>, ... },
  "reasoning": "<2-3 sentence explanation>"
}`;

Default evaluation dimensions:

naturalness Does the agent sound like a real person? Natural flow, no robotic patterns?
hindi_accuracy For Hindi calls: correct grammar, natural Hinglish mixing, proper pronunciation cues in transcript?
task_completion Did the agent accomplish its goal (as defined in the prompt)?
response_relevance Are responses on-topic and helpful?
latency_perception Based on transcript timing markers, did the conversation feel responsive?
conversation_flow Smooth turn-taking? No awkward interruptions or long silences?

2.2 Implementation

Extend the existing Analysis service or create a new endpoint:

python

# Analysis/src/pipeline/benchmark_judge.py
class BenchmarkJudge:
    async def evaluate(self, transcription: dict, agent_prompt: str, 
                       criteria: list[str], language: str) -> dict:
        # Build judge prompt with criteria
        # Call OpenAI/Gemini structured output
        # Return scores + reasoning

The judge runs once per BenchmarkRun after all calls complete. Results are stored in BenchmarkRun.judgeScores.

Phase 3: Frontend SuperAdmin Benchmark UI (2-3 days)

3.1 Benchmark Creator Page

Location: oktivo_client/src/pages/superadmin/BenchmarkCreator.jsx

Form sections:

Basic Info: Benchmark name
Test Setup: Phone numbers (input field, comma-separated), calls per combination (1-3), max call duration, language
Agent Config: Prompt textarea, context textarea, first message
Model Matrix:
- STT section: Add/remove rows, each row = provider dropdown + model input
- TTS section: Add/remove rows, each row = provider dropdown + model input + voice input
- LLM section: Add/remove rows, each row = provider dropdown + model input + temperature slider
- Show "Total combinations: X, Total calls: Y" live counter
Evaluation: Toggle LLM judge on/off, select judge model, checkboxes for evaluation criteria
Preview & Start: Show matrix summary, estimated time, start button

3.2 Benchmark Results Page

Location: oktivo_client/src/pages/superadmin/BenchmarkResults.jsx

Sections:

Status bar: Running/Completed, progress (X/Y calls done)
Comparison Table: Rows = combinations, Columns = avg latency, cost/min, judge scores per dimension, overall score. Sortable by any column. Highlight the winner row.
Detail View (click a row): Show individual call recordings, transcriptions, per-call metrics, judge reasoning
Side-by-side Player: Select 2 combinations, play their recordings simultaneously to hear the difference

Add to superadmin sidebar:

"Benchmarks" menu item → lists all benchmarks
"New Benchmark" button → opens creator

Cost Estimation & Pricing

The benchmark system needs to estimate costs before running. Build a simple pricing table:

typescript

// client_backend/src/config/modelPricing.ts
const MODEL_PRICING = {
  stt: {
    "deepgram/nova-3": { perMinute: 0.0043 },
    "openai/whisper-large-v3": { perMinute: 0.006 },
    "assemblyai/universal-3": { perMinute: 0.01 },
    "huggingface/*": { perMinute: 0.0 },  // free tier
  },
  tts: {
    "cartesia/sonic-3": { per1kChars: 0.06 },
    "elevenlabs/eleven_turbo_v2_5": { per1kChars: 0.18 },
    "deepgram/aura-2": { per1kChars: 0.015 },
    "huggingface/*": { per1kChars: 0.0 },
  },
  llm: {
    "google/gemini-2.5-flash": { perMillionInputTokens: 0.15, perMillionOutputTokens: 0.60 },
    "openai/gpt-4o-mini": { perMillionInputTokens: 0.15, perMillionOutputTokens: 0.60 },
    "huggingface/*": { perMillionInputTokens: 0.0, perMillionOutputTokens: 0.0 },
    "groq/llama-3.3-70b": { perMillionInputTokens: 0.0, perMillionOutputTokens: 0.0 },
  }
};

Show estimated cost before starting: "This benchmark will make ~24 calls, estimated cost: $X.XX"

What NOT to Build (Keep Scope Tight)

❌ Don't build synthetic callee (AI answering the phone) use real test numbers for now. This is a future enhancement.
❌ Don't build automatic A/B routing on live campaigns that's a separate feature.
❌ Don't build a custom Langfuse dashboard use the existing Langfuse UI for deep metric analysis. Our UI just shows the summary.
❌ Don't modify the existing campaign system benchmarks are a separate flow that reuses the same underlying call infrastructure.
❌ Don't build real-time streaming of benchmark progress simple polling (refresh page or auto-refresh every 10s) is fine.

File Structure

client_backend/
├── src/
│   ├── models/
│   │   ├── Benchmark.ts              # NEW
│   │   └── BenchmarkRun.ts           # NEW
│   ├── routes/
│   │   └── benchmark.routes.ts       # NEW (mounted under /api/superadmin/)
│   ├── controllers/
│   │   └── benchmark.controller.ts   # NEW
│   ├── services/
│   │   └── benchmark.service.ts      # NEW core orchestration
│   └── config/
│       └── modelPricing.ts           # NEW cost estimation

Analysis/
├── src/
│   └── pipeline/
│       └── benchmark_judge.py        # NEW LLM-as-judge evaluator

oktivo_client/
├── src/
│   └── pages/
│       └── superadmin/
│           ├── BenchmarkCreator.jsx   # NEW
│           ├── BenchmarkResults.jsx   # NEW
│           └── BenchmarkList.jsx      # NEW

Acceptance Criteria

[ ] SuperAdmin can create a benchmark with model matrix from the UI
[ ] Benchmark expands matrix into combinations and shows total call count
[ ] Benchmark executes calls sequentially, one combination at a time
[ ] Each call reuses existing LiveKit dispatch → Agent → SIP → webhook pipeline
[ ] Contacts created with isTestCall: true
[ ] Metrics collected after each call: duration, latency (if available), cost estimate
[ ] LLM judge scores each conversation on configurable dimensions
[ ] Results page shows comparison table sortable by any metric
[ ] Recordings are accessible from results page
[ ] Benchmark doesn't consume customer credits (SuperAdmin-only, no billing)
[ ] Benchmark calls don't count toward org concurrent call limits
[ ] Temporary agent configs are cleaned up after benchmark completes

Future Enhancements (Not in This Issue)

Synthetic Callee: Deploy an AI that answers test calls and follows a script (simulates a real customer). This removes the need for a human to answer test calls.
Scheduled Benchmarks: Run the same benchmark weekly to track model improvements over time.
Auto A/B on Live Traffic: Route X% of real campaign calls through a different model combo and compare.
Statistical Significance: Run enough calls per combo to compute confidence intervals on metrics.
Cost Optimizer: Given a quality threshold, automatically recommend the cheapest model combo that meets it.

🧪 Automated Model Benchmarking Pipeline SuperAdmin A/B Testing for Voice Agent Models ​

Priority: High | Type: Feature | Estimated Effort: 8-12 days ​

Problem ​

How Competitors / Industry Does This ​

1. A/B Testing on Live Traffic (Vapi, Bland.ai, Retell) ​

2. Synthetic Evaluation (Hamming.ai, Vocode) ​

3. LLM-as-Judge (Industry Standard) ​

4. Metric Collection (Everyone) ​

Our Approach: Combine All Four ​

Architecture Overview ​

Data Model ​

New Collection: Benchmark ​

New Collection: BenchmarkRun ​

Implementation Plan ​

Phase 1: Backend Benchmark CRUD & Execution Engine (4-5 days) ​

1.1 Models & Routes ​

1.2 Benchmark Execution Service ​

1.3 Waiting for Call Completion ​

1.4 Metric Collection from Langfuse ​

Phase 2: LLM Judge (2 days) ​

2.1 Judge Prompt Design ​

2.2 Implementation ​

Phase 3: Frontend SuperAdmin Benchmark UI (2-3 days) ​

3.1 Benchmark Creator Page ​

3.2 Benchmark Results Page ​

3.3 Navigation ​

Cost Estimation & Pricing ​

What NOT to Build (Keep Scope Tight) ​

File Structure ​

Acceptance Criteria ​

Future Enhancements (Not in This Issue) ​

🧪 Automated Model Benchmarking Pipeline SuperAdmin A/B Testing for Voice Agent Models

Priority: High | Type: Feature | Estimated Effort: 8-12 days

Problem

How Competitors / Industry Does This

1. A/B Testing on Live Traffic (Vapi, Bland.ai, Retell)

2. Synthetic Evaluation (Hamming.ai, Vocode)

3. LLM-as-Judge (Industry Standard)

4. Metric Collection (Everyone)

Our Approach: Combine All Four

Architecture Overview

Data Model

New Collection: `Benchmark`

New Collection: `BenchmarkRun`

Implementation Plan

Phase 1: Backend Benchmark CRUD & Execution Engine (4-5 days)

1.1 Models & Routes

1.2 Benchmark Execution Service

1.3 Waiting for Call Completion

1.4 Metric Collection from Langfuse

Phase 2: LLM Judge (2 days)

2.1 Judge Prompt Design

2.2 Implementation

Phase 3: Frontend SuperAdmin Benchmark UI (2-3 days)

3.1 Benchmark Creator Page

3.2 Benchmark Results Page

3.3 Navigation

Cost Estimation & Pricing

What NOT to Build (Keep Scope Tight)

File Structure

Acceptance Criteria

Future Enhancements (Not in This Issue)