Benchmark Pipeline Automated Model A/B Testing

Automated system for testing STT/TTS/LLM model combinations on real phone calls, collecting metrics, and ranking them using an LLM judge. Available to customers (feature-gated via benchmark feature) and superadmins.

---

Table of Contents

1. Overview
2. Architecture
3. Execution Flow
4. Multi-Provider Support
5. Agent Provider Registry
6. HuggingFace Integration
7. API Endpoints
8. Database Collections
9. LLM Judge
10. Frontend UI
11. Configuration & Environment
12. Known Limitations

---

Overview

The benchmark pipeline lets a SuperAdmin define a "model matrix" (e.g., 2 STT × 3 TTS × 2 LLM = 12 combinations), then automatically:

1. Expands the matrix into all combinations
2. Creates a temporary Agent document per combination
3. Dispatches real phone calls sequentially (one at a time for fair latency comparison)
4. Collects call metrics (duration, cost, transcription)
5. Runs an LLM-as-Judge to score each conversation on configurable dimensions
6. Ranks combinations and picks a winner

Calls run through the existing LiveKit dispatch → Agent → SIP → webhook pipeline. No separate call infrastructure is needed.

---

Architecture

---

Execution Flow

Step-by-step

1. Create SuperAdmin fills out the form (name, phone numbers, prompt, model matrix, evaluation criteria). Backend validates the matrix against the model registry and saves a Benchmark doc with status draft.

2. Start Backend sets status to running and kicks off executeBenchmark() as a fire-and-forget background task. The HTTP response returns immediately.

3. Matrix expansion expandMatrix() generates all STT × TTS × LLM combinations. Each gets a deterministic ID (MD5 hash of its config, truncated to 12 chars).

4. Temp Agent creation For each combination, a temporary Agent document is created with the specific provider/model config. Agent IDs follow the pattern bmk_{benchmarkUID}_{comboId}_{nanoid}.

5. Call dispatch Uses the existing createDispatch() from livekit.service.ts. The call goes through the standard LiveKit → SIP → phone pipeline. The Agent process reads the temp Agent doc and builds the session with the specified providers.

6. Wait for completion Polls the Contact document every 5 seconds until status becomes done or failed, or until timeout (maxCallDuration + 30s).

7. Metric collection After each call, reads callDuration, billingDuration, transcription, and recordingUrl from the Contact doc. Calculates estimated cost from billing duration × agent cost_per_min.

8. LLM Judge If enabled, sends each completed run's transcription to the Analysis service's /analyse/benchmark-judge endpoint. The judge scores on configurable dimensions (1-10 scale) using structured JSON output.

9. Summary Groups runs by combination, averages metrics and judge scores, sorts by overall score (descending) then cost (ascending), assigns ranks.

10. Cleanup Temp agents are deleted after all calls for that combination finish. On benchmark stop/delete, all temp agents and contacts are cleaned up via regex match on the bmk_{benchmarkUID}_ prefix.

---

Multi-Provider Support

STT Providers (6)

Provider	Key	Streaming	Hindi	Notes
Deepgram	deepgram	Yes	Good	Default fallback. Models: nova-3, nova-2
OpenAI	openai	Yes	Good	Models: whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe
Sarvam	sarvam	Yes	Native	Indian language specialist. Models: saaras:v3, saaras:v2
Groq	groq	Yes	Good	Ultra-fast Whisper on Groq hardware. Uses OpenAI-compatible API
HuggingFace	huggingface	No*	Good	Uses InferenceClient. Wrapped in StreamAdapter + Silero VAD
Google	google	Yes	Native	Requires GCP service account (GOOGLE_APPLICATION_CREDENTIALS)

*HuggingFace STT is non-streaming. It's wrapped in LiveKit's StreamAdapter with Silero VAD to buffer complete speech segments before sending to the API.

TTS Providers (7)

Provider	Key	Streaming	Hindi	Notes
Cartesia	cartesia	Yes	Good	Default fallback. Low-latency streaming. Models: sonic-2, sonic-turbo
ElevenLabs	elevenlabs	Yes	Good	Natural voices. Models: eleven_turbo_v2_5, eleven_multilingual_v2
Murf	murf	Yes	Native	Indian voice specialist. Model: Falcon
Deepgram	deepgram	Yes	Limited	Aura TTS. Models: aura-2-en, aura-2-hi
OpenAI	openai	Yes	Basic	Models: tts-1, tts-1-hd, gpt-4o-mini-tts
HuggingFace	huggingface	No	English only*	Uses InferenceClient. Models: hexgrad/Kokoro-82M, canopylabs/orpheus-3b-0.1-ft
Google	google	No*	Native	Requires GCP service account. Wavenet + Neural2 voices. Uses custom direct Cloud TTS API (not LiveKit plugin)

*No Hindi TTS models are currently available on HuggingFace Inference Providers. Use Cartesia, ElevenLabs, or Murf for Hindi TTS.

*Google Cloud TTS uses a custom non-streaming provider that calls the Cloud TTS API directly. LiveKit handles non-streaming TTS natively via its pipeline.

LLM Providers (7)

Provider	Key	Hindi	Notes
OpenAI	openai	Good	Default fallback. Models: gpt-4o-mini, gpt-4o, gpt-4.1-*
Google	google	Good	Uses google.LLM() plugin. Models: gemini-2.5-flash, gemini-2.5-pro
Groq	groq	Basic-Good	Ultra-low latency. OpenAI-compatible API
HuggingFace	huggingface	Good	OpenAI-compatible via router.huggingface.co/v1
DeepSeek	deepseek	Good	OpenAI-compatible. Models: deepseek-chat, deepseek-reasoner
Together AI	together	Good	OpenAI-compatible. Fast inference for open-source models
OpenRouter	openrouter	Good	Gateway to 200+ models. OpenAI-compatible

All LLM providers except Google use the OpenAI-compatible API pattern same LiveKit openai.LLM() plugin with different base_url and api_key.

---

Agent Provider Registry

The Agent service uses a registry pattern for all providers. Each category (STT, TTS, LLM) has:

1. A base class (BaseSTT, BaseTTS, or constructor pattern for LLM)
2. Provider classes that implement from_meta(AgentMetaSchema) → instance and build() → LiveKit plugin
3. A registry dict mapping provider name strings to classes
4. A factory function (build_stt(), build_tts(), build_llm()) that looks up the registry and falls back to a default if the provider is unknown

Fallback behavior

If a provider string doesn't match any registry key:

• STT falls back to Deepgram
• TTS falls back to Cartesia
• LLM falls back to OpenAI

A warning is logged but the call proceeds. This means benchmark runs with typos in provider names won't crash they'll silently use the default.

Adding a new provider

1. Create a class following the base pattern in the appropriate *_providers.py
2. Add it to the *_REGISTRY dict
3. Add the provider + models to modelRegistry.ts in client_backend
4. Add any required env vars to the Agent's .env

---

HuggingFace Integration

HuggingFace is unique because it has three different integration patterns:

LLM OpenAI-compatible (simple)

# Uses LiveKit's openai.LLM() with HF's OpenAI-compatible endpoint
openai.LLM(
    model="meta-llama/Llama-3.3-70B-Instruct",
    base_url="https://router.huggingface.co/v1",
    api_key=os.getenv("HUGGINGFACE_API_KEY"),
)

STT Custom plugin with InferenceClient

HuggingFaceSTT (custom stt.STT subclass)
  → wrapped in StreamAdapter + Silero VAD
  → VAD detects end-of-speech
  → _recognize_impl() called with AudioBuffer
  → converts to WAV bytes via rtc.combine_audio_frames().to_wav_bytes()
  → sends to InferenceClient.automatic_speech_recognition()
  → returns SpeechEvent with transcribed text

TTS Custom plugin with InferenceClient

HuggingFaceTTS (custom tts.TTS subclass)
  → synthesize() creates _HFChunkedStream
  → _run() splits text into sentence chunks
  → each chunk sent to InferenceClient.text_to_speech()
  → audio bytes decoded from WAV to raw PCM
  → pushed to AudioEmitter for LiveKit playback

Why InferenceClient instead of raw HTTP?

HuggingFace deprecated api-inference.huggingface.co (returns 410 Gone). The new router.huggingface.co/hf-inference/models endpoint has zero TTS models and only 2 ASR models. The huggingface_hub.InferenceClient automatically routes to the best available inference provider (fal-ai, replicate, etc.), solving the routing problem.

Model validation

The validateHuggingFaceModel() function in modelRegistry.ts checks the HF Hub API:

GET https://huggingface.co/api/models/{model}?expand=inferenceProviderMapping

This returns which inference providers support the model and for which task. The frontend uses this to validate custom model IDs before adding them to the benchmark matrix.

Requires HUGGINGFACE_API_KEY in client_backend/.env for authenticated access (some models are gated).

---

API Endpoints

All routes are mounted at /api/superadmin/benchmarks and require verifyJwt + requireRole("superadmin").

Method	Path	Handler	Description
POST	/	create	Create a benchmark (validates matrix, resolves SIP config)
GET	/	list	List benchmarks with pagination and progress
GET	/registry	registry	Get the full model registry (STT/TTS/LLM providers + models)
POST	/validate-model	validateModel	Validate a HuggingFace model ID against the Hub API
GET	/:id	getOne	Get benchmark details + all runs
PUT	/:id	update	Update a draft/stopped benchmark
POST	/:id/start	start	Start benchmark execution (fire-and-forget)
POST	/:id/stop	stop	Stop a running benchmark
POST	/:id/judge	judge	Trigger LLM judge evaluation (fire-and-forget)
DELETE	/:id	remove	Delete benchmark + all runs, temp agents, contacts

Create request body

{
  "name": "Hindi TTS Comparison",
  "testConfig": {
    "phoneNumbers": ["+91XXXXXXXXXX"],
    "prompt": "You are a helpful real estate agent...",
    "context": "Property details...",
    "firstMessage": "Hello",
    "maxCallDuration": 60,
    "callsPerCombination": 1,
    "language": "hi"
  },
  "modelMatrix": {
    "stt": [{ "provider": "deepgram", "model": "nova-3" }],
    "tts": [
      { "provider": "cartesia", "model": "sonic-2", "voice": "f786b574-..." },
      { "provider": "elevenlabs", "model": "eleven_turbo_v2_5", "voice": "H8bdWZHK2OgZwTN7ponr" }
    ],
    "llm": [{ "provider": "openai", "model": "gpt-4o-mini" }]
  },
  "evaluation": {
    "useLLMJudge": true,
    "judgeModel": "gpt-4o",
    "criteria": ["naturalness", "task_completion", "response_relevance"]
  },
  "sipConfig": {
    "trunkUID": "trunk_xxx",
    "phoneNumber": "+1XXXXXXXXXX"
  }
}

Validation rules

• At least 1 model per category (STT, TTS, LLM)
• 1-5 phone numbers
• 1-3 calls per combination
• Max 100 total calls per benchmark
• Provider/model validated against the registry (unknown providers produce warnings, not errors)

---

Database Collections

Benchmark

Stores the benchmark configuration and results summary.

Field	Type	Description
benchmarkUID	string	Unique ID (auto-generated, prefix bmk_)
name	string	Human-readable name
status	enum	draft → running → completed / failed / stopped
createdBy	string	SuperAdmin userUID
testConfig	object	Phone numbers, prompt, context, language, duration, etc.
modelMatrix	object	Arrays of STT, TTS, LLM configs
evaluation	object	Judge settings (model, criteria, custom instructions)
totalCombinations	number	STT × TTS × LLM count
totalCalls	number	combinations × callsPerCombination × phoneNumbers
sipConfig	object	Resolved SIP trunk ID, phone number, provider
summary	object	Winner + ranked combinations (populated after completion)
startedAt	Date	When execution began
completedAt	Date	When execution finished
error	string	Error message if failed

BenchmarkRun

One document per individual call in the benchmark.

Field	Type	Description
runUID	string	Unique ID (prefix run_)
benchmarkUID	string	Parent benchmark
combinationId	string	MD5 hash of STT+TTS+LLM config (12 chars)
combinationIndex	number	Order index for display
callIndex	number	Which repeat (0-based)
models	object	{ stt, tts, llm } config for this run
phoneNumber	string	Which number was called
contactUID	string	Links to Contact doc
agentId	string	Temp agent ID
roomId	string	LiveKit room name
status	enum	pending → calling → completed / failed
metrics	object	callDuration, billingDuration, estimatedCost, costPerMin
transcription	mixed	Call transcript from Contact doc
recordingUrl	string	Call recording URL
judgeScores	object	{ overall, dimensions, reasoning, judgeModel }

---

LLM Judge

The judge evaluates call transcriptions using OpenAI's structured JSON output. Located in Analysis/src/pipeline/benchmark_judge.py.

Evaluation dimensions

Dimension	Description
naturalness	How human-like does the agent sound?
hindi_accuracy	Hindi grammar, vocabulary, transliteration quality
task_completion	Did the agent accomplish its goal?
response_relevance	Are responses on-topic and contextually aware?
latency_perception	Does the conversation feel responsive?
conversation_flow	Smooth turn-taking, no awkward pauses?

Scoring

• Scale: 1-10 per dimension
• 1-3: Significant problems
• 4-5: Below average
• 6-7: Good (most average conversations)
• 8-9: Excellent
• 10: Near-perfect

The overall score is a weighted average. The judge also provides 2-4 sentences of reasoning.

Custom criteria

SuperAdmins can add free-text evaluation instructions (e.g., "Pay special attention to how the agent handles objections about pricing"). These are appended to the judge's system prompt.

Flow

The judge runs sequentially over all completed runs to avoid overwhelming the Analysis service. After all runs are judged, the benchmark summary is recomputed with judge scores factored into the ranking.

---

Frontend UI

Three pages under the SuperAdmin section:

BenchmarkCreator (BenchmarkCreator.jsx)

Multi-section form:

• Basic info: Name
• Test setup: Phone numbers, calls per combination, max duration, language
• Agent config: Prompt, context, first message
• Model matrix: Add/remove STT, TTS, LLM entries with provider dropdowns and model selectors
  • HuggingFace models support a "Custom" option with a text input + validate button
  • Live counter shows total combinations and total calls
• Evaluation: Toggle LLM judge, select judge model, pick criteria
• SIP config: Trunk and phone number selection

BenchmarkList (BenchmarkList.jsx)

Table of all benchmarks with:

• Name, status, total combinations, total calls
• Progress bar (completed/total runs)
• Actions: View, Start, Stop, Delete

BenchmarkResults (BenchmarkResults.jsx)

Detailed view of a completed benchmark:

• Status bar with progress
• Ranked comparison table (sortable by score, duration, cost)
• Per-combination detail: individual call metrics, transcriptions, judge reasoning
• Winner highlight

Frontend service (benchmark.service.ts)

Axios-based service with typed methods for all API endpoints. Uses the shared api instance from ../lib/axios.

---

Configuration & Environment

Agent service (Agent/.env)

HUGGINGFACE_API_KEY=hf_xxxxx          # Required for HuggingFace STT/TTS/LLM
GROQ_API_KEY=gsk_xxxxx                # Required for Groq STT/LLM
DEEPSEEK_API_KEY=sk-xxxxx             # Required for DeepSeek LLM
TOGETHER_API_KEY=xxxxx                # Required for Together AI LLM
OPENROUTER_API_KEY=sk-or-xxxxx        # Required for OpenRouter LLM
GOOGLE_API_KEY=xxxxx                  # For Google Gemini LLM (not Cloud STT/TTS)
GOOGLE_APPLICATION_CREDENTIALS=/app/google-credentials.json  # Required for Google Cloud STT/TTS

client_backend (client_backend/.env)

HUGGINGFACE_API_KEY=hf_xxxxx          # Required for HF model validation endpoint
ANALYSIS_URL=http://cb-analysis:8000  # Analysis service URL (for LLM judge)

Docker

• Dev mode uses volume mounts (./Agent:/app), so code changes are picked up on container restart without rebuild
• Production builds need --no-cache flag when files change to bust cached layers

---

Known Limitations

1. No streaming for HuggingFace STT/TTS HF Inference API is request/response only. Latency will be higher than streaming providers (Deepgram, Cartesia). This is inherent to the API, not a bug.

2. No Hindi TTS on HuggingFace As of the current HF Inference Provider landscape, no Hindi TTS models are available. Use Cartesia, ElevenLabs, or Murf for Hindi TTS benchmarks.

3. Google Cloud TTS is non-streaming Google Cloud TTS uses a custom direct API integration (google_cloud_tts.py) instead of the LiveKit google plugin, because the plugin (v1.5.0) defaults to Gemini TTS which requires a GOOGLE_API_KEY. Our setup uses service account credentials (GOOGLE_APPLICATION_CREDENTIALS), which only works with Cloud TTS. The custom provider is non-streaming (single request/response), so latency will be slightly higher than streaming providers like Cartesia.

4. No real-time progress streaming Benchmark progress is fetched by polling the API. The frontend auto-refreshes but there's no WebSocket push.

5. Sequential execution only Calls run one at a time for fair latency comparison. A 12-combination benchmark with 2 calls each at 60s max = ~30 minutes minimum.

6. No cost estimation before running The model pricing table from the original spec was not implemented. Cost is calculated post-call from billing duration × agent cost_per_min.

7. No resume on failure If the benchmark process crashes mid-execution, completed runs are preserved but remaining runs must be re-triggered by stopping and re-starting the benchmark (which clears all runs).

8. Adaptive Interruption 401 LiveKit Agents v1.5.0 auto-enables AdaptiveInterruptionDetector which tries to connect to wss://agent-gateway.livekit.cloud/v1/bargein. On self-hosted setups, this fails with 401. The SDK self-heals after 3 retries. No public API to disable in v1.5.0.

9. No rate limiting on benchmark creation Any SuperAdmin can create unlimited benchmarks. Consider adding rate limits for production.

10. LLM judge has no timeout If the OpenAI API hangs, the judge call will block indefinitely. The judge retries up to 2 times on failure but doesn't enforce a per-call timeout.