Provider Parameter Validation Tests

Purpose

These integration tests verify that every parameter combination we expose in the UI or hardcode in provider functions is actually accepted by the LiveKit plugin constructors. They catch config mismatches before they reach production avoiding runtime crashes during live calls.

The Problem They Solve

There are two paths parameters take to reach LiveKit plugins:

Path	Source of Truth	How Params Flow
Path 1: BYOK Custom Preset	app/src/config/byokPresetConfig.ts	Frontend config → stored on agent → custom_preset_builder.py → _prepare_kwargs() → plugin(**kwargs)
Path 2: Normal Preset	Provider functions in stt_providers.py, tts_providers.py, llm_providers.py	SystemConfig preset → preset.resolver.ts → AgentMetaSchema → provider function → hardcoded kwargs → plugin(**kwargs)

If a param is defined in the config but the plugin doesn't support it, or if a hardcoded kwarg in a provider function uses a name the plugin doesn't recognize, the call crashes at dispatch time with a TypeError or ValueError.

Test Files

tests/test_provider_params_integration.py Path 1 (BYOK)

Tests every provider/model/param combination from byokPresetConfig.ts by passing them through _prepare_kwargs() (the same transform used in production) and verifying the LiveKit plugin constructor accepts them.

What it catches:

• Wrong param names (e.g., endpoint_sensitivity not in Soniox STTOptions)
• Params that need wrapping (e.g., ElevenLabs stability must be in VoiceSettings)
• Wrong API key param naming (api_key vs key vs api_token)
• Model-specific feature support (e.g., keyterm is Nova-3 only)

Test classes:

• TestTTSProviderParams 7 TTS provider/model combos
• TestLLMProviderParams 5 LLM provider/model combos
• TestSTTProviderParams 6 STT provider/model combos
• TestCustomPresetFullStack 3 complete STT+LLM+TTS stacks via build_custom_preset_providers()
• TestParamMismatchDetector 10 edge-case tests for known problem areas

tests/test_normal_preset_providers.py Path 2 (Normal Presets)

Tests every registered provider by calling build_stt(), build_tts(), build_llm() with real AgentMetaSchema objects. Verifies the hardcoded kwargs inside each provider function are valid.

What it catches:

• Hardcoded params that the plugin no longer supports (after a LiveKit version upgrade)
• Model-specific restrictions (e.g., keyterm passed to nova-2)
• Non-streaming STTs used in FallbackAdapter without StreamAdapter wrapping
• Missing package dependencies (e.g., aws_sdk_bedrock_runtime)

Test classes:

• TestNormalPresetSTT 13 STT provider/model/language combos (includes 60db)
• TestNormalPresetTTS 12 TTS provider/model/voice combos (includes 60db)
• TestNormalPresetLLM 12 LLM provider/model combos (including realtime)
• TestDeepgramKeytermHandling Nova-3 vs Nova-2 keyterm edge cases
• TestSTTFallbackChain Verifies fallback adapter builds correctly
• TestLLMFallbackChain Verifies fallback adapter builds, realtime skips it

How to Run

# All provider validation tests (requires Docker needs LiveKit plugins)
docker exec cb-agent python -m pytest tests/test_provider_params_integration.py tests/test_normal_preset_providers.py -v

# Quick summary
docker exec cb-agent python -m pytest tests/test_provider_params_integration.py tests/test_normal_preset_providers.py -q

# Just BYOK custom preset tests
docker exec cb-agent python -m pytest tests/test_provider_params_integration.py -v

# Just normal preset tests
docker exec cb-agent python -m pytest tests/test_normal_preset_providers.py -v

# Filter by category
docker exec cb-agent python -m pytest tests/test_normal_preset_providers.py -k "stt" -v
docker exec cb-agent python -m pytest tests/test_normal_preset_providers.py -k "tts" -v
docker exec cb-agent python -m pytest tests/test_normal_preset_providers.py -k "llm" -v

When to Run

• Before releasing a new provider or model to production
• After upgrading livekit-agents or any plugin package
• After editing byokPresetConfig.ts (adding/removing params)
• After editing provider functions in stt_providers.py / tts_providers.py / llm_providers.py

Adding a New Provider/Model

For BYOK (Path 1):

1. Add the provider/model in app/src/config/byokPresetConfig.ts
2. Add it to the appropriate PLUGIN_MAP in custom_preset_builder.py
3. Add a test case in test_provider_params_integration.py (TTS_CASES / LLM_CASES / STT_CASES)
4. Run the tests if it fails, the param name or value is wrong

For Normal Preset (Path 2):

1. Add/modify the provider function in stt_providers.py / tts_providers.py / llm_providers.py
2. Add the provider to the registry (STT_REGISTRY / TTS_REGISTRY / LLM_REGISTRY)
3. Add a test case in test_normal_preset_providers.py (STT_PRESET_CASES / TTS_PRESET_CASES / LLM_PRESET_CASES)
4. Run the tests if it fails, the hardcoded kwargs are wrong

Bugs Found by These Tests

Bug	Where	Impact	Fix
endpoint_sensitivity not supported by Soniox STTOptions	byokPresetConfig.ts + custom_preset_builder.py	BYOK Soniox STT crashes at dispatch	Removed unsupported param
deepgram_provider() passes keyterm to all models	stt_providers.py	Nova-2 calls crash with ValueError	Only pass keyterm for nova-3
Non-streaming STTs (OpenAI, Groq) in FallbackAdapter	stt_providers.py	Dispatch crashes when fallback is configured	Auto-wrap with StreamAdapter

Provider-Specific Notes

60db (STT + TTS)

Both 60db providers stream over WebSocket and are installed from the 60db-ai/livekit-agents fork (not PyPI). Their plugin constructors expose a deliberately small parameter surface, so the provider functions forward only what the plugin accepts:

• STT (sixtydb_provider in stt_providers.py): languages, encoding="mulaw", sample_rate=8000, continuous_mode=True, plus optional BYOK api_key.
• TTS (sixtydb_provider in tts_providers.py): voice_id (from meta.tts_voice), encoding="LINEAR16", sample_rate=8000, plus optional BYOK api_key.

Important: the 60db TTS plugin does not accept speed or language kwargs. meta.tts_speed and meta.tts_language are intentionally not forwarded — passing them would raise a TypeError. Voice/language selection is driven entirely by the 60db voice_id. The 60db TTS plugin also reconnects and re-authenticates its WebSocket on every utterance, which adds per-turn latency; this is a known limitation of the upstream plugin, not of the provider function.

API key resolution for both providers: explicit BYOK api_key → global _60dbClient default → SIXTY_DB_API_KEY env var. Tests skip automatically when no key is available.

Environment Requirements

• Tests run inside the cb-agent Docker container (has LiveKit plugins installed)
• API keys are loaded from /app/env/shared.env (auto-loaded by test setup)
• Providers that require missing keys are skipped (not failed) pytest.skip
• Providers that require missing packages (e.g., AWS Bedrock) are skipped via marks