Recorded Message Agent Features

This document describes the TTS (Text-to-Speech) features available for Recorded Message agents, including preset-based voices, custom BYOK presets, and MistralAI reference audio voice cloning.

Overview

Recorded Message agents support three TTS configuration modes:

1. Standard Preset - Uses pre-configured TTS providers with language and voice selection
2. Custom BYOK Preset - Bring Your Own Key configuration for custom TTS providers
3. MistralAI Reference Audio - Voice cloning using a reference audio sample

---

1. Standard Preset Mode

This is the default mode where customers select a preset, language, and voice from pre-configured options.

Required Fields

• preset_id - The selected voice plan preset ID
• tts_voice - Selected voice ID from the preset's available voices
• tts_language - Selected language code
• tts_speed (optional) - Speech speed multiplier (default: 1.0)

Flow

1. Customer selects a preset from the Voice Plan dropdown
2. Language selector shows languages available for that preset
3. Voice grid shows voices available for the selected language
4. Customer can adjust speed if the preset supports it

Backend Validation

// All fields are required for standard preset mode
if (messageType === "tts" && !isCustomPreset && !isRefAudioMode) {
  if (!preset_id || !tts_voice || !tts_language) {
    // Returns 400 error
  }
}

---

2. Custom BYOK Preset Mode

For customers with BYOK feature enabled, they can configure their own TTS provider and API key.

Required Fields

• preset_id - Must be set to 'custom'
• customPresetEnabled - Must be true
• byokConfig.tts_config - Custom TTS configuration object

tts_config Structure

{
  provider: "cartesia" | "elevenlabs" | "deepgram" | "openai" | "azure",
  model: string,
  voice: string,
  language?: string,
  key_id?: string,        // Reference to API key (UI only)
  api_key?: string,       // Decrypted API key (injected at dispatch)
  api_key_param?: string  // Custom param name for the API key
}

Flow

1. Customer selects a preset first (to show BYOK toggle)
2. Customer enables "Use Custom TTS Preset (BYOK)" toggle
3. Custom configuration panel appears
4. Customer selects provider, model, voice, and API key

Backend Handling

When customPresetEnabled is true:

• preset_id, tts_voice, tts_language are set to 'custom'
• TTS config comes from byokConfig.tts_config
• API key is decrypted and injected at dispatch time

---

3. MistralAI Reference Audio Mode

For MistralAI presets, customers can upload a reference audio file for voice cloning instead of selecting a voice.

Key Concept

MistralAI's Voxtral model supports voice cloning using a 3-25 second reference audio sample. When enabled, the TTS will generate speech that mimics the voice in the reference audio.

Required Fields

• preset_id - Must be a MistralAI preset (e.g., preset_oktivo_friendly)
• use_ref_audio - true to enable reference audio mode
• ref_audio_url (optional) - S3 URL of uploaded reference audio

Fields NOT Required in Ref Audio Mode

• tts_voice - Not used (voice comes from reference audio)
• tts_language - Not used (MistralAI doesn't accept language param)

Flow

1. Customer selects a MistralAI preset
2. "Use Reference Audio (Voice Cloning)" toggle appears (ON by default)
3. If toggle is ON:
   • Upload section appears for custom reference audio
   • Language and voice selectors are hidden
4. If toggle is OFF:
   • Standard language/voice selection appears

Reference Audio Requirements

• Duration: 3-25 seconds
• Format: MP3, WAV, or other common audio formats
• Quality: Clear voice sample with minimal background noise

Default Reference Audio

If use_ref_audio is true but no file is uploaded, the system uses a default reference audio:

https://oktivo-website-assets.s3.ap-south-1.amazonaws.com/Public/agent_voices/elevenlabs/2bNrEsM0omyhLiEyOwqY.mp3

S3 Storage Path

Uploaded reference audio files are stored at:

Public/recorded_messages/{orgUID}/{agentId}/ref_{timestamp}.{ext}

Backend Validation

// Only preset_id is required for ref audio mode
if (messageType === "tts" && isRefAudioMode && !preset_id) {
  // Returns 400 error
}

// Standard fields NOT required when use_ref_audio is true
if (messageType === "tts" && !isCustomPreset && !isRefAudioMode) {
  if (!preset_id || !tts_voice || !tts_language) {
    // Returns 400 error
  }
}

Agent Handler Logic

# In recorded_message_handler.py
if provider == "mistralai":
    use_ref_audio = getattr(self.meta, "use_ref_audio", False)
    ref_audio_url = getattr(self.meta, "ref_audio_url", None)

    if use_ref_audio and ref_audio_url:
        config["ref_audio"] = ref_audio_url  # Will be downloaded and base64-encoded
    elif use_ref_audio:
        config["ref_audio"] = MISTRAL_DEFAULT_REF_AUDIO_URL  # Fallback
    elif self.tts_voice:
        config["voice"] = self.tts_voice
    else:
        config["ref_audio"] = MISTRAL_DEFAULT_REF_AUDIO_URL  # Ultimate fallback

---

API Endpoints

Create Recorded Message Agent

POST /agents

{
  "AgentName": "My Message",
  "agentType": "recorded_message",
  "messageType": "tts",
  "ttsMessage": "Hello {{name}}, this is a test message.",
  "preset_id": "preset_oktivo_friendly",
  "use_ref_audio": true,
  "ref_audio_url": "https://...s3.../ref_audio.mp3",
  "repeatCount": 1
}

Upload Reference Audio

POST /agents/:agentId/ref-audio

Multipart form data with audio field containing the file.

Response:

{
  "success": true,
  "message": "Reference audio uploaded successfully",
  "url": "https://oktivo-website-assets.s3.../ref_123456.mp3"
}

---

Technical Implementation Details

Sample Rate Handling

MistralAI TTS outputs audio at 24kHz, but telephony requires 8kHz. The agent handler automatically resamples:

if tts_sample_rate and tts_sample_rate != SAMPLE_RATE:
    # Resample from 24000Hz to 8000Hz using linear interpolation
    duration = len(combined) / tts_sample_rate
    target_len = int(duration * SAMPLE_RATE)
    combined = np.interp(
        np.linspace(0, len(combined) - 1, target_len),
        np.arange(len(combined)),
        combined.astype(np.float64)
    ).astype(np.int16)

Reference Audio URL Resolution

Ref audio URLs are converted to base64 in _prepare_kwargs:

if provider == "mistralai":
    ref_audio = kwargs.get("ref_audio", "")
    if ref_audio.startswith("http"):
        encoded = _download_and_encode_ref_audio(ref_audio)
        if encoded:
            kwargs["ref_audio"] = encoded
            kwargs.pop("voice", None)  # Mutually exclusive

Provider Field Visibility

The tts_provider field is NOT stripped for customer users (unlike other internal fields) because the frontend needs it to:

1. Show/hide the reference audio toggle for MistralAI presets
2. Potentially enable other provider-specific UI in the future

---

Database Schema

Agent Document (MongoDB)

{
  // ... other fields ...
  agentType: "recorded_message",
  messageType: "tts",
  preset_id: "preset_oktivo_friendly",
  tts_voice: null,           // null for ref audio mode
  tts_language: null,        // null for ref audio mode
  tts_speed: 1.0,
  use_ref_audio: true,
  ref_audio_url: "https://...",
  byokConfig: null           // Only set for custom BYOK mode
}

---

Troubleshooting

Toggle Not Showing for MistralAI Preset

• Ensure tts_provider is not being stripped by sanitize.middleware.ts
• Check that the preset in SystemConfig has tts_provider: "mistralai"

"Missing required fields" Error on Save

• If ref audio toggle is ON, ensure use_ref_audio: true is being sent
• Check browser DevTools Network tab to verify payload

Bad Audio Quality (Garbled/Fast)

• The TTS output is being played at wrong sample rate
• Check agent logs for "Resampling TTS audio from XXXHz to 8000Hz"

Call Doesn't End Properly

• Check for "entrypoint did not exit in time, cancelling" in logs
• The audio drain wait may need adjustment (currently 1.5s)

Reference Audio Not Being Used

• Verify ref_audio_url and use_ref_audio are in dispatch metadata
• Check calling.service.ts for proper metadata construction