Analysis Schema System Complete Guide

Overview

The Analysis Schema system enables automated post-call conversation analysis with configurable feature extraction and success criteria evaluation. It covers the full lifecycle: schema creation → agent assignment → call execution → analysis pipeline → results storage → webhook notifications.

---

Architecture

---

Data Model

AnalysisSchema Document

{
  analysis_id: string;          // Unique ID (e.g., "anl_abc123")
  name: string;                 // Human-readable name
  industry: string | null;      // Industry category
  version: number;              // Schema version
  is_active: boolean;           // Soft delete flag
  is_template: boolean;         // Template vs custom schema
  orgUID: string | null;        // Organization owner
  customerUID: string | null;   // Customer owner (null for templates)
  createdBy: string | null;     // Creator user ID
  config: {
    model: { name: string; temperature: number; max_output_tokens: number };
    runtime: { min_user_messages_required: number; run_ai_only_if_min_messages: boolean; max_retries_on_failure: number };
    system_prompt: string;      // LLM instruction for feature extraction
  };
  features: Feature[];          // What to extract from conversation
  conditions: Condition[];      // Derived field logic (e.g., lead_temperature)
  success_criteria: SuccessCriterion[];  // Expected outcomes for pass/fail
}

Feature Definition

{
  name: string;           // Field name in output (e.g., "sentiment")
  type: string | string[];// "boolean" | "string" | "number" | ["string", "null"]
  description?: string;   // LLM guidance for extraction
  enum?: string[];        // Allowed values (e.g., ["positive", "neutral", "negative"])
  default?: any;          // Default when not extractable
  derived?: boolean;      // Computed via conditions, not LLM
}

Condition (Derived Fields)

{
  target: string;         // Feature to set (e.g., "lead_temperature")
  logic: "all" | "any";  // AND/OR for rules
  set_value: any;         // Value to assign when rules match
  rules: [{ field: string; operator: string; value?: any }];
}

Success Criterion

{
  feature_name: string;       // Must reference a boolean or enum feature
  expected_value: any;        // boolean | string | string[] (multi-value for enum)
  required_by_user: boolean;  // UI flag (does not affect evaluation logic)
}

Type restriction: Only boolean and enum features are eligible for success criteria. Free-text string features (e.g., summary, budget_raw) produce unpredictable LLM output that cannot be reliably compared with equality checks. This is enforced at both the UI level (feature dropdown filters) and the backend validation layer.

Multi-value enum: For enum features, expected_value can be a list of acceptable values (e.g., ["positive", "neutral"]). The criterion passes if the actual value matches ANY of the listed values. This allows users to define "sentiment should be positive OR neutral" as a single criterion.

Contact Document (Results)

{
  // ... existing fields ...
  analysis: Record<string, any>;           // Extracted features from LLM
  success_criteria_met: boolean | null;    // true/false if evaluated, null if skipped or pending
  success_criteria_skipped: string | null; // Reason AI didn't run (null if criteria was evaluated)
  success_criteria_details: [{             // Per-criterion breakdown (null if skipped)
    feature_name: string;
    expected_value: any;
    actual_value: any;
    passed: boolean;
  }] | null;
}

State machine for success_criteria_met:

State	success_criteria_met	success_criteria_skipped	Meaning
Pending	null	null	Call hasn't been analyzed yet
Skipped	null	"Not enough user messages..."	AI didn't run, criteria not evaluated
Passed	true	null	All criteria met
Failed	false	null	One or more criteria not met

---

Complete Lifecycle

Phase 1: Schema Creation (Frontend → Client Backend)

Endpoints:

Method	Path	Description
GET	/analysis-schemas	List templates + customer schemas
GET	/analysis-schemas/:id	Get single schema
POST	/analysis-schemas	Create custom schema
POST	/analysis-schemas/clone	Clone template into custom
PUT	/analysis-schemas/:id	Update schema
DELETE	/analysis-schemas/:id	Delete schema
GET	/analysis-schemas/industries	List supported industries
GET	/campaigns/:campaignUID/success-rate	Aggregate success rate

Ownership model:

• Templates (is_template=true): Seeded at startup, read-only for customers, managed by superadmin
• Custom schemas (is_template=false): Owned by individual customers (customerUID)
• Legacy "default": Backward compat for agents without explicit analysis_id

Validation: On create/update, validateSuccessCriteria() ensures:

1. Every criterion's feature_name references an existing feature in the schema
2. The referenced feature is of a deterministic type (boolean or has an enum array)

Free-text string features are rejected with a descriptive error. This prevents unreliable equality comparisons against unpredictable LLM output.

Phase 2: Agent Assignment

Each Agent has an optional analysis_id field. When set, the Agent Server uses this ID to trigger analysis after every call.

// Agent model
{ analysis_id?: string; }  // References AnalysisSchema.analysis_id

Phase 3: Call Ends → Analysis Triggered

When an AI-agent call ends, the Agent Server's post_call_event() fires:

1. POST /transcription-sync → stores transcription on Contact
2. POST /analyse/conversation → triggers analysis in background

No-agent bridge calls (human / phone_to_phone / human_callback) reach the same two endpoints, but the transcript comes from a live in-room Deepgram STT listener instead of an AI session. Their bridge handlers call sync_and_analyse (bridge_post_call.py) on call end. See Agent/docs/LIVE-STT-NO-AGENT-ROOMS.md and the "Analysis for No-Agent Bridge Calls" section below.

# Agent sends to Analysis Service
payload = {
    "analysis_id": meta.analysis_id,    # From Agent config
    "contactUID": contactUID,
    "organization_id": meta.organization_id,
    "customer_id": meta.customer_id,
    "campaign_id": meta.campaign_id,
    "room_name": room_name,             # Links Langfuse traces
    "org_name": meta.org_name,
    "customer_name": meta.customer_name,
    "customer_email": meta.customer_email,
    "campaign_name": meta.campaign_name,
}

Phase 4: Analysis Pipeline (Analysis Service)

The AnalysisOrchestrator.run() method executes:

Key steps:

1. Fetch conversation: DBFetcher.fetch_conversation_by_contact_id(contactUID)
2. Fetch schema: DBFetcher.fetch_analysis_schema(analysis_id)
3. Run pipeline: AnalysisPipeline.run_analysis(conversation, schema) sends conversation + schema to LLM, extracts features per schema definition
4. Check if AI ran: Pipeline returns meta.feature_source only evaluate criteria if source is "ai"
5. If AI ran: evaluate_success_criteria(schema.success_criteria, features) → write success_criteria_met + details
6. If AI didn't run: Write success_criteria_skipped with reason (e.g., "Not enough user messages for AI analysis")
7. Write to Contact: Analysis features written regardless of whether AI ran
8. Notify: POST /analysis-ready with just { contactUID }

Why skip criteria when AI doesn't run?

When the pipeline skips AI (due to min_user_messages_required or AI failure), it returns default feature values. Evaluating criteria against defaults would produce misleading false results the user would think the call failed criteria when in reality the analysis never ran.

Phase 5: Post-Analysis Events (Client Backend)

The /analysis-ready endpoint:

1. Validates contactUID in request body
2. Reads the Contact document (now has analysis + success_criteria_met)
3. If contact.customerUID exists:
   • Fires call.analysis_ready webhook event
   • Fires call.analysis_ready API action event
   • If isContactComplete(contact) → fires call.completed events
4. Forwards to AsliProperty (fire-and-forget)
5. Returns 200 { success: true }

---

Success Criteria Evaluation Logic

Why Only Boolean & Enum Features?

The evaluator uses strict equality (actual == expected). This works reliably for:

• Boolean features LLM is constrained to output true or false
• Enum features LLM is constrained to output one of the predefined values (e.g., "positive", "hot")

Free-text string features are not eligible because LLM output is unpredictable:

• Expected: "3BHK" → LLM might output: "user wants three BHK", "3 BHK flat", etc.
• These would all fail an equality check despite being semantically correct

This restriction is enforced at:

1. Frontend SuccessCriteriaBuilder only shows boolean/enum features in the dropdown
2. Backend validateSuccessCriteria() rejects criteria referencing non-boolean/non-enum features

How It Works

The evaluator uses AND-logic: ALL criteria must pass for success_criteria_met = true.

def evaluate_success_criteria(criteria, extracted_features):
    if not criteria:
        return None  # No criteria → no write

    details = []
    all_passed = True

    for criterion in criteria:
        actual = extracted_features.get(criterion.feature_name)
        # Multi-value: if expected is a list, check if actual is in the list
        if isinstance(criterion.expected_value, list):
            passed = (actual in criterion.expected_value)
        else:
            passed = (actual == criterion.expected_value)
        if not passed:
            all_passed = False
        details.append({
            "feature_name": criterion.feature_name,
            "expected_value": criterion.expected_value,
            "actual_value": actual,
            "passed": passed,
        })

    return (all_passed, details)

Evaluation Rules

Scenario	Result
No criteria defined	Returns None no write to Contact
All criteria pass	success_criteria_met = true
Any criterion fails	success_criteria_met = false
Feature missing from results	actual_value = None, passed = false (unless expected_value is also None)
expected_value = None and feature missing	passed = true
expected_value is a list (multi-value enum)	passed = true if actual_value is in the list
required_by_user field	No impact on evaluation (UI-only flag)

Example

Schema defines:

{
  "success_criteria": [
    { "feature_name": "is_interested", "expected_value": true, "required_by_user": true },
    { "feature_name": "sentiment", "expected_value": ["positive", "neutral"], "required_by_user": false }
  ]
}

Analysis extracts: { "is_interested": true, "sentiment": "neutral" }

Result:

{
  "success_criteria_met": true,
  "success_criteria_details": [
    { "feature_name": "is_interested", "expected_value": true, "actual_value": true, "passed": true },
    { "feature_name": "sentiment", "expected_value": ["positive", "neutral"], "actual_value": "neutral", "passed": true }
  ]
}

If sentiment was "negative" instead:

{
  "success_criteria_met": false,
  "success_criteria_details": [
    { "feature_name": "is_interested", "expected_value": true, "actual_value": true, "passed": true },
    { "feature_name": "sentiment", "expected_value": ["positive", "neutral"], "actual_value": "negative", "passed": false }
  ]
}

---

Campaign Success Rate

The getCampaignSuccessRate endpoint aggregates success criteria results across all contacts in a campaign:

// GET /campaigns/:campaignUID/success-rate
Contact.aggregate([
  { $match: { campaignUID, success_criteria_met: { $ne: null } } },
  { $group: {
    _id: null,
    total: { $sum: 1 },
    passed: { $sum: { $cond: [{ $eq: ["$success_criteria_met", true] }, 1, 0] } },
  }},
]);
// Returns: { successRate: 75, total: 100, passed: 75 }

A compound index { campaignUID: 1, success_criteria_met: 1 } ensures this query is fast.

---

Webhook & Notification Flow

Retry logic (Analysis → Client Backend):

• Up to BACKEND_MAX_RETRIES attempts
• 2-second delay between retries
• Returns False without raising on exhaustion

---

Frontend: SuccessCriteriaBuilder

The SuccessCriteriaBuilder component provides a type-aware UI for defining criteria:

• Feature dropdown: Only shows boolean and enum features (free-text features are filtered out)
• Expected value input: Adapts based on feature type:
  • boolean → Switch toggle (true/false)
  • enum → Multi-select checkboxes for each allowed value (select one or more)
• Multi-value enum: Users can check multiple enum values. The criterion passes if the actual value matches ANY of the selected values. Stored as string[] when multiple selected, string when single.
• Required by user: Toggle switch (advisory flag, does not affect evaluation)
• Add button: Disabled when no eligible features exist
• Empty state: Shows explanation when features exist but none are eligible

The getFeatureType helper determines eligibility:

• type === "boolean" → eligible
• feature.enum array exists and is non-empty → eligible (regardless of type field value)
• Everything else → ineligible (filtered out)

---

API Reference

Analysis Schema CRUD

Method	Path	Auth	Body
GET	/analysis-schemas	JWT
GET	/analysis-schemas/:id	JWT
POST	/analysis-schemas	JWT	{ name, industry?, features, conditions?, config?, success_criteria? }
POST	/analysis-schemas/clone	JWT	{ sourceAnalysisId, name?, features?, conditions?, config?, success_criteria? }
PUT	/analysis-schemas/:id	JWT	Partial update fields
DELETE	/analysis-schemas/:id	JWT

Inter-Service Webhooks

Method	Path	Auth	Body
POST	/analysis-ready	x-api-secret	{ contactUID }
POST	/analyse/conversation	x-api-secret	{ analysis_id, contactUID, org/campaign metadata }

Manual Analysis Generation (Customer-facing)

Method	Path	Auth	Description
POST	/api/v2/contacts/:contactUID/generate-analysis	JWT	Triggers analysis for a contact that has transcription but no analysis
GET	/api/v2/contacts/:contactUID/analysis	JWT	Returns just the analysis field for a contact (used for polling)

Campaign Analytics

Method	Path	Auth	Description
GET	/campaigns/:campaignUID/success-rate	JWT	Returns { successRate, total, passed }

---

Resilience: Contact Completion Monitor

The Contact Completion Monitor (contact-completion-monitor.service.ts) runs every 2 minutes and catches contacts where analysis was not generated despite having a transcription.

How it works:

1. Finds contacts with status=done/failed, transcription != null, analysis == null, ended within last 30 minutes
2. After a 3-minute grace period, resolves the agent's analysis_id from Contact.AgentID → Agent.analysis_id
3. Triggers POST /analyse/conversation with the correct payload (analysis_id, contactUID, org/campaign metadata)
4. Retries up to 3 times per contact (tracked via Redis with 30-min TTL)
5. Once analysis arrives, fires call.analysis_ready and call.completed webhooks

Bug fix (June 2026): Previously called /analyse/call-metrics (wrong endpoint) without analysis_id. Now correctly calls /analyse/conversation with full metadata resolution.

---

Error Handling

Scenario	Behavior
Analysis schema not found	Orchestrator logs error, returns early
Conversation not found	Orchestrator logs error, returns early
MongoDB write fails (analysis)	Logs error with contactUID + analysis_id, skips notification
Contact not found during write	Logs warning, skips notification
/analysis-ready notification fails	Retries up to max, logs failure, returns False
Success criteria write fails	Logs warning, continues with analysis write
Invalid criterion feature_name	Rejected at create/update time with descriptive error
Criterion on free-text feature	Rejected at create/update time only boolean/enum allowed

---

MongoDB Indexes

// Contact collection
{ campaignUID: 1, success_criteria_met: 1 }  // Success rate aggregation

// AnalysisSchema collection
{ analysis_id: 1 }       // Unique, primary lookup
{ is_template: 1 }       // Template listing
{ customerUID: 1 }       // Customer schema listing
{ orgUID: 1 }            // Org-level queries
{ industry: 1 }          // Industry filtering

---

CRM Integration: Schema Selection & Analysis for Human/P2P Calls

Schema Selection for CRM Calls

CRM-initiated human and phone-to-phone calls use an analysis schema selected at the organization level via the CRM's Settings > Integration page.

Flow:

1. CRM frontend calls GET /api/oktivo/analysis-schemas → CRM backend → Voice GET /api/crm/analysis-schemas
2. Voice returns available schemas (system schemas + customer's own custom schemas)
3. CRM user selects a schema via PUT /api/oktivo/settings/analysis-schema
4. Preference is stored as OktivoConnection.crmAnalysisSchemaId
5. When initiating a call, CRM passes analysisSchemaId to Voice in the call request
6. Voice stores it on Contact.analysisSchemaId and uses it when triggering analysis

Schema listing query (listSchemasForCustomer):

{
  $or: [
    { customerUID: null, is_template: false, is_active: true },  // System schemas ("default", "universal")
    { is_template: true, is_active: true },                       // Industry templates
    { customerUID, is_active: true, is_template: false },         // Customer's own schemas
  ]
}

Endpoints (on Voice, protected by crmAuthMiddleware):

Method	Path	Description
GET	/api/crm/analysis-schemas	List schemas available to the CRM customer
POST	/api/crm/retry-analysis	Retry analysis for a specific contact

Analysis for No-Agent Bridge Calls (Human / P2P / Human-Callback)

These calls have no AI agent in the room. This covers callType = human (web-to-phone), phone_to_phone (outbound P2P), and human_callback (inbound call routed to a human agent).

Primary flow live in-room transcription:

The transcript is produced live while the call is active by a backend-only STT listener (Agent/src/services/room_transcriber.py) that the bridge handlers start once both parties connect:

• phone_bridge_handler.py (P2P) — see also Phone Bridge Reliability for the debounced presence check and event-driven hangup monitoring.
• cloud_telephony_handler.py (web-to-phone human calls)
• human_agent_inbound_handler.py (human_callback)

The listener subscribes to each participant's audio track and runs one STT stream per track (no LLM, no TTS, no diarization LiveKit already separates participants, so each transcript is tagged with the raw participant.identity). The STT provider / model / language are resolved from the selected human_agent's preset (via resolveHumanAgentSttConfig), so any LiveKit STT provider works (BYOK supported; fallback deepgram/nova-3/multi). On call end, the handler calls sync_and_analyse (Agent/src/services/bridge_post_call.py): POST /transcription-sync (store transcript) → POST /analyse/conversation (same endpoint AI calls use).

The backend no longer triggers POST /analyse/crm-recording for these call types on egress_ended; the egress handlers only save the recording + fire recording_ready. There is no recording fallback if the live transcript is empty, no analysis runs.

Fallback flow recording-based (manual retry / reanalyze only):

POST /analyse/crm-recording (Sarvam STT on the saved recording) is still used when a user manually retries or reanalyzes (/api/crm/retry-analysis, reprocessCall, reanalyze-contacts.ts).

Analysis steps (both flows converge on the same pipeline):

1. Transcription:
   • Live flow: Deepgram STT in the room (roles are raw participant identities).
   • Recording flow: Sarvam AI on the recording with speaker diarization (roles are speaker_1/speaker_2).
2. Role Identification (always on for CRM bridge calls): An LLM maps the generic labels (participant identities or speaker_1/speaker_2) to agent/user/supervisor.
   • Role identification is auto-enabled in the orchestrator whenever the Contact's callType is human / phone_to_phone / human_callback, so it runs regardless of which endpoint triggered it (/analyse/conversation, /analyse/recording, or /analyse/crm-recording).
   • The role prompt is direction-aware, driven by the Contact's callDirection:
     • outbound → the AGENT initiated the call and the USER received it.
     • inbound → the USER (lead/customer) dialed in and the AGENT (human team member) received it. Critical for human_callback, where the lead calls and the assigned member answers.
   • Roles are never guessed positionally (e.g. "speaker_2 = user"). If the LLM cannot confidently label a speaker, the generic label is kept we do not fabricate a user/agent attribution, because that would corrupt analysis.
3. Feature Extraction: Schema-based analysis runs (same pipeline as AI agent calls).
4. Quality Analysis: Uses a human-specific prompt (not the AI agent evaluator prompt).
5. Conversation Summary: Generated alongside quality evaluation, stored as Contact.conversation_summary.

Min-message gate (why analysis is / isn't skipped)

The pipeline can skip AI when a call isn't a real conversation. The gate is satisfied if either:

• there are enough explicitly-labelled user messages (user_message_count >= min_user_messages_required), or
• it was a genuine two-sided conversation distinct_speaker_count >= 2 with enough total turns.

This two-sided signal is what makes the live in-room transcript work without any forced role mapping: the two participants surface as two distinct speaker labels (their raw LiveKit identities), so the gate passes even before role identification labels a user/agent.

The two-sided signal (distinct_speaker_count / conversational_turn_count from extract_conversation) is a gate-only signal. It decides whether to run analysis; it is never used to decide who the user is. This prevents a diarized-only transcript (roles still speaker_1/speaker_2) from being wrongly skipped as "no user messages", without ever mislabeling a speaker.

Conversation Summary vs Schema Analysis Summary

Two distinct summaries exist:

Field	Source	Purpose
Contact.analysis.summary	Schema feature extraction (if schema has a "summary" feature)	Customer-defined, schema-dependent
Contact.conversation_summary	Quality analysis LLM call (always generated)	System-generated, consistent across all call types

CRM uses conversation_summary exclusively for the AI Summary display. The schema's analysis.summary is shown separately in the Analysis section.

Human/P2P Quality Analysis

For human/P2P calls (identify_roles=True), the quality analysis uses:

• System prompt: "You are analyzing a phone conversation between human participants..." (no AI agent framing)
• Quality dimensions: Communication clarity, active listening, goal achievement, professionalism, context awareness, resolution effectiveness
• Time context: Minimal just today's date (not the full "Resolve relative dates..." instruction)
• Agent prompt: None (no AI agent involved)

For AI agent calls, the existing behavior is unchanged:

• System prompt: "You are an expert AI evaluator..." (evaluates AI agent performance)
• Quality dimensions: 12 dimensions including instruction_following, hallucination_check, etc.
• Time context: Full (with date resolution instructions)
• Agent prompt: Fetched from Agent config and injected into context

Contact Document Fields

{
  // Schema-based analysis features (customer-defined)
  analysis: Record<string, any>;
  
  // Quality analysis results (system-generated, always present when AI runs)
  conversation_summary: string | null;     // 3-4 sentence summary
  quality_evaluation: Record<string, any>; // Dimension scores
  
  // Analysis schema used for this contact
  analysisSchemaId: string | null;         // Set at creation for CRM calls
}

Webhook Payload

The call.completed webhook includes both:

{
  analysis: { ... },              // Schema feature extraction results
  conversation_summary: "...",    // Quality analysis summary (used by CRM as AI Summary)
}