Oktivo Backend Docs

Appearance

🚀 [EPIC] Production Infrastructure Scaling 10 Lakh Monthly Calls, 100+ Concurrent per Org ​

Priority: P0 Critical (Client with 7-8 Lakh monthly calls ready to onboard) ​

Timeline: 1 Week Sprint ​

Assignee: DevOps Engineer (Part-time) ​

📋 Executive Summary ​

Oktivo is a white-label AI voice calling platform. We need to scale from our current single-EC2 setup to handle:

  • 10 Lakh (1 Million) calls/month across all tenants (~33,000 calls/day)
  • 100+ concurrent calls per organization as baseline
  • 7-8 Lakh monthly calls from first onboarding client (immediate need)
  • Future: scale to multiple orgs, each with 100+ concurrent

Current State (BROKEN for scale) ​

ComponentCurrent SetupProblem
LiveKit (Server+SIP+Egress+Redis)Single t3.xlarge EC2 (4vCPU, 16GB)ALL on ONE machine. Egress needs 1.5 CPU/recording. 100 concurrent = 150 CPU
Backend (Node+Agent+Analysis+Nginx)Single t3a.large EC2 (2vCPU, 8GB)Campaign state IN MEMORY, no horizontal scaling
MongoDBAtlas (managed)OK but needs pool/tier review
S3AWS S3OK, auto-scales
FrontendCloudflareOK
RedisDocker on LiveKit EC2Single point of failure, no HA

🔴 CRITICAL ISSUES FOUND IN CODEBASE AUDIT ​

Issue 1: Campaign State is IN-MEMORY (Biggest Blocker for Scaling Backend) ​

File: client_backend/src/services/calling.service.ts

typescript
// This is a local Map, NOT shared across instances
const campaignStates = new Map<string, CampaignState>();

fillCallSlots() runs every 2s via setInterval and manages all active calls in a local Map. If you run 2+ backend instances behind a load balancer, each has its own Map. Campaign state is lost on restart. No horizontal scaling possible.

Impact: Single Node.js process = single thread = ~5000-8000 concurrent connections max before event loop blocks.

Fix Required:

  1. Move campaign state to Redis (ElastiCache)
  2. Use Redis distributed locks for fillCallSlots() (only one instance fills slots per campaign)
  3. Use BullMQ for campaign job orchestration
  4. Active call tracking must be in Redis, not in-memory Map

Issue 2: maxConcurrent is Hardcoded to 2 ​

File: client_backend/src/services/calling.service.ts

typescript
async function initializeCampaignState(campaignUID: string, maxConcurrent = 2)
// and again:
const state = await initializeCampaignState(campaignUID, 2);

Every campaign can only run 2 concurrent calls. For 100+ concurrent per org, this must be configurable.

Fix Required:

  1. Add maxConcurrentCalls to Organization model (org-level limit)
  2. Add maxConcurrentCalls to Campaign model (campaign-level override)
  3. Read from org/campaign config instead of hardcoding
  4. Add rate limiting per org to prevent resource hogging

Issue 3: LiveKit Egress Cannot Handle 100 Concurrent Recordings on Single Machine ​

File: livekit_selfhosted/docker-compose.yml

yaml
cpu_cost:
  room_composite_cpu_cost: 1.5   # Each recording needs 1.5 CPU cores!

Math: 100 concurrent recordings × 1.5 CPU = 150 CPU cores just for egress. t3.xlarge has 4 cores = max ~2 simultaneous recordings.

Fix Required:

  1. Run multiple egress instances across multiple machines
  2. Auto-scale based on livekit_egress_requests metric
  3. Switch to Track Egress (0.5 CPU) or audio-only file output instead of Room Composite (1.5 CPU)

Issue 4: Single LiveKit Server Needs Multi-Node for 100+ Concurrent Rooms ​

A single LiveKit server on t3.xlarge (4 vCPU) handles ~50-100 audio-only rooms before CPU saturation. For 100+ concurrent per org (and multiple orgs), you need headroom.

Fix: Deploy 2-3 LiveKit server instances sharing same Redis. LiveKit auto-distributes rooms across nodes. Use NLB for WebSocket connections. This also gives you HA if one node dies, others keep serving.

Issue 5: SIP Gateway is Single Instance ​

All SIP traffic through one gateway. Port 5060 bound to one machine. Crash = all calls drop.

Fix: Run multiple SIP gateways. Configure SIP provider for multi-IP or SRV records. SIP gateways coordinate via Redis.

Issue 6: Redis is a Docker Container with No HA ​

yaml
redis:
  image: redis:7-alpine
  command: redis-server --bind 0.0.0.0 --appendonly yes

If Redis crashes, ALL LiveKit services lose coordination. No replication, no cluster mode.

Fix: Migrate to AWS ElastiCache Redis (cluster mode, Multi-AZ). Minimum r6g.large.

Issue 7: No Health Checks, No Monitoring, No Alerting ​

Zero observability. No way to know if calls fail, CPU maxes, egress drops recordings, or MongoDB connections exhaust.

Fix: CloudWatch agent, Prometheus + Grafana for LiveKit metrics (/metrics endpoint), application health checks, PagerDuty/Slack alerts.

Issue 8: No Connection Pooling for MongoDB ​

Default Mongoose pool = 5 connections. 200 concurrent calls with multiple DB ops per call = pool exhaustion.

Fix: maxPoolSize: 50+, ensure Atlas is on M30 tier, review indexes for high-frequency queries.

Issue 9: Agent Workers Cannot Scale ​

Single Python agent container. Each agent handles one call at a time. 200 concurrent = need 200 agent slots.

Fix: Run multiple agent containers, use num_idle_processes in WorkerOptions, auto-scale based on pending jobs.

Issue 10: No Rate Limiting on Webhook Endpoints ​

200 concurrent calls = 600-1000 webhook events/min hitting single Node.js process.

Fix: Webhook processing queue (BullMQ), async processing, rate limiting per org.

📐 CAPACITY MATH Why Current Setup Fails ​

Call Volume Breakdown ​

Target: 10 Lakh (1,000,000) calls/month
Working days: ~26 days/month
Calls per day: ~38,500 calls/day
Calling window: 9 AM - 9 PM IST = 12 hours = 720 minutes
Average call duration: ~2 minutes (assumption)

Calls per minute needed: 38,500 / 720 = ~53 calls/minute
Peak (2x average): ~106 calls/minute

If avg call = 2 min, concurrent calls at any time:
  53 calls/min × 2 min avg duration = ~106 concurrent calls at steady state
  Peak: ~200 concurrent calls

First client (7-8 Lakh/month):
  ~30,000 calls/day → ~42 calls/min → ~84 concurrent at steady state
  Peak: ~150 concurrent calls

Key insight: At 10 lakh/month, the real concurrency target is 100-200 concurrent calls, not thousands. This is very achievable with a modest multi-node setup. But your current single-EC2 setup still can't handle it because of egress CPU and in-memory state.

Resource Requirements per Component ​

LiveKit Server (audio-only SIP rooms, 2 participants each) ​

  • Each audio-only room: ~0.02 CPU cores (very lightweight for SFU)
  • 200 concurrent rooms: ~4 CPU cores
  • Recommendation: 2 c6i.xlarge instances (4 vCPU each) behind NLB
  • Gives you HA + headroom up to ~400 concurrent rooms

LiveKit SIP Gateway ​

  • Each concurrent SIP call: ~0.01-0.02 CPU for signaling + RTP relay
  • 200 concurrent: ~2-4 CPU cores
  • Recommendation: 2 c6i.large instances (2 vCPU each)
  • SIP provider must support sending to multiple IPs (or use NLB)

LiveKit Egress (BIGGEST resource hog) ​

  • Room Composite: 1.5 CPU per recording (current config)
  • Track Egress (audio-only): ~0.3-0.5 CPU per recording
  • MUST switch to Track Egress or audio-only file output
  • 200 concurrent recordings at 0.5 CPU = 100 CPU cores
  • Recommendation: 4-5 c6i.2xlarge instances (8 vCPU each) = 32-40 CPU cores
  • With Track Egress at 0.3 CPU: 32 cores handles ~100 concurrent recordings
  • Alternative: Skip egress entirely, record at SIP provider level (VoiceLink/Telnyx often offer recording)

Python Agent Workers ​

  • Each agent: ~0.3-0.5 CPU (STT + LLM + TTS pipeline)
  • But STT/LLM/TTS are external API calls (Deepgram, OpenAI, Cartesia) mostly I/O wait
  • Real CPU per agent: ~0.1-0.2 CPU (mostly async I/O)
  • 200 concurrent: ~20-40 CPU cores
  • Recommendation: 3-5 c6i.xlarge instances (4 vCPU each) with auto-scaling
  • LiveKit agents framework handles job distribution via Redis automatically

Node.js Backend ​

  • Webhook processing + campaign orchestration + DB operations
  • Mostly I/O bound (MongoDB, Redis, LiveKit SDK calls)
  • 200 concurrent: ~2-4 CPU cores with proper async handling
  • Recommendation: 2 t3.xlarge instances behind ALB
  • REQUIRES: Campaign state moved to Redis first

Redis (ElastiCache) ​

  • LiveKit room coordination + campaign state + job queues
  • 200 concurrent rooms metadata: ~100MB RAM
  • Recommendation: r6g.large (2 vCPU, 13GB) cluster mode, Multi-AZ

MongoDB Atlas ​

  • Write-heavy: ~300 writes/min (contact status updates, webhook data)
  • Read-heavy: ~500 reads/min (fillCallSlots queries, campaign checks)
  • Recommendation: M30 is sufficient. Add read replica if needed later.

🗓️ 1-WEEK SPRINT PLAN ​

Day 1-2: Foundation (DevOps) ​

Task 1.1: Set Up AWS ElastiCache Redis ​

  • Create ElastiCache Redis cluster (r6g.large, Multi-AZ)
  • VPC peering with existing EC2 instances
  • Update all services to point to ElastiCache instead of Docker Redis
  • Test LiveKit multi-node coordination works with ElastiCache
bash
# AWS CLI example
aws elasticache create-replication-group \
  --replication-group-id oktivo-redis \
  --replication-group-description "Oktivo LiveKit + Backend Redis" \
  --engine redis \
  --engine-version 7.1 \
  --cache-node-type cache.r6g.large \
  --num-node-groups 1 \
  --replicas-per-node-group 1 \
  --automatic-failover-enabled \
  --multi-az-enabled \
  --at-rest-encryption-enabled \
  --transit-encryption-enabled

Task 1.2: Set Up ECS Cluster ​

  • Create ECS cluster in ap-south-1
  • Set up ECR repositories for all images (or continue using DockerHub)
  • Create task definitions for: client_backend, agent, analysis, livekit-server, livekit-sip, livekit-egress
  • Set up ALB for backend services
  • Set up NLB for LiveKit WebSocket + SIP traffic

Task 1.3: LiveKit Multi-Node Setup ​

  • Deploy 2 LiveKit server instances on ECS with host networking
  • All pointing to ElastiCache Redis
  • NLB routing WebSocket traffic (port 7880) to all LiveKit nodes
  • Each node needs: UDP 50000-60000 (WebRTC), TCP 7881 (TCP fallback)
  • Security group: allow SIP provider IPs on port 5060

Day 2-3: LiveKit Scaling (DevOps) ​

Task 2.1: Scale SIP Gateways ​

  • Deploy 2 SIP gateway instances
  • Configure SIP provider (VoiceLink) to send to multiple IPs
  • Or use NLB for SIP traffic (TCP 5060)
  • Test failover: kill one SIP gateway, verify calls route to others

Task 2.2: Scale Egress Workers ​

  • CRITICAL: Change from Room Composite to Track Egress (audio-only)
  • Deploy 3-5 egress instances initially
  • Auto-scaling policy based on CPU utilization (target 70%)
  • Each egress instance: c6i.xlarge (4 vCPU) = ~8-13 concurrent audio recordings
yaml
# Updated egress config for audio-only track egress
cpu_cost:
  room_composite_cpu_cost: 1.5
  web_cpu_cost: 1.5
  track_composite_cpu_cost: 0.5  # Use this for audio-only
  track_cpu_cost: 0.3

Task 2.3: Scale Agent Workers ​

  • Deploy 3-5 agent containers initially
  • LiveKit agents framework auto-distributes jobs via Redis
  • Auto-scaling based on: pending jobs in Redis queue
  • Each agent container can run multiple processes (configure num_idle_processes)

Day 3-4: Backend Scaling (DevOps + Backend Dev) ​

Task 3.1: Move Campaign State to Redis (CODE CHANGE Backend Dev) ​

This is the most critical code change. The calling.service.ts needs refactoring:

typescript
// BEFORE (in-memory, single instance)
const campaignStates = new Map<string, CampaignState>();

// AFTER (Redis-backed, multi-instance safe)
// Use Redis hashes for campaign state
// Use Redis sorted sets for active calls
// Use Redlock for distributed locking on fillCallSlots

Key changes needed in calling.service.ts:

  1. Replace campaignStates Map with Redis hash operations
  2. Replace activeCalls Map with Redis set per campaign
  3. Add distributed lock around fillCallSlots() per campaign
  4. Replace setInterval with BullMQ repeatable job
  5. Make recoverCampaignStates() work across instances

Task 3.2: Deploy Multiple Backend Instances ​

  • 2 instances behind ALB
  • Sticky sessions NOT needed (state is in Redis now)
  • Health check endpoint: GET /health
  • Auto-scaling: CPU > 70% or request count > 500/min

Task 3.3: Webhook Processing Queue ​

  • Add BullMQ queue for LiveKit webhook processing
  • Webhooks are received by any backend instance, pushed to queue
  • Workers process webhooks from queue (idempotent processing)
  • This prevents webhook storms from blocking the event loop

Day 4-5: Monitoring & Observability (DevOps) ​

Task 4.1: CloudWatch Setup ​

  • CloudWatch agent on all EC2/ECS instances
  • Custom metrics: active_calls, campaigns_running, webhook_queue_depth
  • Alarms: CPU > 80%, Memory > 85%, Disk > 90%
  • Dashboard for real-time monitoring

Task 4.2: Prometheus + Grafana ​

  • Deploy Prometheus to scrape LiveKit metrics (/metrics on port 7880)
  • Key LiveKit metrics to monitor:
    • livekit_room_count active rooms
    • livekit_participant_count active participants
    • livekit_packet_loss call quality
    • livekit_egress_requests active recordings
    • livekit_sip_calls active SIP calls
  • Grafana dashboards for: LiveKit, Backend, Agent, MongoDB, Redis

Task 4.3: Alerting ​

  • Slack/PagerDuty integration
  • Critical alerts: service down, egress failures, Redis disconnection
  • Warning alerts: high CPU, high memory, slow DB queries, webhook queue growing

Day 5-6: Load Testing (DevOps + Backend Dev) ​

Task 5.1: Load Testing Setup ​

  • Use lk CLI (LiveKit CLI) for SIP load testing
  • Or build a custom load test script using LiveKit SDK
python
# Load test script concept spawn N concurrent SIP calls
import asyncio
from livekit import api

async def simulate_calls(n_concurrent: int):
    """Simulate N concurrent outbound SIP calls"""
    lk = api.LiveKitAPI(
        url="https://voice.oktivo.com",
        api_key="...",
        api_secret="..."
    )
    
    tasks = []
    for i in range(n_concurrent):
        room_name = f"loadtest-room-{i}"
        # Create room + dispatch agent
        task = asyncio.create_task(
            lk.room.create_room(api.CreateRoomRequest(name=room_name))
        )
        tasks.append(task)
    
    results = await asyncio.gather(*tasks, return_exceptions=True)
    print(f"Created {len([r for r in results if not isinstance(r, Exception)])} rooms")
    
    # Wait for calls to complete
    await asyncio.sleep(120)  # 2 min call duration
    
    # Cleanup
    for i in range(n_concurrent):
        await lk.room.delete_room(api.DeleteRoomRequest(room=f"loadtest-room-{i}"))

asyncio.run(simulate_calls(100))

Task 5.2: Staged Load Testing Plan ​

  1. 10 concurrent calls baseline, verify everything works
  2. 50 concurrent calls identify first bottlenecks
  3. 100 concurrent calls target for first client
  4. 200 concurrent calls peak capacity test (covers 10 lakh/month with headroom)

For each stage, monitor:

  • Call success rate (target: >98%)
  • Call setup time (target: ❤️ seconds)
  • Recording success rate (target: >99%)
  • CPU/Memory across all services
  • MongoDB query latency
  • Redis memory usage
  • Webhook processing latency

Task 5.3: SIP Provider Capacity ​

  • Contact VoiceLink to confirm they can handle 200+ concurrent calls
  • Verify SIP trunk capacity (channels)
  • Test with Telnyx as backup provider
  • Ensure caller ID rotation works at scale (multiple phone numbers per org)

Day 6-7: Hardening & Documentation (DevOps) ​

Task 6.1: Auto-Scaling Policies ​

  • ECS Service Auto Scaling for all services
  • Target tracking: CPU utilization 70%
  • Step scaling for burst traffic
  • Minimum instances: LiveKit=2, SIP=2, Egress=3, Agent=3, Backend=2
  • Maximum instances: LiveKit=5, SIP=3, Egress=10, Agent=10, Backend=4

Task 6.2: Disaster Recovery ​

  • Redis: Multi-AZ with automatic failover
  • MongoDB: Atlas handles this (ensure backups are configured)
  • S3: Cross-region replication for recordings (optional, cost consideration)
  • Backend: Multi-AZ ECS deployment
  • LiveKit: Draining mode handles graceful shutdown (built-in)

Task 6.3: Security Hardening ​

  • All inter-service communication over private VPC
  • No public IPs on backend services (only through ALB/NLB)
  • Redis: VPC-only access, encryption in transit + at rest
  • Rotate LiveKit API keys
  • Webhook signature verification (already implemented ✅)
  • Rate limiting on all public endpoints

Task 6.4: CI/CD Pipeline ​

  • GitHub Actions for building + pushing Docker images
  • ECS deployment via aws ecs update-service
  • Blue/green deployment for zero-downtime updates
  • Rollback strategy: keep previous task definition

💻 CODE CHANGES REQUIRED (Backend Dev Tasks) ​

Code Task 1: Campaign State → Redis Migration ​

Priority: P0 Must be done before any horizontal scaling

Files to modify:

  • client_backend/src/services/calling.service.ts (major refactor)
  • client_backend/src/config/index.ts (add Redis config)
  • client_backend/src/server.ts (initialize Redis connection)

New dependencies:

bash
npm install ioredis bullmq redlock

Architecture:

Redis Keys:
  campaign:{campaignUID}:state     → Hash (isRunning, maxConcurrent, phase, etc.)
  campaign:{campaignUID}:active    → Set of phone numbers currently being called
  campaign:{campaignUID}:config    → Hash (callConfig: sipTrunkId, sipPhoneNumber)
  campaign:{campaignUID}:lock      → Redlock key for fillCallSlots mutex

BullMQ Queues:
  campaign-orchestrator            → Repeatable job per campaign (replaces setInterval)
  webhook-processor                → LiveKit webhook events
  call-completion                  → Post-call processing (transcription, analysis triggers)

Code Task 2: Make maxConcurrent Configurable ​

Files to modify:

  • client_backend/src/models/Organization.ts add maxConcurrentCalls: Number
  • client_backend/src/models/Campaign.ts add maxConcurrentCalls: Number
  • client_backend/src/services/calling.service.ts read from org/campaign
typescript
// In initializeCampaignState:
const org = await Organization.findOne({ orgUID: campaign.orgUID });
const maxConcurrent = campaign.maxConcurrentCalls || org?.maxConcurrentCalls || 10; // default 10

Code Task 3: Switch Egress to Track-based (Audio-Only) ​

File: client_backend/src/services/livekit.service.ts

Current code uses startRoomCompositeEgress (1.5 CPU per recording). Switch to startTrackCompositeEgress or startTrackEgress for audio-only (0.3-0.5 CPU).

typescript
// BEFORE:
const info = await client.startRoomCompositeEgress(roomName, output, opts);

// AFTER (track composite, audio-only):
const info = await client.startTrackCompositeEgress(roomName, output, {
  audioTrackId: agentTrackId,  // or use participant identity
});

Note: This changes the recording format. Test thoroughly. Dual-channel agent mode may need adjustment.

Code Task 4: Add Health Check Endpoint ​

File: client_backend/src/app.ts

typescript
app.get('/health', async (req, res) => {
  const checks = {
    mongodb: mongoose.connection.readyState === 1,
    redis: await redisClient.ping() === 'PONG',
    uptime: process.uptime(),
    memory: process.memoryUsage(),
    activeCampaigns: await getActiveCampaignCount(), // from Redis
  };
  const healthy = checks.mongodb && checks.redis;
  res.status(healthy ? 200 : 503).json(checks);
});

Code Task 5: MongoDB Connection Pool ​

File: client_backend/src/config/database.ts

typescript
await mongoose.connect(config.mongoUri, {
  maxPoolSize: 50,         // Up from default 5, sufficient for 200 concurrent
  minPoolSize: 5,
  maxIdleTimeMS: 30000,
  serverSelectionTimeoutMS: 5000,
  socketTimeoutMS: 45000,
});

Code Task 6: Add Per-Org Rate Limiting ​

Prevent one org from consuming all system resources:

typescript
// In fillCallSlots, check global concurrent limit
const globalActiveCalls = await redis.scard('global:active_calls');
const orgActiveCalls = await redis.scard(`org:${orgUID}:active_calls`);
const orgLimit = org.maxConcurrentCalls || 100;

if (orgActiveCalls >= orgLimit) {
  console.log(`Org ${orgUID} at concurrent limit (${orgActiveCalls}/${orgLimit})`);
  return;
}

🧪 LOAD TESTING PLAN ​

Option A: LiveKit CLI Load Test (Simplest) ​

bash
# Install LiveKit CLI
curl -sSL https://get.livekit.io/cli | bash

# Create 100 test rooms
for i in $(seq 1 100); do
  lk room create --url wss://voice.oktivo.com \
    --api-key $LIVEKIT_API_KEY \
    --api-secret $LIVEKIT_API_SECRET \
    "loadtest-$i" &
done
wait

Create scripts/load_test.py:

python
"""
Oktivo Load Test Simulates concurrent campaign calls
Usage: python load_test.py --concurrent 100 --duration 120
"""
import asyncio
import argparse
import time
import aiohttp
from livekit import api

async def create_test_call(session, backend_url, call_id):
    """Simulate a single call via the backend API"""
    start = time.time()
    try:
        # Option 1: Hit the backend start-campaign endpoint
        # Option 2: Directly create LiveKit rooms + dispatches
        async with session.post(f"{backend_url}/api/v2/test/simulate-call", json={
            "callId": f"loadtest-{call_id}",
            "phone": f"+9199999{call_id:05d}",
        }) as resp:
            elapsed = time.time() - start
            return {"id": call_id, "status": resp.status, "time": elapsed}
    except Exception as e:
        return {"id": call_id, "status": "error", "error": str(e)}

async def run_load_test(concurrent: int, duration: int, backend_url: str):
    print(f"Starting load test: {concurrent} concurrent calls, {duration}s duration")
    
    async with aiohttp.ClientSession() as session:
        tasks = [create_test_call(session, backend_url, i) for i in range(concurrent)]
        results = await asyncio.gather(*tasks)
    
    success = len([r for r in results if r.get("status") == 200])
    failed = len(results) - success
    avg_time = sum(r.get("time", 0) for r in results) / len(results)
    
    print(f"\nResults: {success} success, {failed} failed, avg setup time: {avg_time:.2f}s")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--concurrent", type=int, default=10)
    parser.add_argument("--duration", type=int, default=120)
    parser.add_argument("--backend", default="https://backend.oktivo.com/client")
    args = parser.parse_args()
    asyncio.run(run_load_test(args.concurrent, args.duration, args.backend))

Option C: Full End-to-End Test with Real SIP Calls ​

For this you need a SIP load testing tool like SIPp:

bash
# Install SIPp
apt-get install sipp

# Run 100 concurrent calls to your SIP gateway
sipp -sf uac.xml -d 120000 -l 100 -r 10 \
  -s +919876543210 voice.oktivo.com:5060 -t t1

Load Test Checklist ​

For each concurrency level (10, 50, 100, 200):

  • [ ] All calls connect successfully (>98% success rate)
  • [ ] Call setup time < 3 seconds
  • [ ] All recordings saved to S3 (>99% success rate)
  • [ ] No MongoDB connection errors
  • [ ] No Redis connection errors
  • [ ] CPU < 80% on all instances
  • [ ] Memory < 85% on all instances
  • [ ] Webhook processing latency < 2 seconds
  • [ ] No orphaned rooms in LiveKit (rooms cleaned up after call ends)
  • [ ] Credit deduction accurate for all calls
  • [ ] Transcription and analysis complete for all calls

💰 ESTIMATED AWS COSTS (Monthly) ​

For 100 Concurrent Calls (First Client 7-8 Lakh/month) ​

ServiceSpecCountMonthly Cost (USD)
LiveKit Serverc6i.xlarge (4 vCPU)2~$250
LiveKit SIPc6i.large (2 vCPU)2~$130
LiveKit Egressc6i.xlarge (4 vCPU)3~$375
Agent Workersc6i.xlarge (4 vCPU)3~$375
Backend (Node.js)t3.xlarge (4 vCPU)2~$240
Analysist3.large (2 vCPU)1~$60
ElastiCache Redisr6g.large (2 vCPU)1 (Multi-AZ)~$200
MongoDB AtlasM301~$400
ALB + NLB-2~$50
S3 Storage~200GB/month recordings-~$5
Data Transfer~1TB/month-~$90
CloudWatchMetrics + Logs-~$50
TOTAL~$2,225/month

For 200 Concurrent Calls (10 Lakh/month Full Target) ​

ServiceSpecCountMonthly Cost (USD)
LiveKit Serverc6i.xlarge (4 vCPU)2~$250
LiveKit SIPc6i.large (2 vCPU)2~$130
LiveKit Egressc6i.2xlarge (8 vCPU)4~$1,000
Agent Workersc6i.xlarge (4 vCPU)5~$625
Backend (Node.js)t3.xlarge (4 vCPU)2~$240
Analysist3.large (2 vCPU)2~$120
ElastiCache Redisr6g.large (2 vCPU)1 (Multi-AZ)~$200
MongoDB AtlasM301~$400
ALB + NLB-2~$50
S3 Storage~500GB/month recordings-~$12
Data Transfer~2TB/month-~$180
CloudWatch + Grafana--~$100
TOTAL~$3,307/month

Note: These are on-demand prices. With Reserved Instances (1-year) you save ~30-40%. With Spot Instances for agent workers you save ~60-70%.

🚦 ECS vs EKS Decision ​

Recommendation: ECS is the right choice for this scale ​

Why ECS:

  • 10 lakh/month = ~100-200 concurrent calls ECS handles this comfortably
  • Simpler to set up (1-2 days vs 1 week for EKS)
  • Lower operational overhead (no K8s cluster management)
  • Auto-scaling works well with ECS Service Auto Scaling
  • Cost-effective (no EKS control plane fee of $73/month)
  • Your team doesn't have deep K8s expertise ECS is the pragmatic choice

When to consider EKS (future):

  • When you have 10+ orgs each with 100+ concurrent (1000+ total concurrent)
  • When you need LiveKit's official Helm chart for complex multi-region
  • When you have a dedicated DevOps team

ECS Gotcha for LiveKit: LiveKit requires host networking (network_mode: host). In ECS, this means using EC2 launch type (NOT Fargate). One LiveKit task per EC2 instance.

⚡ QUICK WINS (Do These First, Before Full Migration) ​

If you need to onboard the first client ASAP before the full ECS migration:

Quick Win 1: Vertical Scale LiveKit EC2 ​

  • Upgrade from t3.xlarge to c6i.4xlarge (16 vCPU, 32GB) handles ~10 concurrent recordings
  • Cost: ~$500/month (vs current ~$120)
  • Time: 30 minutes (stop instance, change type, start)

Quick Win 2: Run Multiple Egress Containers ​

  • On the same machine, run 2-3 egress containers (they coordinate via Redis)
  • Each gets a share of CPU
  • Handles ~6-8 concurrent recordings on c6i.4xlarge

Quick Win 3: Increase maxConcurrent in Code ​

  • Change hardcoded 2 to 10 or 20 in calling.service.ts
  • Quick code change, deploy immediately

Quick Win 4: Upgrade Backend EC2 ​

  • Upgrade from t3a.large to c6i.2xlarge (8 vCPU, 16GB)
  • Handles more concurrent webhook processing
  • Cost: ~$250/month

Quick Win 5: Run Multiple Agent Containers ​

  • On backend EC2, run 3-5 agent containers
  • They auto-register with LiveKit via Redis
  • Handles 15-25 concurrent calls

Quick Win 6: MongoDB Connection Pool ​

  • Add maxPoolSize: 50 to Mongoose connect options
  • Zero-downtime deploy

With Quick Wins alone, you can handle ~20-30 concurrent calls. Enough for initial testing but NOT for 100+ concurrent.

📎 REFERENCE: Current Codebase File Map ​

livekit_selfhosted/
├── docker-compose.yml          # LiveKit Server + SIP + Egress + Redis
├── .env                        # LiveKit API keys, S3 creds, webhook URL
├── Dockerfile                  # Minimal Alpine image
└── scripts/init-server.sh      # Nginx + SSL setup for LiveKit EC2

oktivo_backend/
├── docker-compose.yml          # Backend services: analysis, agent, client_backend, nginx
├── docker-compose.dev.yml      # Dev environment with hot reload
├── nginx.conf                  # Reverse proxy: /client/ → Node.js, /analysis/ → FastAPI
├── Agent/
│   ├── agent.py                # LiveKit agent entrypoint (Python)
│   ├── config.py               # Agent config
│   ├── requirements.txt        # livekit-agents, STT/TTS plugins
│   └── src/                    # Agent logic (voice assistant, call handler, etc.)
├── Analysis/
│   └── requirements.txt        # FastAPI, OpenAI for post-call analysis
├── client_backend/
│   ├── src/
│   │   ├── services/
│   │   │   ├── calling.service.ts    # ⚠️ Campaign orchestration (IN-MEMORY state)
│   │   │   ├── livekit.service.ts    # LiveKit SDK integration
│   │   │   ├── telephony.service.ts  # SIP trunk + phone number resolution
│   │   │   ├── s3.service.ts         # Recording storage
│   │   │   ├── credit.service.ts     # Credit management
│   │   │   └── scheduler.service.ts  # Cron-based campaign scheduling
│   │   ├── models/                   # Mongoose models
│   │   ├── controllers/              # Express route handlers
│   │   └── config/                   # DB connection, env config
│   └── package.json
└── docs/
    ├── ARCHITECTURE-CAMPAIGN-CALL-FLOW.md  # Full call flow documentation
    ├── CREDIT-MANAGEMENT-SYSTEM.md         # Credit system docs
    ├── PHONE-NUMBER-ASSIGNMENT.md          # Phone number pool management
    └── DEVELOPER-GUIDE.md                  # Coding conventions

✅ ACCEPTANCE CRITERIA ​

  • [ ] LiveKit cluster running 2+ server nodes with shared Redis
  • [ ] SIP gateway running 2 instances
  • [ ] Egress running 3+ instances with auto-scaling
  • [ ] Agent workers running 3+ instances with auto-scaling
  • [ ] Backend running 2 instances behind ALB
  • [ ] Campaign state stored in Redis (not in-memory)
  • [ ] ElastiCache Redis (Multi-AZ) replacing Docker Redis
  • [ ] MongoDB Atlas M30 with connection pooling (maxPoolSize: 50+)
  • [ ] Prometheus + Grafana monitoring dashboard
  • [ ] CloudWatch alarms for critical metrics
  • [ ] Load test passing at 100 concurrent calls with >98% success rate
  • [ ] Health check endpoints on all services
  • [ ] CI/CD pipeline for automated deployments
  • [ ] Runbook document for common operational tasks
  • [ ] maxConcurrentCalls configurable per org/campaignrentCalls configurable per org/campaign

🔗 Useful References ​

Last updated: