Oktivo Testing Strategy
This folder contains documentation for the 3 testing tools built to validate the Oktivo voice calling pipeline without making real phone calls.
Why Test This Way?
Manual testing (calling from the UI and talking to the agent) doesn't scale. You can't manually verify:
- 1000 concurrent calls don't corrupt data
- Credit deduction is accurate and idempotent across retries
- Every contact ends up with recording, transcription, and analysis
- No contacts get stuck in
in_progressforever - The webhook pipeline handles all call outcomes (answered, rejected, timeout, media failure)
These tools simulate the entire backend pipeline from contact creation through webhook processing to credit billing without touching LiveKit, SIP, or real phone numbers.
The 5 Tools
| Tool | What It Does | When To Use |
|---|---|---|
| Integration Tests | 15 vitest tests covering the full webhook pipeline | Before every deploy, in CI |
| Load Test Pipeline | Synthetic campaign with N contacts at configurable concurrency | Before scaling up, after infra changes |
| E2E Call Test | Triggers a REAL test call through the full stack | Daily health check, after infra changes |
| E2E Campaign Test | Starts a REAL campaign via API, validates BullMQ orchestration | Before go-live, after orchestration changes |
| Campaign Validator | Post-hoc validation of any campaign's data integrity | After real campaigns, after load tests |
Testing Flow
1. Run integration tests (no infra needed, fast)
npx vitest --run src/tests/webhook-pipeline.test.ts
2. Run E2E single call (real stack, one phone call)
npx tsx src/tests/e2e-call-test.ts
3. Run E2E campaign (real stack, small batch)
npx tsx src/tests/e2e-campaign-test.ts --campaign <uid> --agent <id>
4. Run load test (synthetic, no real calls, high throughput)
npx tsx src/tests/load-test-pipeline.ts --contacts 500 --concurrency 50
5. Validate any campaign post-hoc
npx tsx src/scripts/validate-campaign.ts <campaignUID>Coverage Matrix
| Component | Integration Tests | Load Test | E2E Call | E2E Campaign |
|---|---|---|---|---|
| MongoDB operations | ✅ | ✅ | ✅ | ✅ |
| Webhook data pipeline | ✅ | ✅ | ✅ | ✅ |
| Billing duration logic | ✅ | ✅ | ✅ | ✅ |
| Credit deduction | ✅ | ✅ | ✅ | ✅ |
| High-throughput concurrency | ✅ | |||
| Real LiveKit server | ✅ | ✅ | ||
| Real SIP/VoiceLink/PSTN | ✅ | ✅ | ||
| Agent joining room | ✅ | ✅ | ||
| S3 recording upload | ✅ | ✅ | ||
| Transcription (real) | ✅ | ✅ | ||
| Analysis (real) | ✅ | ✅ | ||
| BullMQ orchestration | ✅ | |||
| Redis state management | ✅ | |||
| Campaign lifecycle | ✅ |
What's NOT Covered
- Frontend SSE real-time updates (frontend loads from DB, so backend correctness covers it)
- Agent AI quality (response latency, conversation naturalness) requires human evaluation
- SIP-level load testing (thousands of simultaneous real calls) would need SIPp + dedicated SIP trunk