CSV Export System

Comprehensive CSV export for campaign contacts and call history with streaming support, dynamic column discovery, retry attempt row expansion, and email delivery.

---

Architecture Overview

---

How It Works

1. User triggers export from UI

• Campaign export: Three-dot menu → "Download CSV" (visible only when campaign is not running)
• History export: "Download CSV" button in History page toolbar (passes all active filters)

2. Frontend calls API with blob response type

The useExportCsv hook manages loading state, calls the export service with 120s timeout, and triggers browser download via blob URL.

3. Backend generates CSV

Two-pass approach:

1. First pass Discover all unique dynamicVariables keys and analysis keys across matching contacts (for consistent column headers)
2. Second pass Stream contacts in batches of 500, expand each into rows (one per retry attempt), write CSV

Sorting:

• Campaign export: csvRowIndex ascending (nulls last), then createdAt ascending
• History export: createdAt descending (most recent first)

Streaming vs Buffer:

• ≤10,000 contacts: Generate full buffer, send as response, attach to email

• 10,000 contacts: Stream directly to response with chunked transfer encoding (no email attachment)

4. Email delivery (fire-and-forget)

After CSV is sent to the client:

• Campaign: 25 MB email attachment limit
• History: 10 MB email attachment limit
• If over limit: X-Email-Skipped: file-too-large response header
• Email is queued via BullMQ never blocks the download

---

File Reference

File	Role	Key Functions
services/csvExport.service.ts	Core CSV logic	escapeCsvValue(), expandContactToRows(), discoverDynamicColumns(), generateCampaignCsv(), generateHistoryCsv(), streamCsvToResponse(), streamHistoryCsvToResponse(), buildHistoryQuery()
controllers/export.controller.ts	API handlers	exportCampaignCsv(), exportHistoryCsv(), queueCampaignExportEmail(), queueHistoryExportEmail()
routes/export.routes.ts	Route definitions	GET /export-campaign-csv/:campaignId, GET /export-history
app/src/services/export.service.ts	Frontend API client	downloadCampaignCsv(), downloadHistoryCsv()
app/src/hooks/useExportCsv.ts	React hook	exportCampaignCsv(), exportHistoryCsv(), isExporting state

---

CSV Column Structure

Order	Columns	Source
1	Contact Name, Phone Number, Original Phone Number	Fixed
2	Dynamic variable columns (alphabetical)	contact.dynamicVariables keys
3	Call Start Time, Call End Time, Call Duration (seconds), Billing Duration (seconds)	Fixed
4	Analysis columns (alphabetical)	contact.analysis keys
5	Success Criteria Met, Success Criteria Skipped, Success Criteria Details	Conditional (only when criteria data exists)
6	Recording URL, Call Status, Disconnect Reason, Agent Name, Contact ID, Call Direction, Campaign Name, Attempt Number	Fixed

Note on durations:

• Call Duration = actual call length in seconds
• Billing Duration = call duration rounded UP to the nearest minute in seconds (e.g., 90s → 120s, 30s → 60s). Legacy data is recomputed during export.

Note on Success Criteria columns:

• Only included when at least one contact in the dataset has success_criteria_met != null
• "Success Criteria Met" shows: "Yes" / "No" / "Skipped" / "" (empty if not applicable)
• "Success Criteria Details" shows per-criterion pass/fail breakdown

Retry Attempt Row Expansion

Each contact produces N+1 rows where N = callHistory.length:

• Rows 1..N: Previous retry attempts from callHistory[]
• Row N+1: Current/latest attempt from contact root fields
• Attempt numbers are sequential starting at 1
• Shared fields (name, phone, dynamicVariables, campaignName) are identical across all rows

---

API Endpoints

Export Campaign CSV

• Method: GET
• Path: /export-campaign-csv/:campaignId
• Auth: verifyJwt + requireFeatureAccess(exportData)
• Response: CSV file download
• Headers: Content-Type: text/csv, Content-Disposition: attachment; filename="{name}_{date}.csv"
• Error: 404 if campaign not found, 500 on server error

Export History CSV

• Method: GET
• Path: /export-history
• Auth: verifyJwt + requireFeatureAccess(exportData)
• Query Params: from, to, status, campaignUID, agentId, search, direction, minDuration, maxDuration, bookmarked, analysis.*
• Response: CSV file download
• Headers: Content-Type: text/csv, Content-Disposition: attachment; filename="call_history_{date}.csv"

---

How to Test (End-to-End)

Prerequisites

• Backend running locally with MongoDB connection
• Frontend running (npm run dev in app/)
• At least one completed campaign with contacts that have call data
• User account with exportData feature enabled

Test Campaign Export

1. Navigate to Campaigns page
2. Find a campaign that is not running (completed or stopped)
3. Click the three-dot menu (⋮) on that campaign row
4. Click "Download CSV"
5. Verify:
   • Loading spinner appears immediately
   • CSV file downloads after a few seconds
   • Toast shows "CSV downloaded. An email copy has been sent to your inbox."
   • Open CSV verify columns match the structure above
   • Verify retry attempts are expanded into separate rows
   • Verify dynamic variables and analysis fields are present as columns
   • Check email inbox for the CSV attachment

Test History Export

1. Navigate to History page
2. Apply some filters (date range, status, campaign, etc.)
3. Click "Download CSV" button in the toolbar
4. Verify:
   • Loading spinner appears
   • CSV downloads with filename call_history_YYYY-MM-DD.csv
   • Toast notification appears
   • Open CSV verify all rows match the applied filters
   • Verify agent names and campaign names are resolved (not UIDs)

Test Edge Cases

Scenario	Expected Behavior
Export while already exporting	Button disabled, no duplicate request
Empty campaign (no contacts)	CSV with header row only
Campaign with 10K+ contacts	Streaming response (no email)
Network timeout (>120s)	Error toast: "Export timed out..."
Server error (500)	Error toast: "Export failed due to server error..."
File >25MB (campaign) / >10MB (history)	X-Email-Skipped header, toast: "File was too large to email"

Verify in Database

// Check a contact has the expected fields for CSV
db.contacts.findOne({ campaignUID: "YOUR_CAMPAIGN_UID" }, {
  name: 1, phone: 1, dynamicVariables: 1, analysis: 1,
  callStatus: 1, callHistory: 1, recordingUrl: 1
})

---

Troubleshooting

Symptom	Cause	Fix
Recording URL empty in CSV	LiveKit webhook didn't deliver recording data	Check LiveKit egress webhook is configured and receiving events
Transcription/analysis empty	Analysis webhook not firing or agent not sending transcription	Check /transcription-sync and /analysis-sync endpoints are being called
"Download CSV" not visible	User doesn't have exportData feature	Enable feature in user's plan/org settings
Export times out	Dataset too large for 120s timeout	Apply filters to reduce dataset size
Email not received	Buffer >25MB (campaign) or >10MB (history)	Expected check X-Email-Skipped header in response
Duplicate columns in CSV	Same key in both dynamicVariables and analysis	By design they appear in separate column groups
Button stays disabled	Previous export errored without resetting state	Refresh the page

---

Changelog

• 2025-07-XX: Initial implementation campaign + history CSV export with streaming, email delivery, and UI integration