Webhook Failure Detection

Background cron job that detects contacts stuck in an indeterminate call state due to missed LiveKit webhook callbacks, auto-fails them, deducts credits, pauses affected campaigns, and alerts administrators.

---

Architecture Overview

How It Works

1. Cron Trigger

The monitor runs every 5 minutes via node-cron, registered during initializeSimpleScheduler(). A guard flag prevents overlapping runs within the same process.

2. Distributed Lock

Acquires a Redis lock (webhook-monitor:lock, TTL 300s) using SET NX EX. If another instance holds the lock, the run is skipped. Retries every 500ms for up to 10 seconds before giving up.

3. Stale Contact Detection

Queries MongoDB for contacts matching ALL of:

• callStartedAt older than 30 minutes
• callEndedAt is null (never received end-of-call webhook)
• callStatus not in ["completed", "failed", "error"]
• disconnectReason is not "contact status update error" (idempotency guard)
• status is "calling" or "in_progress"

Uses the compound index idx_stale_contact_detection for performance.

4. Auto-Fail Contacts

For each stale contact, performs an atomic findOneAndUpdate:

• Sets callStatus: "failed", status: "failed"
• Sets disconnectReason: "contact status update error"
• Sets callEndedAt: now
• Computes callDuration (elapsed seconds) and billingDuration (rounded up to nearest minute)

5. Credit Deduction

Calls creditService.consumeCredits() with a deterministic serviceRef: "webhook-autofail:{contactUID}". The existing idempotency check in consumeCredits prevents duplicate deductions.

6. Campaign Pause

After processing contacts, checks each affected campaign:

• Counts ALL contacts with disconnectReason: "contact status update error" (cumulative across runs)
• If count >= 20 AND campaign isRunning is true → pauses the campaign
• Persists DB state first (isRunning: false, stopRequested: true), then attempts BullMQ/Redis cleanup
• If cleanup fails, DB state is still persisted (fail-safe)

7. Admin Alert

When a campaign is paused:

• Checks Redis deduplication key (webhook-monitor:alert:{campaignUID}, TTL 1hr)
• If no recent alert, finds admin/superadmin users in the same org
• Falls back to campaign owner email if no admins found
• Sends via BullMQ email queue with template webhook_failure_alert
• Retries up to 3 times with 5s delay on failure

8. Summary Logging

Every run logs a summary (even when zero stale contacts found):

📊 Webhook monitor run complete stale: 5, failed: 5, paused: 1, alerts: 1, errors: 0, duration: 342ms

File Reference

File	Role	Key Functions
src/services/webhook-monitor.service.ts	Core monitoring logic	initWebhookMonitor(), runWebhookMonitor(), findStaleContacts(), autoFailStaleContacts(), checkAndPauseCampaigns(), shouldSendAlert(), sendCampaignAlert()
src/services/email/templates/webhook_failure_alert.ts	Email template	webhookFailureAlertTemplate()
src/services/scheduler.service.ts	Startup integration	Calls initWebhookMonitor()
src/config/index.ts	Configuration	config.webhookMonitor.*
src/models/Contact.ts	Database index	idx_stale_contact_detection

Data Model

Database Index

// Contact model compound index for stale detection
contactSchema.index(
  { callStartedAt: 1, callEndedAt: 1, callStatus: 1, disconnectReason: 1 },
  { name: "idx_stale_contact_detection" }
);

Redis Keys

Key	TTL	Purpose
webhook-monitor:lock	300s	Distributed lock preventing concurrent runs
webhook-monitor:alert:{campaignUID}	3600s	Alert deduplication per campaign

Contact Fields Updated

Field	Value Set
callStatus	"failed"
status	"failed"
disconnectReason	"contact status update error"
callEndedAt	Current timestamp
callDuration	Elapsed seconds since callStartedAt
billingDuration	Math.ceil(elapsedSeconds / 60) * 60

Configuration

Environment Variable	Default	Description
WEBHOOK_MONITOR_ENABLED	true	Feature flag set to false to disable
WEBHOOK_MONITOR_CRON	*/5 * * * *	Cron schedule (every 5 minutes)
WEBHOOK_MONITOR_STALE_THRESHOLD_MINUTES	30	Minutes before a contact is considered stale
WEBHOOK_MONITOR_PAUSE_THRESHOLD	20	Cumulative auto-failed contacts to trigger campaign pause
WEBHOOK_MONITOR_LOCK_TTL_SECONDS	300	Redis lock TTL
WEBHOOK_MONITOR_ALERT_DEDUP_SECONDS	3600	Alert deduplication window (1 hour)

External Integrations

Service	What's Sent	What's Received	On Failure
Redis (Valkey)	Lock SET/DEL, alert dedup SET	OK/null	Skip run (lock) or skip alert (dedup)
MongoDB	Stale query, atomic updates, count queries	Contact/Campaign docs	Per-item try/catch, continue processing
BullMQ Email Queue	Alert email job	Job ID	Retry 3x with 5s delay, log and continue
CreditService	consumeCredits() call	Success/failure result	Log error, continue (contact still marked failed)

How to Test

Prerequisites

• MongoDB running with test data
• Redis running
• WEBHOOK_MONITOR_ENABLED=true in env

Manual Testing

1. Create contacts with callStartedAt > 30 minutes ago, callEndedAt: null, status: "calling"
2. Wait for the cron to fire (or call runWebhookMonitor() directly)
3. Verify in MongoDB: contacts have callStatus: "failed", disconnectReason: "contact status update error"
4. Verify credits were deducted (check credit transaction with serviceRef: "webhook-autofail:{contactUID}")
5. If 20+ contacts for one campaign: verify campaign has isRunning: false

Automated Tests

npx vitest --run src/services/webhook-monitor.service.test.ts

The integration test covers: stale detection, auto-fail fields, credit deduction, campaign pause, alert email, lock concurrency, and idempotency.

Troubleshooting

Symptom	Cause	Fix
Monitor not running	WEBHOOK_MONITOR_ENABLED=false	Set to true in env
"Lock held by another instance" every run	Previous run crashed without releasing lock	Wait for TTL (5min) to expire, or manually DEL webhook-monitor:lock in Redis
Contacts not being detected as stale	callStartedAt is less than 30 min ago	Wait or lower WEBHOOK_MONITOR_STALE_THRESHOLD_MINUTES
Campaign not pausing	Auto-failed count < 20	Check cumulative count: db.Contact.countDocuments({ campaignUID: "...", disconnectReason: "contact status update error" })
Alert email not sent	Deduplication window active	Check Redis key webhook-monitor:alert:{campaignUID} wait for TTL or DEL it
Credits not deducted	consumeCredits idempotency check	Verify serviceRef isn't already recorded this is by design for safety
Monitor runs but finds 0 stale contacts	All contacts already processed	Check disconnectReason field already-processed contacts are excluded

Changelog

Date	Change
2026-05-26	Initial implementation webhook monitor service, email template, integration tests