Sentry Error Monitoring & Google Chat Notifications Setup

Overview

Sentry is integrated into 3 backend services:

1. client_backend (Node.js/Express) oktivo_voice/backend/client_backend/
2. Analysis (Python/FastAPI) oktivo_voice/backend/Analysis/
3. Agent (Python/LiveKit) oktivo_voice/backend/Agent/

All services share the same Sentry DSN (single project) and forward error alerts to a Google Chat space via a webhook relay endpoint in client_backend.

---

Architecture

┌─────────────────┐     ┌──────────────┐     ┌──────────────────┐
│  client_backend │────▶│   Sentry     │────▶│  Internal        │
│  (Express)      │     │   (ingest)   │     │  Integration     │
├─────────────────┤     └──────────────┘     │  Webhook         │
│  Analysis       │────▶│              │     └────────┬─────────┘
│  (FastAPI)      │     │              │              │
├─────────────────┤     │              │              ▼
│  Agent          │────▶│              │     ┌──────────────────┐
│  (LiveKit)      │     │              │     │ /sentry-webhook  │
└─────────────────┘     └──────────────┘     │ (client_backend) │
                                             └────────┬─────────┘
                                                      │
                                                      ▼
                                             ┌──────────────────┐
                                             │  Google Chat     │
                                             │  "Dev Oktivo"    │
                                             │  Space           │
                                             └──────────────────┘

---

Step-by-Step Setup

Step 1: Create Google Chat Space Webhook ✅ DONE

• Space: Dev Oktivo
• Webhook Name: Sentry Alerts
• Webhook URL: Already configured in shared.env as GOOGLE_CHAT_WEBHOOK_URL

How it was set up:

1. Open Google Chat → Go to the Dev Oktivo space
2. Click the space name at the top
3. Click Apps & integrations → Webhooks
4. Click Create Webhook
5. Name: Sentry Alerts
6. Avatar URL (optional): https://sentry.io/favicon.ico
7. Click Save
8. Copy the webhook URL

---

Step 2: Set Environment Variables ✅ DONE

Added to env/shared.env (used by all 3 services):

SENTRY_DSN=https://4e51b3a8a45db7c6f4242c4848f40a86@o4511512196939776.ingest.de.sentry.io/4511512203034704
GOOGLE_CHAT_WEBHOOK_URL=https://chat.googleapis.com/v1/spaces/AAQADw0yb-0/messages?key=AIzaSyDdI0hCZtE6vySjMm-WEfRq3CPzqKqqsHI&token=Viz_Ofy7XhqFXHcBlHLOKbX_gRljG6_T7iNebKhVuYA
SENTRY_WEBHOOK_SECRET=b87729625169695bb88823b37a4304e9709a42e92ee97681edce9b3f68f91773

For production, add these same keys to your AWS Secrets Manager secret: oktivo/prod/shared

---

Step 3: Create Sentry Internal Integration

1. Go to https://sentry.io → Log in
2. Navigate to Settings (gear icon) → Developer Settings
3. Click Create New Integration → Choose Internal Integration
4. Fill in:
   • Name: Google Chat Relay
   • Webhook URL: https://client-backend.oktivo.com/sentry-webhook
   • Overview: "Forwards alerts to Google Chat Dev Oktivo space"
5. Under Permissions leave all as "No Access" (relay doesn't need to read Sentry data)
6. Toggle ON: ☑ Alert Rule Action (this makes it available as an alert action)
7. Click Save
8. Copy the Client Secret displayed on the integration page
9. Verify it matches SENTRY_WEBHOOK_SECRET in your env, or update accordingly

---

Step 4: Create Alert Rules in Sentry

1. Go to Alerts → Create Alert
2. Select Issues (for error monitoring)
3. Configure:

Rule A: New Production Error

• Environment: production
• When: "A new issue is created"
• If: "The issue's level is equal to error" (or fatal)
• Then: "Send a notification via Google Chat Relay" (the integration you created in Step 3)
• Rate limit: "Perform this action at most once every 30 minutes for each issue"
• Name: Production Error → Google Chat
• Click Save Rule

Rule B: High Error Volume (Optional)

• When: "The issue is seen more than 10 times in 5 minutes"
• Then: "Send a notification via Google Chat Relay"
• Name: High Error Volume → Google Chat
• This catches high-frequency errors that might indicate an outage.

---

Step 5: Deploy & Test

1. Rebuild Docker images:

   docker compose -f docker-compose.dev.yml build
   docker compose -f docker-compose.dev.yml up -d

2. For production:

   docker compose build
   docker compose up -d

3. Test the integration:

   # Test client_backend Sentry → Google Chat flow
   curl https://client-backend.oktivo.com/debug-sentry
   
   # Test Analysis Sentry integration
   curl https://client-backend.oktivo.com:8001/debug-sentry

4. Within ~30 seconds you should see:

   • The error in Sentry dashboard
   • An alert card in your Google Chat "Dev Oktivo" space

---

Step 6: Clean Up

After verifying everything works, remove the test endpoints:

• Delete /debug-sentry route from app.ts (client_backend)
• Delete /debug-sentry route from app.py (Analysis)

---

What The Google Chat Alert Looks Like

┌──────────────────────────────────────────┐
│ 🚨 Sentry ERROR: oktivo-client-backend   │
│ Production Error → Google Chat           │
├──────────────────────────────────────────┤
│ Error: Cannot read property 'x' of null  │
│                                          │
│ Culprit: POST /campaigns/start           │
│ Environment: production                  │
│ Project: oktivo-client-backend           │
│                                          │
│ [View in Sentry]                         │
└──────────────────────────────────────────┘

---

Production Checklist

#	Action	Where	Status
1	Create webhook in Google Chat space	Google Chat	✅ Done
2	Set GOOGLE_CHAT_WEBHOOK_URL in env	shared.env	✅ Done
3	Set SENTRY_DSN in env	shared.env	✅ Done
4	Create Internal Integration in Sentry	sentry.io	⬜ TODO
5	Set SENTRY_WEBHOOK_SECRET in env	shared.env	✅ Done
6	Create alert rules in Sentry	sentry.io	⬜ TODO
7	Add env vars to AWS Secrets Manager (prod)	AWS Console	⬜ TODO
8	Deploy & test	Production server	⬜ TODO
9	Remove /debug-sentry endpoints	Code	⬜ After testing

---

Key URLs

Service	Production URL
client_backend	https://client-backend.oktivo.com
Sentry webhook endpoint	https://client-backend.oktivo.com/sentry-webhook
Sentry debug/test endpoint	https://client-backend.oktivo.com/debug-sentry
Google Chat space	Dev Oktivo
Sentry dashboard	https://sentry.io

---

Troubleshooting

• Sentry not receiving events: Check SENTRY_DSN is set correctly. Look for [sentry] ✅ Initialized in container logs.
• Google Chat not receiving alerts: Verify GOOGLE_CHAT_WEBHOOK_URL is set. Check client_backend logs for [sentry-webhook] messages.
• HMAC validation failing: Ensure SENTRY_WEBHOOK_SECRET matches the Client Secret in Sentry Integration settings. Or remove the env var to skip validation (safe for now).
• Too many alerts: Adjust tracesSampleRate in instrument.ts / sentry_setup.py and alert rule frequency settings.
• Docker build failing: Run npm install locally in client_backend/ to regenerate package-lock.json, then rebuild.

---

Files Modified/Created

Service	File	Purpose
client_backend	src/instrument.ts	Sentry SDK init (NEW)
client_backend	src/server.ts	Import instrument + flush on shutdown
client_backend	src/app.ts	Sentry error handler + import
client_backend	src/routes/sentry-webhook.routes.ts	Google Chat relay (NEW)
client_backend	package.json	Added @sentry/node, @sentry/profiling-node
Analysis	sentry_setup.py	Sentry SDK init (NEW)
Analysis	app.py	Import sentry_setup + flush on shutdown
Analysis	requirements.txt	Added sentry-sdk[fastapi]
Agent	sentry_setup.py	Sentry SDK init (NEW)
Agent	agent.py	Import sentry_setup + capture exceptions
Agent	requirements.txt	Added sentry-sdk
Shared	docker-compose.yml	SENTRY_DSN env for all services
Shared	.env.example	Documented new env vars
Shared	env/shared.env	Actual env values
Docs	SENTRY-SETUP.md	This file