A multi-tenant platform for building, configuring, and deploying real-time voice conversational AI agents. Organizations can create agents, define their behavior through prompts/governance/workflows, wire in external data, deploy them to telephony (inbound and outbound), and monitor every conversation — all config-driven, with minimal code.
The reference deployment ships a working telecom assistant ("Aria", an Airtel customer-care agent) that speaks Hindi, English, and Hinglish with sub-second latency and natural barge-in (interrupt) handling.
ai-workflow/
├── backend/ Python FastAPI — control plane: API, auth, agent/voice/model management, DB
├── frontend/ Next.js 16 — dashboard UI for building and managing agents
├── voice-agents/ Node.js + WebSocket — real-time runtime plane: STT → LLM → TTS pipeline
└── docs/
└── architecture.html Interactive system-architecture diagram (open in a browser)
Two planes, one platform. The
backend/is the control plane — it stores what an agent is (definitions, users, models, voices) in a database. Thevoice-agents/service is the runtime plane — it executes a live conversation over WebSocket. Thefrontend/is how humans configure the control plane.
A full, interactive version of the diagram below lives in docs/architecture.html — open it in any browser.
┌────────────┐ (outbound)
Client ─────► │ Task Queue │ ─► Outbound cron ─┐
└────────────┘ │ green = outbound flow
▼
User ─(inbound)─► Telephony/IVR ─► CDN ─► Haproxy ─► Telephony ─► Queue ─► Stream ─► Load ─► API ─► Voice
│ Load Connector (routing) Manager Bal. Gateway Agent
│ Balancer │ │ Core
│ │ │ │
(WebSocket proxy, Cached Resp. S3 (call ┌───────┴────────┐
edge routing) (Redis) recording) │ DATA GATEWAY │
│ Cassandra DB │
open telemetry ── logs · prompt tracing · latency metrics ─────────────────────────────────────┤ PostgreSQL │
└────────────────┘
Voice Agent Core ─► Client Data Integrations ─► CTA Manager ─► Queue ─► 3rd-party
│ (WhatsApp,
▼ email, DMs)
Client-side Data Layer ─► Analytics ─► CRM
(Database · API Gateway · CRM)
Flow colors (in the HTML diagram): white = system flow · blue = inbound (a person calls in) · green = outbound (the platform calls out) · dashed = monitoring/telemetry.
| Component | Role |
|---|---|
| Telephony / IVR | Bridges PSTN/SIP/WebRTC calls into the platform (socket, gRPC, WebRTC). |
| CDN | WebSocket proxying, edge routing, TCP optimization for low latency. |
| Haproxy + Load Balancer | Distribute connections across runtime instances. |
| Telephony Connector | Normalizes provider audio streams onto the internal WebSocket protocol. |
| Queue | Python routing/waiting-line manager — pairs callers with available agent workers. |
| Stream Manager | Owns the live session: audio chunks, partial transcripts, streaming responses. |
| API Gateway | Single auth/routing entry point; validates JWTs, routes to services. |
| Voice Agent Core | Runs the STT → LLM → TTS pipeline, governance, and tool/RAG calls. |
| Cached Responses (Redis) | Greeting cache, common-phrase TTS cache, session/context store. |
| S3 Storage | On-demand call recording and audio artifacts. |
| Data Gateway | Cassandra (high-volume conversations, transcripts, states) + PostgreSQL (users, preferences, configurations). |
| Open Telemetry | Logs, prompt tracing, latency metrics across the pipeline. |
| CTA / Client Data Integrations | Push results to client systems; pull live data during a call. |
| 3rd-party Integrations | WhatsApp, email, direct messages — post-call follow-ups. |
| Analytics → CRM | Reports, insights, and write-back to client CRMs. |
Implementation status. The real-time core (Stream Manager / Voice Agent Core), auth, and the configuration dashboard are implemented today (see subprojects below). The telephony, CDN, queue, Cassandra, S3, and integration tiers are the target production topology documented here as the platform roadmap.
Layered REST API: endpoint → service → repository → SQLAlchemy model, with Pydantic schemas for validation. Provides CRUD for agents, voices, LLM/STT/TTS models, realtime/custom agents, organizations, and users, plus full authentication.
Tech: FastAPI, SQLAlchemy 2.0, Pydantic 2.5, psycopg2, python-jose, passlib[bcrypt]
cd backend
python3 -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000Swagger at http://localhost:8000/docs. Key resources: auth, agents, realtime-agents, custom-agents, voices, models, llm-models, stt-models, tts-models, organizations, users.
App-Router dashboard for building agents. NextAuth.js wraps the backend's auth (email/password + Google), and server-side API routes proxy authenticated requests to FastAPI. The agent builder is a multi-step wizard with two templates: Realtime and Custom.
Tech: Next.js 16, React 19, TypeScript, Tailwind v4, NextAuth.js, shadcn/ui
cd frontend
npm install
npm run dev # http://localhost:3000Low-latency voice pipeline over WebSocket with streaming LLM/TTS and barge-in. STT/LLM/TTS are real provider integrations (Sarvam STT + Whisper fallback, OpenAI GPT-4.1, Sarvam TTS); the MCP "tools" and RAG knowledge base use built-in sample data.
Tech: Node.js 18+, Express, ws, onnxruntime-web, @ricky0123/vad-web, Vite, TypeScript
cd voice-agents
npm install
npm run dev # Frontend :3000 · Backend WS :8080/ws · MCP API :8080/api/mcpThe platform is best understood as a set of end-to-end workflows. Each one below describes the actors, the steps, and the components involved.
Goal: get a team onto the platform with the right permissions.
- A user signs up (
POST /api/v1/auth/signup) with email + password, or signs in with Google (POST /api/v1/auth/google). The backend hashes passwords with bcrypt and records anAuthProviderrow per method, so a single account can carry both email and Google login (account linking by matching email). - On success the backend issues a short-lived access token (30 min) and a long-lived refresh token (7 days). The refresh token is stored server-side and can be revoked.
- The frontend (NextAuth) stores both tokens in its session and silently refreshes the access token via
POST /api/v1/auth/refreshwhen it expires.logoutrevokes one token;logout-allrevokes every refresh token for the user. - Users belong to an organization (
user_type = ORG) or act as an individual (INDIVIDUAL). Roles map to access levels:- Admin — full access
- Builder — can configure agents and workflows
- Viewer — read-only
- Every protected endpoint runs through the
get_current_userdependency, which validates the JWT, checks the token type, and confirms the account is active.
Sign up / Google ─► Issue access + refresh tokens ─► Org + role assigned
│ │
└──────────► silent refresh on expiry ◄─────────────────┘
Goal: turn a blank form into a deployable agent definition. This is the dashboard's core job.
An Agent is a generic record (name, description, type, language, status, owner_type, owner_id). Status moves through Draft → Active → Disabled. Every agent is specialized by exactly one of two configuration tables, matching the two builder templates:
Template A — Realtime agent (a single bundled model handles STT+LLM+TTS):
- Persona & Scope — tone, personality, and what the agent may/may not do.
- Governance & Security — compliance rules (GDPR/HIPAA/internal), PII masking, restricted topics, auth requirements.
- Model — pick a realtime model, a voice, and a language. The builder fetches available models/voices live, and cascades the voice selection to populate that voice's supported languages.
- Submitting creates an
Agentrow and arealtime_agentrow (persona,scope,governance,security,voice_id,model_id) linked 1:1.
Template B — Custom agent (separately chosen STT, LLM, and TTS stages):
- STT Config — model + language.
- LLM Config — model + temperature slider.
- TTS Config — model + language + voice (filtered by the model's provider) + pace and expressiveness sliders.
- System — persona, system prompt, governance, guardrails.
- Submitting creates an
Agentrow and acustom_agentrow holding per-stage model IDs and tuning (llm_temperature,tts_pace,tts_temperature, language codes, etc.).
Builder wizard ──► Next.js API proxy (attaches Bearer token) ──► FastAPI
│
creates Agent + (realtime_agent | custom_agent)
Goal: construct a safe, contextual prompt for every single LLM turn. This is the intelligence layer.
Each request to the model is assembled from layers:
System prompt (voice rules: ≤2 sentences, no markdown, numbers as words, warm tone)
+ Base prompt / persona
+ Scope definition (what topics are in-bounds)
+ Governance rules (restrictions, compliance, refusals)
+ Dynamic context (conversation history + tool data + retrieved docs)
In the running runtime (voice-agents), buildSystemPrompt() enforces the voice rules and language ("reply only in the detected language"), the scope rule keeps the agent on Airtel topics, and a verification rule prevents it from asking for already-verified identity. Account data and retrieved knowledge are injected as tagged context ([Verified account: …], [Account data: …], [Relevant info: …]) so the model answers grounded rather than guessing.
Goal: a caller dials in and has a natural spoken conversation. (blue/inbound flow in the diagram.)
User audio → Telephony/IVR → CDN → Load Balancer → Telephony Connector
→ Queue (assign worker) → Stream Manager → Voice Agent Core
Inside the Voice Agent Core, every turn runs the streaming pipeline:
- Capture & VAD — the browser/telephony edge uses Silero VAD (ONNX-WASM) to detect speech start/end, with an echo-mute window so the agent's own audio doesn't trigger it.
- STT — buffered audio (≥ ~0.5s) is transcribed by Sarvam (Hindi/Hinglish) with Whisper fallback. A hallucination filter discards classic silence artifacts ("you", "thank you", short noise).
- Context build — in parallel: detect which MCP tool the query needs (bill, plan, data usage, network, recharge history, account), run RAG keyword retrieval over the FAQ knowledge base, and always inject verified account identity.
- LLM — OpenAI GPT-4.1 streams tokens. A sentence chunker emits each sentence (split on
.!?।, ≥3 words) the moment it completes. - TTS — each finished sentence is sent to Sarvam TTS (
bulbul:v2) immediately, so audio for sentence 1 plays while sentence 2 is still being written. - Playback — the client schedules audio buffers back-to-back via the Web Audio API for gapless speech.
This streaming overlap (token → sentence → speech) is what keeps perceived latency low.
Audio ─► STT ─► [tool + RAG + identity] ─► LLM (stream) ─► sentence chunker ─► TTS ─► gapless audio
Goal: let the user cut the agent off mid-sentence, like a real phone call.
The platform uses a generation counter per session to make interrupts clean and race-free:
- While the agent is speaking, incoming user audio is watched. If the user speaks past a threshold (server: >500ms; client VAD: immediate on confident speech-start), it's a barge-in.
- The session's generation is incremented (
nextGen). This instantly invalidates all in-flight work — the streaming LLM is aborted, the in-flight TTS HTTP request is aborted viaAbortController, and queued sentences stop. - Every audio chunk carries a
responseId(sessionId:generation). The client plays only chunks whoseresponseIdmatches the latest reset and discards stale audio from the aborted turn — so the old answer never bleeds over the new one. - An echo-mute window (~1.5s after the agent starts speaking) plus a barge-in cooldown (~1.2s) prevent the agent's own voice (via speakers) from falsely triggering an interrupt.
Agent speaking ──► user speaks ──► nextGen() ──► abort LLM + TTS ──► drop stale audio ──► listen
Goal: the platform places calls (reminders, surveys, collections). (green/outbound flow in the diagram.)
- A client/operator adds a task to the Task Queue (who to call, which agent, what context).
- An Outbound cron worker drains the queue on schedule and hands each task to the Telephony Connector, which initiates the outbound call.
- Once the callee answers, the call joins the same real-time pipeline as Workflow 4 — there is one conversation engine for inbound and outbound.
Client ─► Task Queue ─► Outbound cron ─► Telephony Connector ─► (dial) ─► live pipeline
Goal: ground responses in real, current data instead of static prompts.
- During a call, the Voice Agent Core calls tools (MCP) and RAG to fetch account specifics and policy answers, then injects them into the prompt (see Workflow 3).
- Client Data Integrations / CTA Manager connect to the client-side data layer (their Database, API Gateway, CRM) to read live customer data and write back outcomes.
- The target design supports file ingestion (PDF/Excel), API connectors, and DB connections as additional context sources.
- The builder saves an agent as Draft.
- A telephony number/route is mapped to the agent (target tier).
- Deploy flips the agent to Active — its configuration becomes callable.
- Disable takes it offline without deleting the definition. Configuration versioning lets you roll changes forward/back.
- Open Telemetry captures logs, prompt traces, and per-stage latency (STT, LLM, TTS).
- Conversations, transcripts, and states are written to Cassandra (append-heavy, time-series friendly); users/preferences/configs live in PostgreSQL.
- Call recordings are stored on-demand in S3.
- Analytics turns raw logs into reports and insights, with optional write-back to the client CRM and follow-ups over WhatsApp/email/DMs via 3rd-party integrations.
Inbound support for a telecom operator: data balance, recharge, bill payment, plan upgrades, network issues, SIM/port queries. Multilingual (Hindi/English/Hinglish), grounded in account data via tools, and policy answers via RAG. This is the agent that ships in voice-agents/.
Payment-due reminders, renewal nudges, and gentle collections via outbound campaigns (Workflow 6), with results written back to the CRM and escalations sent over WhatsApp/email.
Inbound scheduling assistants that read availability from a client API/DB during the call and confirm bookings, with barge-in so callers can correct details naturally.
Outbound surveys or qualification calls that branch on responses, capture structured answers, and push qualified leads into the client's sales pipeline/analytics.
First-line support that answers FAQs from a knowledge base (RAG), performs simple account actions through tools, and hands off to a human only when scope/governance requires it.
An organization builds several agents (different personas, scopes, voices, languages) under one account with role-based access — Admins manage, Builders configure, Viewers observe.
backend/.env
DATABASE_URL=postgresql://user:password@localhost:5432/ai-workflow
SECRET_KEY=your-jwt-secret
GOOGLE_CLIENT_ID=... # for Google OAuth verificationfrontend/.env.local
NEXTAUTH_URL=http://localhost:3000
NEXTAUTH_SECRET=your-secret
NEXT_PUBLIC_BACKEND_URL=http://localhost:8000
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...voice-agents/backend/.env
PORT=8080
OPENAI_API_KEY=sk-... # GPT-4.1 (LLM) + Whisper (STT fallback)
SARVAM_API_KEY=... # Sarvam STT (primary) + Sarvam TTSBoth backend/ and voice-agents/backend/ carry an agent.config.json documenting the agent's STT/TTS/LLM providers, model names, voice settings, VAD thresholds, and the system-prompt voice rules (≤2 sentences, no markdown, numbers as words, no filler).
- JWT auth with refresh-token rotation and revocation (see
backend/AUTHENTICATION.md). passlib[bcrypt]for password hashing;python-jose(HS256) for signing.- CORS is open in development — lock it down for production in
backend/app/main.py. - Keep all API keys in
.envfiles; never commit them. - Governance layer enforces scope, restricted topics, and PII handling per agent.
- Use TLS for all provider and client/server traffic in production.
Recommended metrics: API request rate & P95 latency per endpoint · STT latency and error rate · TTS synthesis latency and payload size · LLM tokens/cost per request · VAD false-trigger and missed-detection rates.
# Terminal 1 — Backend API
cd backend && source .venv/bin/activate && uvicorn app.main:app --reload
# Terminal 2 — Frontend UI
cd frontend && npm run dev
# Terminal 3 — Voice runtime (optional)
cd voice-agents && npm run devContributing
- API changes: update
backend/app/api/v1, add Pydantic schemas, updatefrontend/API_SCHEMA.md. - Frontend changes: follow patterns in
frontend/src/components; keep the bundle lean. - Voice-agent changes: document
agent.config.jsonchanges; keep field names backward-compatible.
See LICENSE in the repository root. If no license file is present, treat this code as private until one is applied.