I built Nova Suraksha Voice Agent, called voiceagent on my @NitinSukthe-G GitHub, as a Python terminal agent for hospital appointment booking. Its assistant, Priya, listens through a microphone and speaks replies through speakers or headphones. I started and last updated the project in 2026-09. It is one of the projects on my github profile.
I also have a separate Python repository called VOICE-AGENT on my @NitinSukthe-G GitHub. Unlike my hand-wired terminal agent described above, this is a multilingual booking demo that runs locally in the LiveKit Agents Playground and books appointments into MongoDB. It supports Hindi, Telugu, English, and Hinglish; I started it and last updated it in 2026-09.
For this demo, I use LiveKit Cloud for the agent runtime, Sarvam for speech-to-text, Google AI Studio for the language model, Cartesia for text-to-speech, and MongoDB Atlas for bookings. The README describes the listed services as free with no card required unless noted. Atlas setup uses a free M0 cluster, a database-access user, and network access allowing my IP; it also gives <redacted>/0 as an option for local development. I get the connection string through Connect → driver connection string. The expected booking destination is the voice_agent.bookings collection.
cp .env.example .env
uv venv && source .venv/bin/activate
uv pip install -r requirements.txt
python agent.py dev
# Open the LiveKit Agents Playground link printed by the command,
# connect, and talk in the browser; a microphone is required.
# Suggested test: “I'd like to book an appointment for tomorrow at 3pm, my name is ...”
Deployment is intentionally out of scope for this demo and handled separately by me. Deferred work includes multilingual prompt tuning beyond “reply in the caller’s language,” provider failover, per-turn latency and cost logging, and real phone calls through Twilio. The README warns that the livekit-agents plugin API changes quickly: if the agent fails on AgentSession, Agent, or plugin constructors, I should check the current LiveKit Agents documentation for renamed arguments; it characterizes this as a quick fix rather than a rewrite.
I hand-wired the real-time pipeline in Python with asyncio, rather than using LiveKit, Pipecat, Vapi, or another voice-agent framework: microphone → VAD → STT → LLM → TTS → speaker. The pipeline diagram describes 20 ms microphone frames; VAD checks whether the user is talking, STT converts speech to text, the LLM decides what to say or which tools to call, and TTS turns the reply into audio. I included interruption handling, tool calling, and per-turn latency instrumentation so I can understand, measure, and tune each stage.
I use a cascading, streaming pipeline for voiceagent: sounddevice captures local 16 kHz mono audio in 20 ms frames; local webrtcvad plus an adaptive loudness gate detects speech; Sarvam saaras:v3 transcribes over WebSocket with REST fallback; Sarvam sarvam-105b-conversations streams tokens and tool calls; and Sarvam bulbul:v3 streams Priya’s speech over WebSocket to a sounddevice player with an instantly clearable buffer. I chose this over speech-to-speech because the intermediate text exposes the booking rules and tool calls for inspection. Streaming overlaps transcription, model generation, and sentence-level speech: Priya can begin sentence one while the model is still writing sentence three.
I measure latency from the end of the caller’s speech to the first audible reply word. Emergency responses take about 0.8 s because they are pre-synthesized and make no API call; “One moment.” takes about 1.7 s and starts when a tool call begins; full answers take 1.7–3.0 s, with the best turns at 1.65 s. Direct component measurements put the floor at about 250 ms for STT + 950 ms for the LLM + 450 ms for TTS ≈ 1.65 s, plus a 450 ms endpointing wait. The LLM’s roughly 950 ms to first token is the main latency floor; there is one usable conversational API model, no prompt caching, and TTS is already at its fastest accepted settings. I pre-synthesize fixed phrases, start tool-call filler on the first streamed tool-call chunk, and warm the LLM connection at startup to avoid a 2.5 s first-turn TLS handshake. Further measurements and rejected approaches are documented in latency.md.
For barge-in, I cancel the reply task, synchronously detach the TTS socket so no more audio arrives, and clear the player buffer; missing any one action lets the agent keep talking over the caller. If Priya has not spoken yet, I retain a partial transcript and merge it with the next speech, so a pause in “I want to book… with the heart doctor” does not split the utterance. Speaker echo can peak as loudly as real speech, so I use duration rather than a simple loudness threshold: at RMS ≥ 800, echo lasted no longer than 160 ms while real speech sustained 600–800 ms. A speaker-mode interruption must therefore stay loud for 300 ms continuously. There is no acoustic echo cancellation; headphones allow instant, sensitive interruption, while speakers require a firm voice. VAD also cannot distinguish the caller from a nearby person talking, and recognition accuracy varies with accent and speaking rate; speaking more slowly helps.
Sarvam streaming STT returned nothing for clips under about one second: “hi” and “ok” failed 24 times out of 24. I route short turns directly to REST, which transcribed every tested short clip in about 300 ms, and retain REST as fallback when streaming drops a turn; after routing, 10 of 10 short words were transcribed in about 400 ms. Business rules live in Python rather than the prompt: the model cannot access the database, emits structured tool calls, and must relay results after Python validates the rules. The rationale is that a model can be talked into anything, whereas validate_slot() cannot. tools.py contains six booking functions. I keep the tunable values in config.py: SILENCE_MS = 450 (lower values split natural pauses); START_MS = 120 (200 ms missed “hi” and “ok”); MIN_RMS = 200 with the adaptive gate rising to 4× rolling room noise; BARGE_RMS / BARGE_START_MS = 800 / 300; HISTORY_MESSAGES = 40 (12 lost the chosen doctor before the phone number arrived); and SPEAKERS = auto, which can be set to speaker or headphone mode.
I keep the implementation divided by responsibility: agent.py handles the turn state machine, barge-in, and reply loop; config.py holds tunable numbers; audio_io.py provides Mic and Player; vad.py emits start/end events from 20 ms frames; stt.py, llm.py, and tts.py handle their respective Sarvam stages; tools.py contains booking functions; prompt.py contains Priya’s and the emergency desk’s prompts and the emergency check; db.py connects to MongoDB; transcript.py records conversations live; latency.py records per-turn timings; and seed.py loads hospital, doctor, and department data. All data lives in MongoDB, with nothing stored on disk: hospital holds one document with name, address, hours, and rule constants; doctors has one document per doctor with department, days, hours, and fee; departments describes treatments in plain language; bookings is the booking source of truth; appointments records BOOKED, RESCHEDULED, or CANCELLED actions; conversations stores each session’s timestamped lines and per-turn latency; emergencies records each escalation, what was said, when, and its status; and phrases stores fixed lines pre-synthesized to audio. I record conversations live, including tool calls, barge-ins, and transcription failures, so I can investigate failures from what was heard and said. This applies the reliability and timing practices I describe in ai agent engineering practices.
- Priya can book, reschedule, and cancel appointments for 8 doctors across 7 departments; explain what departments treat and suggest one for a symptom; and answer questions about timings, fees, address, and parking. She refuses medical advice, including diagnoses, medicine names, and dosages. For an emergency such as “chest pain,” she writes an escalation record for the emergency desk and answers in about 0.8 seconds without reaching the language model.
- I enforce booking rules in Python rather than trusting the language model: appointments can be booked within a 14-day window, with at least 30 minutes’ lead time; there are no Sunday or lunch-hour bookings; children under 14 go to Pediatrics; follow-ups are free within 7 days; and cancellations have a 2-hour cutoff.
The README’s database-sourced example checks general-medicine availability for D01 and D02 on 2026-09-21, returning 16 and 24 free slots respectively. It books Karthik, age 22, with Doctor Ramesh Varma at 12:15; the tool returns booking ID NS6855 and a fee of 500. Priya confirms the booking and asks him to arrive fifteen minutes early. The example describes Ramesh Varma’s slots as tomorrow morning from nine AM to twelve forty five and Doctor Kavitha Reddy’s as afternoon slots from two PM to eight PM.
I do not implement acoustic echo cancellation. At startup, the agent detects whether I am using speakers or headphones. With speakers, the microphone hears the agent, so interrupting takes a firm voice; with headphones, interruption is instant and sensitive. On first run, the agent synthesizes fixed phrases such as the greeting and emergency line and stores them so they play instantly on later runs. Pressing Ctrl+C stops a session and prints its latency table.
I need Python 3.10 or newer, a microphone and speakers or headphones, a Sarvam AI API key, and MongoDB. One Sarvam key covers speech-to-text, the language model, and text-to-speech; a free MongoDB Atlas cluster works. The README does not name the Sarvam model or specify the MongoDB schema, full doctor roster, hospital address, or latency-table format.
git clone https://github.com/NitinSukthe-G/voiceagent.git
cd voice-agent
python -m venv venv
venv\Scripts\activate # macOS/Linux: source venv/bin/activate
pip install -r requirements.txt
copy .env.example .env # macOS/Linux: cp .env.example .env
# Set SARVAM_API_KEY=your_sarvam_key_here
# Set MONGODB_URL=mongodb+srv://user:password@cluster.mongodb.net/
python seed.py # once: loads the hospital, doctors and departments
python agent.py # talk to it