voice-agent
Voice Appointment Agent
A phone line that answers itself: the caller speaks normally and the program books, moves, or cancels an appointment, asking for whatever details are missing and never giving away a slot that is already taken. It was proved by driving a real three-turn booking over the open internet into a real calendar entry; the live phone number was never used, so no human voice has tested it yet.
A three-turn booking conversation was driven through a real HMAC-SHA1-signed request (the same algorithm Twilio uses, computed from the real auth token) over a public internet tunnel, ending in a real row written to the SQLite calendar. The live phone number itself was left untouched throughout.
Overview
A phone agent that answers a real Twilio line and books, reschedules, or cancels appointments in natural conversation. Gemini handles the language, a SQLite calendar handles real conflict detection, and the whole multi-turn flow has been proven end to end over a real public tunnel with a genuine Twilio signature, not just unit-tested in isolation.
Most voice-agent demos show a scripted happy path. This one is built to survive the actual shape of a phone call: silence instead of speech, a caller who changes their mind mid-sentence, a time slot that conflicts with an existing booking, and a caller who never answers a question clearly. The conversation state machine handles all of it, backed by a real calendar with real conflict detection, not a mocked one.
The system prompt to production Gemini calls follow the same discipline as the rest of this portfolio: everything that can be tested offline is tested offline, and the small number of things that must touch a live, billed service are marked, minimal, and their real cost is reported honestly.
Approach and architecture
Twilio calls a FastAPI service for TwiML instructions on every turn. A conversation manager holds per-call state (slot filling for name, time, and phone; a confirmation step; retry and turn caps) and calls a Gemini-based intent engine to interpret free-form speech into a structured action: book, reschedule, cancel, or unclear. A SQLite calendar store enforces real half-open-interval conflict detection so two appointments can touch at a boundary without falsely conflicting, and a session store logs every turn with a timestamp and the eventual outcome.
Key features
- Multi-turn slot filling for booking, rescheduling, and cancelling, with an explicit confirmation step before anything is written.
- Real conflict detection: overlapping and back-to-back adjacent appointments are handled correctly, verified with an explicit boundary test.
- Business-hours and past-date rejection, with a path for the caller to correct a bad time and try again.
- A retry cap and an absolute turn cap that hand off to a human message instead of looping forever on unclear speech.
- An outbound test-call CLI that refuses to run without a public URL and requires explicit confirmation before it places a real, billed call.
- A fully implemented (not stubbed) Google Calendar adapter behind the same interface as the default SQLite store, used only if calendar credentials are supplied.
Results
| Metric | Result |
|---|---|
| Full test suite | 54 / 54 passed |
| Scripted evaluation scenarios | 25 / 25 passed |
| Real public-tunnel booking conversation | 3 / 3 signed turns accepted, 1 real appointment written |
| Real Gemini calls (build + live verification) | 12 total, under $0.01 |
| Real Twilio cost incurred | $0.00 (read-only lookups only, no call placed) |
Reading the numbers honestly
The 25 scripted scenarios run against a deterministic stand-in for Gemini, documented in its own code as a test double, not a production fallback, specifically so a 100+-call evaluation harness doesn't quietly become a recurring bill. I deliberately broke the retry-cap and turn-cap logic once to confirm those two scenarios actually fail when the invariant is broken, then reverted it, since a scenario suite that cannot fail is not proof of anything.
The real public-tunnel test is the one piece of evidence in this project that isn't a unit test: a genuine HMAC-SHA1 Twilio signature, computed from the real auth token, was sent over the actual public internet to a live-booted instance of the app, and Gemini correctly resolved a relative date ("tomorrow") against the real current date across three turns before writing a real database row. The one thing that test still doesn't cover is an actual human voice call: Twilio's real speech-to-text output is noisier than the clean, hand-typed text used here, and that gap is stated plainly rather than glossed over.
Tech stack
Challenges
- Twilio only calls the action URL on captured speech by default; on silence it just moves on, which would have made the unclear-speech and retry logic unreachable. Fixed by setting
actionOnEmptyResult="true"on every Gather, so silence and speech both return through the same state machine. - Twenty-five multi-turn scripted scenarios would be well over a hundred real Gemini calls if run against the live model every time. Solved with an interface-based intent engine: a real Gemini implementation for production and a small number of marked tests, and a deterministic stand-in for the scenario harness.
- Gemini's default thinking mode was measurably too slow for a phone turn (56 thinking tokens on a one-word answer). Disabling thinking and forcing a strict JSON response schema brought a real call down to 1.8 seconds.
- Relative dates and business-hours checks silently broke test determinism until "now" was threaded through the conversation manager as an injectable function, letting every scenario pin to one fixed reference time instead of drifting with the real clock.
What I learned
- Conflict-free interval overlap is a half-open interval check, which is what correctly allows two appointments to touch at a boundary without a false conflict.
- A latency-sensitive product built on an LLM means actively turning features off: disabling thinking and forcing a response schema mattered more than any prompt wording.
- A passing test is not proof of anything by itself; I had to actually break two safety-cap scenarios on purpose to confirm they would catch a real regression.
- Simulating a correctly-signed request over a real public tunnel is a meaningfully stronger proof than a mocked unit test, and it is achievable without ever touching a live, already-configured production webhook.