AI Voice Coach with ElevenLabs

In this workshop we build Luma, a voice coach for rehearsing difficult conversations. You talk, and the coach walks you through Nonviolent Communication. By the end, a tangled conflict has become a clear message you can say out loud.

Links

Useful resources for this workshop:

• The investment coach bot - another conversational coach, with the evaluation setup we would reuse for Luma.
• The eval framework workshop - how to score agent output in batches, the foundation for voice-quality evaluation.
• The coding agent workshop - goes deeper into driving a coding agent the way we drive Codex here.

The app you will build

A voice agent has stricter requirements than a text chat. It has to listen, decide when you finish talking, and reply without sounding robotic. It also has to let you interrupt it mid-sentence, like a real person would. Building that pipeline from scratch takes a long time, so we take a shortcut. We use ElevenLabs Conversational AI, a managed voice-agent platform that gives us the whole pipeline as one service.

The trade-off matters, so let's name it. A managed platform gets us to a working prototype fastest. Use it while you're still asking whether voice helps at all. Switch to a custom pipeline once you need control over latency, cost, or the model behind the coach. We pick the managed path on purpose, because testing the idea matters more than building infrastructure.

The browser never sees the ElevenLabs API key. A small Next.js server holds the key and gives the browser a short-lived signed URL for each session. The secret stays on the server while the audio streams straight to ElevenLabs.

We create the agent, including the coaching prompt that makes it behave like Luma, through the ElevenLabs API. We give the coding agent the API key, and it provisions the agent for us. We skip the dashboard entirely. Under the hood ElevenLabs runs LiveKit for the real-time audio. The same connection problems you would debug in LiveKit show up here too.

Once the voice loop works, we add the things a real product needs. We track how much each session costs by reading ElevenLabs' recorded spend, and we let the user pick a voice. There's one more habit to borrow. Before we declare anything done, we make the coding agent test the whole flow with a headless browser. That way we're not the ones clicking through the UI to find out it's broken.