How we made our voice AI 79% faster without a faster model

We made the voice replies in our AI tour guide 79% faster, from 8.6 seconds to about 1.8 seconds until the first spoken word, without switching to a faster AI model. No single step was slow. Every step waited for the previous one to finish. So we streamed the audio while the user talks, spoke the answer sentence by sentence, kept the conversation history ourselves, fetched the location only when needed and started the search before the user stopped talking.

What was the problem?

When you ask a tour guide a question, you expect an answer before you have walked past the building. Our AI guide took 8.6 seconds.

In our app Guidude, you walk through a city, ask a question out loud, and the guide answers by voice: “Who lived in this house?”, “Is there a good bakery nearby?” The answers were good, but they arrived too late. Eight seconds of silence is long enough to wonder whether the app heard you at all, and by then you have often walked past the place you asked about.

We rebuilt the voice pipeline. The guide now starts speaking about 1.8 seconds after you tap Stop. Here is what we changed and why.

Where did the 8.6 seconds go?

We started by measuring. Before changing any code, we timed each step of a voice question:

Step	Time
App uploads the recording to cloud storage	0.5 s
Backend downloads it again	0.2 s
Transcribe the whole recording	1.5 s
Look up location and search our knowledge base	1.0 s
AI writes the full answer	3.9 s
Convert the full answer to speech	1.4 s
User hears the first word	~8.6 s

No single step was slow. The problem was the design: every step waited for the previous one to finish completely. The recording had to be uploaded before it could be transcribed. The whole answer had to be written before any of it could be spoken.

The fix was not one optimisation. It was to stop waiting.

How did we make it faster?

1. Stream the audio while the user talks

The old app recorded the question, uploaded the file, and sent the backend a reference to it. The backend downloaded the file and transcribed it.

Now the app streams raw audio over the chat’s WebSocket connection while the user is still talking:

• The backend forwards each piece of audio to a live transcription session as it arrives. By the time the user taps Stop, almost everything is already transcribed.
• The final transcript arrives about 0.4 seconds after Stop, where upload, download and transcription used to take about 2.3 seconds.
• The user sees live captions while they talk, so they know the app is listening.
• We no longer store the recording. We used to keep recordings so they could be replayed. Now the audio is transcribed and discarded, and only the text is saved. It is faster, and it is better for privacy.

Before building anything, we tested live transcription in all 11 app languages. It was faster and more accurate than our old batch transcription. The old one heard “New Haven” when people said Nyhavn.

2. Speak the first sentence while the AI writes the rest

The second-largest delay was the answer itself: the AI wrote the full reply, then all of it was converted to speech, uploaded, and sent to the app.

Now the backend works sentence by sentence:

• As the AI streams its answer, the backend cuts it into sentences.
• Each sentence goes to text-to-speech as soon as it is complete, with several sentences processed in parallel.
• Each audio piece is sent to the app straight away, numbered so it plays in the right order.
• While the user listens to the first sentence, the rest is being written and spoken.

The user no longer waits for the whole answer, only for the first sentence. Converting one sentence to speech takes 0.1–0.3 seconds.

3. Keep the conversation memory ourselves

We used an AI API that stored the conversation history on the provider’s side. It was convenient, but our measurements showed it added 2–3 seconds to every reply.

We already saved every message in our own database, so we now send the recent conversation with each request:

• First words of a text reply now appear in under a second, where they used to take about 3.5 seconds.
• We tested how much history to send. With 4 messages, the AI forgot a bakery it had recommended 10 messages earlier and invented a different one. With 10 or 20 it remembered, and the speed was the same. We chose 14.
• As a side effect, the AI now remembers its own place descriptions, so “tell me more about that” works after it describes a landmark.

4. Fetch the location only when the question needs it

We used to look up the user’s address and nearby places for every question, even “When was this church built?” That cost up to a second, and the AI often mentioned the location when it didn’t matter.

Now the AI gets a tool it can call when the answer depends on where the user is. In our tests it called the tool for 16 of 16 location questions and for none of the 8 others. Most questions got faster, and the answers stopped drifting to the street name.

5. Search before the user finishes talking

The last step on the waiting path was the search in our knowledge base of curated city content, about half a second.

Since live captions arrive while the user talks, we start the search on the captions. When the user taps Stop, we check that the final transcript still matches what we searched. If it does, the results are already waiting. If it doesn’t, we search again. This brought the first spoken word from about 2.2 seconds down to about 1.8.

What was the result?

	Before	After
Voice: first spoken word after Stop	~8.6 s	~1.8 s
Text: first words on screen	~3.5 s	under 1 s
Voice recordings stored	Yes	No
AI voice replies stored	Yes	No

We also added a Read aloud button to every AI message. It runs the saved text through the same sentence pipeline, and audio starts in about half a second.

What can you take from this?

1. Measure each step before optimising. Our biggest win, the conversation history, was a delay we would not have guessed.
2. Stream instead of wait. Most of the 8.6 seconds was steps waiting for the previous step to finish.
3. Start work before the user finishes. Transcription and search now run while the user is still talking.
4. Test in every language you support. A pipeline that works in English can fail in Danish or Chinese.
5. Don’t store what you don’t need. Dropping the recording upload made the app faster and better for privacy.

Is your AI product too slow?

This is how we work at InitHere: measure first, change what the numbers point to, and verify the result. The same approach works for any slow product, not only voice: a chatbot that makes users wait, an AI feature that times out, or a system that slows down as you grow.

• Scaling and performance: we find what makes your software slow and fix it, with before-and-after measurements.
• AI integration: AI assistants, smart search and automation built into your existing systems, including voice.

Book a free 30-minute call and tell us what feels slow.

FAQ

Do you need a faster AI model to make a voice assistant faster?

Usually not. In our case the model was not the problem. Most of the 8.6 seconds came from steps waiting for each other: upload before transcription, the full answer before any speech. Streaming each step and overlapping them cut the wait to about 1.8 seconds with the same models.

How fast should a voice AI answer?

Fast enough that the silence does not feel like a failure. At 8.6 seconds our users wondered whether the app had heard them. At about 1.8 seconds, with live captions while they talk, it feels like a conversation.

Does streaming transcription reduce accuracy?

Not in our tests. We compared live transcription with our old batch transcription in all 11 languages of the app before switching, and the live version was both faster and more accurate.

Is it safe to stop storing voice recordings?

For us, yes: the app only needs the text of the question to answer it and to keep the conversation history. Not storing the audio removed an upload step and means we hold less personal data.