Preface

The capabilities of Large Language Models (LLMs) need no further introduction, but "being able to chat" and "being able to call" are two completely different things.

Before this year, to truly integrate AI into real-time voice conversation scenarios, you typically had to stitch together three separate pipelines: STT (Speech-to-Text) → LLM (Language Model) → TTS (Text-to-Speech). You also had to handle voice activity detection (VAD), sentence splitting, interruptions, concurrency, and other edge cases. One slip-up, and the AI would start talking over the user when they just said "uh-huh," or the recognition latency would stack up to 2-3 seconds, turning a voice call into a walkie-talkie experience.

That was until I discovered the LiveKit Agents framework.

This article is not intended as a tutorial — there are already plenty of those online. What I want to share is: the typical challenges and solutions encountered when embedding LiveKit Agents into a backend architecture within real-world IM/calling services.

Selection: Why LiveKit instead of Twilio / Agora

First, a quick background: I needed to add AI voice assistant capabilities to an existing system that already has IM and WebRTC calling capabilities.

There are roughly three categories of options available on the market:

The reason for choosing LiveKit Agents is simple: it has the best Agent API abstraction layer.

LiveKit Agents abstracts the action of "an AI assistant joining a real-time audio/video room" into a standard Job/Worker model:

Room Created → LiveKit Server Publishes Job → Worker Consumes Job → Agent Joins Room → Conversation Starts

This model naturally fits my existing IM calling flow —— the logic for creating call rooms was already written in my system, so now all we need to do is have the AI join as one of the participants after the room is created.

Architecture Breakdown

The actual deployment architecture is as follows:

Client (Browser/App)
    ↕ WebRTC Audio Stream
LiveKit Server (Self-hosted)
    ↕ Audio Stream
Python Agent Worker (LiveKit Agents)
    ↕ HTTP (Fetch Context + Push Logs)
Go Main Backend (Business Services)

There is only one key design principle: The Agent does not handle business logic; it only handles the "Speech ↔ Text" conversion.

All business knowledge, user information, and chat history are fetched from the main backend via HTTP APIs and injected into the LLM as a System Prompt. The text recognized by the Agent and the AI's responses are also written back to the main backend via HTTP for persistence.

The Agent's entrypoint code is very clear, with the core flow consisting of four steps:

async def entrypoint(ctx: JobContext):
    room_name = ctx.room.name

    # 1. Fetch current room context from business backend
    room_context = await fetch_room_context(room_name)
    system_prompt = build_system_prompt(room_context)

    # 2. Connect to the room and wait for user to join
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
    participant = await ctx.wait_for_participant()

    # 3. Build Agent, inject LLM + VAD + TTS
    agent = Agent(
        instructions=system_prompt,
        llm=google.realtime.RealtimeModel(
            api_key=google_api_key, voice="Aoede",
        ),
        tts=google.beta.GeminiTTS(api_key=google_api_key),
        vad=ctx.proc.userdata["vad"],
        allow_interruptions=True,
    )

    # 4. Start the session
    session = AgentSession()
    await session.start(agent, room=ctx.room)
    await session.say("AI assistant has entered the room")

The framework encapsulates all low-level details of audio streaming — WebRTC connections, audio codecs, end-to-end latency control — letting developers focus solely on the LLM's instructions and event callbacks.

Three benefits of doing this:

• Keep the Agent lightweight and replaceable —— If you want to change STT/TTS providers in the future, or even switch the AI engine from LLM to other models, the Agent code changes will be minimal.
• Unified business logic —— Management of all users, permissions, and history remains in the main backend; the Agent does not need to understand business rules.
• Decoupled deployment —— Agent Workers can scale independently; you can spin up more Workers during high voice traffic without affecting the main backend.

Full-Duplex vs Half-Duplex: An Underestimated Detail

Most early AI voice solutions were half-duplex —— the user talks, then the AI responds, and whoever finishes speaking "releases the microphone." This is highly unnatural in interaction; in real human conversations, people don't wait for the other person to completely finish speaking before uttering a word.

LiveKit Agents introduced the allow_interruptions mechanism in version 1.x, which, paired with the VAD module, achieves true full-duplex conversation. In practice, it accomplishes three things:

1. Continuous VAD (Silero) Detection —— Dynamically determines whether the user is speaking and when the user finishes.
2. Streaming TTS Support —— The AI's response is played while being generated, instead of waiting for the entire sentence to complete.
3. Interruption Logic —— Once the user starts speaking, the AI's playback stops immediately, waiting for the user to finish before responding.

The collaboration of these three mechanisms determines the "conversational fluidity" perceived by users. The most common problem I encountered during the tuning phase was: VAD sensitivity being too high, causing frequent interruptions to the AI, or too low, causing the AI to freeze after the user finished speaking.

The VAD model needs to be pre-loaded into memory when the Worker starts, avoiding downloads for every call:

def prewarm(proc: JobProcess):
    """Executed once when the Worker process starts, preloading the Silero VAD model"""
    proc.userdata["vad"] = silero.VAD.load()

Then register it in the Worker entrypoint:

if __name__ == "__main__":
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            prewarm_fnc=prewarm,
        )
    )

The prewarm mechanism is a neat design of LiveKit Agents. Loading voice processing models usually takes several hundred milliseconds to a few seconds. If loaded only when each call starts, the user's first voice interaction will experience a noticeable stutter. By preloading at the process level, the very first word can be accurately detected.

The ultimate parameter experience is: VAD thresholds and silence timeout values must be tuned according to the speech rate and background noise of the actual usage scenarios. The default parameters in the documentation only yield a runnable result, which is still a long way from being "production-ready."

Critical Decision: Choose Multimodal or Split "Three-Piece" Architecture

There is a point worth discussing in the Agent's architectural design: Should the LLM handle audio input and output?

• Option A (Traditional Split Solution):
  User Audio → STT (Speech-to-Text) → LLM (Generate Text) → TTS (Text-to-Speech) → Playback

• Option B (Multimodal Solution):
  User Audio → Multimodal LLM (Directly understand audio and generate audio) → Playback

This decision affects latency, cost, and code complexity.

My current practice is: using the multimodal RealtimeModel solution. The advantages of doing so are obvious:

• Drastic reduction in code volume —— No need to register separate STT and TTS components; the LLM handles input and output all at once.
• Lower latency —— Removes the text serialization and deserialization steps.
• Natural support for emotion and tone —— Multimodal models can perceive user intonation, making the responses feel more "human."

However, the multimodal solution has its trade-offs: the choice of models is narrow, and the dependency on cloud service providers is high. If you want to switch to locally deployed models in the future, the multimodal option is currently not very practical.

Conversation Context Synchronization Issue

This was the trickiest engineering problem encountered in practice.

When the AI speaks in a call, that part of the conversation needs to be synchronized to the IM message records so that users can see the "history." But if a voice call is active between the AI and the user, what should the messages they see on IM look like?

My final design was:

• Agent pushes transcribed content to the backend in real-time —— After each sentence spoken by the user is recognized, and after each response from the AI is generated, it is written back asynchronously to the backend via HTTP.
• Backend persists it as "messages" —— It shares the same message table with text messages in IM, flagged with a "voice transcription" identifier.
• Allows summary processing when the call ends —— Long calls need to generate summaries, which are pushed to participants as special messages.

In terms of implementation, this is done via an asynchronous HTTP push in the Agent's event callback:

@session.on("conversation_item_added")
def on_item_added(ev):
    item = ev.item
    if not isinstance(item, llm.ChatMessage):
        return

    content = item.text_content
    if not content:
        return

    if item.role == "user":
        # User utterance -> push to business backend for persistence
        asyncio.create_task(
            push_utterance(room_name, participant.identity, content)
        )
    elif item.role == "assistant":
        # AI response -> also push to business backend for persistence
        asyncio.create_task(
            push_utterance(room_name, "ai_assistant", content)
        )

And push_utterance itself is a stateless HTTP POST:

async def push_utterance(room_name: str, identity: str, content: str):
    url = f"{backend_url}/api/agent/utterance"
    payload = {
        "room_name": room_name,
        "speaker_identity": identity,
        "content": content,
        "language": "zh-CN",
    }
    async with session.post(url, json=payload, timeout=aiohttp.ClientTimeout(total=5)) as resp:
        if resp.status != 200:
            logger.warning(f"Push failed: {resp.status}")

Looking at these two snippets alone, it seems simple, but in practice, there are several traps:

• Fan-out issues in async calls —— An Agent might process multiple rooms simultaneously. Async tasks in callbacks must have robust exception handling, otherwise, a sentence might be dropped.
• Ordering guarantee —— HTTP requests are asynchronous, and arrival order cannot be guaranteed. However, IM messages need to be ordered, so the backend must re-order based on the frontend timestamps.
• Retry and Idempotency —— Network jitters can cause the same utterance to be pushed multiple times, so the backend must perform deduplication.

Deployment and O&M Experiences

For Agent Worker deployment, we use Docker. Setting network_mode: host was a critical decision:

Every hop in a voice stream increases latency. If you use the bridge network mode, audio data passes through NAT translation inside the container, which is non-negligible in bandwidth-sensitive real-time scenarios. The host mode directly shares the host's network stack, reducing round-trip latency by about 5-10ms.

Additionally, skipping SSL verification for self-signed certificates is worth mentioning.

When LiveKit Server is deployed on an intranet, self-signed certificates are almost always used, while Python's aiohttp/urllib validates SSL by default. Passing ssl=False in every single request is too tedious. Doing a monkey-patch at the module level is the easiest approach:

import ssl
import aiohttp

# Skip SSL verification globally (only for intranet self-signed certificate scenarios)
ssl._create_default_https_context = ssl._create_unverified_context

_original_init = aiohttp.TCPConnector.__init__

def _patched_init(self, *args, **kwargs):
    kwargs["ssl"] = False
    _original_init(self, *args, **kwargs)

aiohttp.TCPConnector.__init__ = _patched_init

Although monkey-patching is not elegant, it is the most practical solution in a trusted intranet.

For Docker deployment, network_mode: host is a crucial parameter:

services:
  axiom-agent:
    build: .
    env_file:
      - .env
    network_mode: "host"          # Directly reuse the host's network stack
    deploy:
      resources:
        limits:
          memory: 1G
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Every hop in a voice stream increases latency. Under bridge mode, audio data passes through container NAT translation; host mode directly shares the host's network stack, reducing round-trip latency by about 5-10ms. For real-time voice, every millisecond counts.

Cost Analysis

The cost structure of using a multimodal model is slightly different from the traditional split solution. Under the traditional scheme, the approximate cost of an active call hour is:

• STT (Speech-to-Text): ~ $0.26
• LLM (Language Model): ~ $0.50
• TTS (Text-to-Speech): ~ 0.30 **Total: ~ 1.06/hour**

Under the multimodal approach, the costs of STT and TTS are "bundled" into the LLM API calls. Although the unit price seems more expensive, it saves the overhead of serializing multiple services and network round-trips, making the actual overall cost potentially lower.

Of course, these numbers are only estimates, depending on conversation density, model version, and API pricing changes. But as a reference, keeping the cost of an AI call hour within one to two dollars is fully acceptable for many B2B scenarios.

Future Exploration Directions

The current solution has successfully implemented the basic closed loop, but there are still several areas worth exploring further:

• More Natural Interruptions —— While VAD-based interruptions are usable, in multi-person conversation scenarios, the AI doesn't know "when to chip in" and currently can only wait passively for the user to finish speaking. If more granular conversation state management can be introduced, the AI's presence will feel much stronger.
• TTS Emotional Control —— Current TTS supports tone adjustment via parameter tuning, but there is no link with the content of the LLM replies yet. The experience would be completely different if the AI could speak more softly when comforting users and more firmly when confirming details.
• Long Conversation Token Management —— For calls exceeding one hour, the context becomes very long, increasing token consumption and model response latency. Currently, simple sliding window strategies lose early information, requiring better summarization and memory mechanisms.
• VAD and Background Noise Reduction —— Background noises (keyboard clicks, ambient chatter) in real-world environments affect VAD accuracy, and can even trick the Agent into thinking it is being "interrupted." Adding a layer of noise reduction preprocessing could significantly improve the experience.

Finally

LiveKit Agents is an outstanding framework that advances AI voice conversation from a "lab toy" to a "production-ready tool."

However, it must be acknowledged that the framework only solves the "how to connect" problem, not the "how to make it work well" problem. What truly determines user experience is the repeated tuning of VAD parameters, how conversation contexts integrate with existing business flows, and every millisecond of latency optimization —— all of which must be refined in specific business scenarios.

If you are doing similar work, feel free to exchange ideas. AI voice assistants are still in their infancy, and the pitfalls and thoughts in this direction deserve more people to explore together.