[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-34":3},{"id":4,"title":5,"title_en":6,"abstract":7,"abstract_en":8,"content":9,"content_en":10,"category":11,"banner_id":12,"banner_path":13,"tags":14,"is_recommend":17,"prev_article":18,"next_article":22,"created_at":26},34,"我用 LiveKit Agents 搭了一个全双工 AI 语音助理：从选型到落地的完整思考","I built a full-duplex AI voice assistant using LiveKit Agents: A complete reflection from selection to implementation","## 写在前面\n\n大语言模型的能力已经不用多说了，但「能聊」和「能通话」是两回事。\n\n在今年之前，要","he capabilities of Large Language Models (LLMs) need no further introduction, but \"being able to chat\" and \"being able to call\" are two completely different things.","## 写在前面\n\n大语言模型的能力已经不用多说了，但「能聊」和「能通话」是两回事。\n\n在今年之前，要让 AI 真正进入实时语音对话场景，通常需要自己拼凑 STT（语音识别）→ LLM（语义理解）→ TTS（语音合成）三条 pipeline，还要处理 VAD（语音活动检测）、断句、打断、并发等一堆边缘情况。一个不小心，用户说「嗯」的时候 AI 就开始抢话，识别延迟堆到两三秒，语音对话就变成了对讲机。\n\n直到我发现了 LiveKit Agents 这个框架。\n\n这篇文章不打算写教程——网上已经有很多了。我想分享的是：**在真实的 IM\u002F呼叫业务中，把 LiveKit Agents 嵌入后端架构时，遇到的那些典型问题和解决思路。**\n\n## 选型：为什么是 LiveKit 而不是 Twilio \u002F Agora\n\n先简单说下背景：我需要在一个已有 IM 和 WebRTC 通话能力的系统中，加入 AI 语音助理的能力。\n\n市面上可选方案大致有三类：\n\n| 方案 | 代表 | 评价 |\n|------|------|------|\n| 全托管方案 | Twilio Voice + OpenAI | 省事，但贵，数据不在自己手里，定制空间小 |\n| 自建 pipeline | 自己拼 STT→LLM→TTS | 灵活，但工作量巨大，VAD、并发、断句都得自己搞 |\n| Agent 框架 | LiveKit Agents, Vocode, Pipecat | 折中方案，框架处理底层细节，自己写业务逻辑 |\n\n最终选择 LiveKit Agents 的原因很简单：**它的 Agent API 抽象层做得最好。**\n\nLiveKit Agents 把「AI 助理加入一个实时音视频房间」这件事，抽象成了一个标准的 Job\u002FWorker 模型：\n\n```\n房间创建 → LiveKit Server 发布 Job → Worker 消费 Job → Agent 加入房间 → 开始对话\n```\n\n这个模型天然契合我已有的 IM 呼叫流程——在我的系统中，创建通话房间的逻辑早就写好了，现在只需要在房间创建后，让 AI 作为参与者之一加入即可。\n\n## 架构拆解\n\n实际的部署架构如下：\n\n```\n客户端（浏览器\u002FApp）\n    ↕ WebRTC 音频流\nLiveKit Server（自托管）\n    ↕ 音频流\nPython Agent Worker（LiveKit Agents）\n    ↕ HTTP（拉取上下文 + 推送记录）\nGo 主后端（业务服务）\n```\n\n关键设计原则只有一条：**Agent 不做业务逻辑，只做「语音 ↔ 文本」的转换。**\n\n所有的业务知识、用户信息、聊天记录，都通过 HTTP 接口从主后端拉取，以 System Prompt 的形式注入 LLM。Agent 识别到的文本和 AI 的回复，也通过 HTTP 写回主后端持久化。\n\nAgent 的入口代码非常清晰，核心流程就三步：\n\n```python\nasync def entrypoint(ctx: JobContext):\n    room_name = ctx.room.name\n\n    # 1. 从业务后端拉取当前房间的上下文\n    room_context = await fetch_room_context(room_name)\n    system_prompt = build_system_prompt(room_context)\n\n    # 2. 连接房间，等待用户进入\n    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)\n    participant = await ctx.wait_for_participant()\n\n    # 3. 构建 Agent，注入 LLM + VAD + TTS\n    agent = Agent(\n        instructions=system_prompt,\n        llm=google.realtime.RealtimeModel(\n            api_key=google_api_key, voice=\"Aoede\",\n        ),\n        tts=google.beta.GeminiTTS(api_key=google_api_key),\n        vad=ctx.proc.userdata[\"vad\"],\n        allow_interruptions=True,\n    )\n\n    # 4. 启动会话\n    session = AgentSession()\n    await session.start(agent, room=ctx.room)\n    await session.say(\"AI 助理已进入房间\")\n```\n\n框架把所有音频流的底层细节都封装了——WebRTC 连接、音频编解码、端到端延迟控制——开发者只需要关注 LLM 的 instructions 和事件回调。\n\n这样做的三个好处：\n\n- **保持 Agent 的轻量和可替换** —— 未来想换 STT\u002FTTS 服务商，或者甚至把 AI 能力从 LLM 换成其他模型，Agent 代码的改动量很小\n- **业务逻辑统一** —— 所有用户、权限、历史记录的管理，都在主后端，Agent 不需要理解业务\n- **解耦部署** —— Agent Worker 可以独立扩缩容，语音密集场景下可以多开 Worker，不影响主后端\n\n## 全双工 vs 半双工：一个被低估的细节\n\n早期的大多数 AI 语音方案都是半双工的——用户说完，AI 再回答，谁说完谁「释放麦克风」。这在交互上非常不自然，真人对话中不可能等对方说完再开口。\n\nLiveKit Agents 在 1.x 版本中引入了 `allow_interruptions` 机制，配合 VAD 模块，实现了真正的全双工对话。实现上，它做了三件事：\n\n1. **VAD（Silero）持续检测** —— 实时判断用户是否在说话，用户何时说完\n2. **流式 TTS 支持** —— AI 的回复不是生成完一整句再播放，而是边生成边播放\n3. **打断逻辑** —— 用户一旦开口，AI 的播放立即停止，等待用户说完再响应\n\n这三个机制的协作，决定了用户感知到的「对话流畅度」。我在调试阶段最常遇到的问题就是：VAD 灵敏度太高导致频繁打断 AI、或者太低导致用户说完 AI 还在发呆。\n\nVAD 模型需要在 Worker 启动时预加载到内存，避免每次通话都重新下载：\n\n```python\ndef prewarm(proc: JobProcess):\n    \"\"\"Worker 进程启动时执行一次，预加载 Silero VAD 模型\"\"\"\n    proc.userdata[\"vad\"] = silero.VAD.load()\n```\n\n然后在 Worker 入口中注册：\n\n```python\nif __name__ == \"__main__\":\n    cli.run_app(\n        WorkerOptions(\n            entrypoint_fnc=entrypoint,\n            prewarm_fnc=prewarm,\n        )\n    )\n```\n\n`prewarm` 机制是 LiveKit Agents 的一个好设计。语音处理模型的加载通常需要几百毫秒到几秒，如果在每次通话开始时才加载，用户的首次语音体验会有一个明显的卡顿。通过进程级预加载，第一个字就能准确检测到。\n\n最终调出来的参数经验是：**VAD 的阈值和沉默超时时间，必须根据实际使用场景的语速和背景噪音来调。** 文档里的默认参数只能给出一个能跑的结果，离「好用」还有一段距离。\n\n## 关键决策：选多模态还是纯拆分的三件套\n\n在 Agent 的架构设计上有一个值得讨论的点：LLM 是否负责音频的输入和输出？\n\n- **方案 A（传统拆分方案）：**  \n  用户音频 → STT（转文本） → LLM（生成文本） → TTS（转音频） → 播放\n\n- **方案 B（多模态方案）：**  \n  用户音频 → 多模态 LLM（直接理解音频并生成音频） → 播放\n\n这个决策会影响延迟、成本、以及代码复杂度。\n\n我现在的做法是：使用多模态 RealtimeModel 方案。这样做的好处非常明显：\n\n- **代码量骤减** —— 不需要注册单独的 STT 和 TTS 组件，LLM 一次性处理输入输出\n- **延迟更低** —— 少了文本序列化和反序列化的环节\n- **天然支持情感和语气** —— 多模态模型能感知用户语调，回复也更有「人情味」\n\n但多模态方案也有代价：模型的选型范围窄，对云服务商的依赖度高。如果未来想换到本地部署的模型，目前多模态方案还不太现实。\n\n## 对话上下文的同步问题\n\n这是实际落地中遇到的最棘手的工程问题。\n\n当 AI 在一场通话中说话时，这部分对话需要同步到 IM 的消息记录中，让用户可以看到「历史记录」。但如果 AI 和用户之间开启了语音通话，他们在 IM 上看到的消息应该是怎样的？\n\n我最终的设计是：\n\n- **Agent 实时推送转写内容到后端** —— 用户说的每一句话被识别后，AI 的每句回复生成后，都通过 HTTP 异步写回后端\n- **后端以「消息」的形式持久化** —— 与 IM 中的文本消息共用同一张消息表，带上一个「语音转写」的标识\n- **通话结束时，允许对摘要进行处理** —— 长通话需要生成摘要，以特殊消息格式推送给参与者\n\n实现起来，就是在 Agent 的事件回调中做一次异步 HTTP 推送：\n\n```python\n@session.on(\"conversation_item_added\")\ndef on_item_added(ev):\n    item = ev.item\n    if not isinstance(item, llm.ChatMessage):\n        return\n\n    content = item.text_content\n    if not content:\n        return\n\n    if item.role == \"user\":\n        # 用户说的话 → 推送给业务后端持久化\n        asyncio.create_task(\n            push_utterance(room_name, participant.identity, content)\n        )\n    elif item.role == \"assistant\":\n        # AI 的回复 → 同样推送给业务后端持久化\n        asyncio.create_task(\n            push_utterance(room_name, \"ai_assistant\", content)\n        )\n```\n\n而 `push_utterance` 本身就是一个无状态的 HTTP POST：\n\n```python\nasync def push_utterance(room_name: str, identity: str, content: str):\n    url = f\"{backend_url}\u002Fapi\u002Fagent\u002Futterance\"\n    payload = {\n        \"room_name\": room_name,\n        \"speaker_identity\": identity,\n        \"content\": content,\n        \"language\": \"zh-CN\",\n    }\n    async with session.post(url, json=payload, timeout=aiohttp.ClientTimeout(total=5)) as resp:\n        if resp.status != 200:\n            logger.warning(f\"Push failed: {resp.status}\")\n```\n\n单看这两段代码很简单，但实际有几个坑：\n\n- **异步中的扇出问题** —— Agent 一次可能处理多个房间，回调中的异步任务必须做好异常处理，不能漏掉某一句\n- **顺序保证** —— HTTP 请求是异步的，不能保证到达顺序。但 IM 消息是需要有序的，所以后端需要在前端时间戳的基础上做重排序\n- **重试和幂等** —— 网络抖动可能导致同一句话被推送多次，后端必须能去重\n\n## 部署和运维的一些经验\n\nAgent Worker 部署我们用 Docker，`network_mode: host` 是关键决策：\n\n语音流的每一跳都会增加延迟。如果用 bridge 网络模式，音频数据要经过容器内部的 NAT 转发，在带宽敏感的实时场景下不可忽视。host 模式直接共享宿主机网络栈，大约能减少 5-10ms 的往返延迟。\n\n另外，自签名证书的 SSL 跳过也值得一提。\n\nLiveKit Server 在内网部署时几乎都会用自签名证书，而 Python 的 aiohttp\u002Furllib 默认会验证 SSL。如果在每个请求的地方传 `ssl=False` 太麻烦。在模块级别做一次 monkey-patch，是最省事的做法：\n\n```python\nimport ssl\nimport aiohttp\n\n# 全局跳过 SSL 验证（仅限内网自签名证书场景）\nssl._create_default_https_context = ssl._create_unverified_context\n\n_original_init = aiohttp.TCPConnector.__init__\n\ndef _patched_init(self, *args, **kwargs):\n    kwargs[\"ssl\"] = False\n    _original_init(self, *args, **kwargs)\n\naiohttp.TCPConnector.__init__ = _patched_init\n```\n\n虽然 monkey-patch 不优雅，但在可信内网中是最实用的方案。\n\nDocker 部署时，`network_mode: host` 是关键参数：\n\n```yaml\nservices:\n  axiom-agent:\n    build: .\n    env_file:\n      - .env\n    network_mode: \"host\"          # 直接复用宿主机网络栈\n    deploy:\n      resources:\n        limits:\n          memory: 1G\n    logging:\n      driver: \"json-file\"\n      options:\n        max-size: \"10m\"\n        max-file: \"3\"\n```\n\n语音流的每一跳都会增加延迟。bridge 模式下音频数据要经过容器 NAT 转发，host 模式直接共享宿主机网络栈，大约能减少 5-10ms 的往返延迟。对于实时语音，每一毫秒都算数。\n\n## 费用分析\n\n用多模态模型的费用结构和传统拆分方案不太一样。传统方案下，一小时活跃通话的大致成本：\n\n- STT（语音识别）：约 $0.26\n- LLM（语义理解）：约 $0.50\n- TTS（语音合成）：约 $0.30  \n**合计：约 $1.06\u002F小时**\n\n而多模态方案下，STT 和 TTS 的成本被「打包」进 LLM 的调用中，单价看似更贵，但省去了多服务串联的开销和网络往返，实际综合成本反而可能会低一些。\n\n当然这些数据只是估算，实际取决于对话密度、模型版本、以及 API 定价变化。但作为一个参考，一小时 AI 通话的成本控制在一到两美元以内，对于很多 B 端场景来说是完全可以接受的。\n\n## 一些还在探索的方向\n\n目前的方案已经跑通了基本的闭环，但还有几个值得继续深入的方面：\n\n- **更自然的打断** —— 现在基于 VAD 的打断虽然可用，但在多人对话场景下，AI 不知道「什么时候该插话」，目前只能被动等用户说完。如果能引入更细粒度的对话状态管理，AI 的参与感会更强。\n- **TTS 情感控制** —— 现在的 TTS 支持通过调节参数改变语气，但目前没有和 LLM 的回复内容做联动。如果能让 AI 在安慰用户时语气更柔和，在确认事项时语气更笃定，体验会完全不一样。\n- **长对话的 token 管理** —— 超过一小时的对话，上下文会变得很长，token 消耗和模型响应延迟都会增加。目前简单的滑动窗口策略会丢失早期信息，需要更好的摘要与记忆机制。\n- **语音活性检测与背景降噪** —— 真实环境下的背景噪音（键盘声、环境人声）会影响 VAD 的准确性，甚至导致 Agent 误以为自己被「打断」。加一层降噪预处理可能会显著提升体验。\n\n## 最后\n\nLiveKit Agents 是一个非常优秀的框架，它把 AI 语音对话从「实验室玩具」推进到了「可投入生产的工具」这个阶段。\n\n但也需要承认，框架只能解决「怎么连接」的问题，不能解决「怎么好用」的问题。 真正决定用户体验的，是 VAD 参数的反复调试、是对话上下文如何与现有业务打通、是延迟的每一毫秒优化——这些都需要在具体的业务场景中去磨。\n\n如果你也在做类似的事情，欢迎交流。AI 语音助理还在早期，这个方向上的坑和思路，值得更多人来一起探索。","## Preface\n\nThe capabilities of Large Language Models (LLMs) need no further introduction, but \"being able to chat\" and \"being able to call\" are two completely different things.\n\nBefore this year, to truly integrate AI into real-time voice conversation scenarios, you typically had to stitch together three separate pipelines: STT (Speech-to-Text) → LLM (Language Model) → TTS (Text-to-Speech). You also had to handle voice activity detection (VAD), sentence splitting, interruptions, concurrency, and other edge cases. One slip-up, and the AI would start talking over the user when they just said \"uh-huh,\" or the recognition latency would stack up to 2-3 seconds, turning a voice call into a walkie-talkie experience.\n\nThat was until I discovered the LiveKit Agents framework.\n\nThis article is not intended as a tutorial — there are already plenty of those online. What I want to share is: **the typical challenges and solutions encountered when embedding LiveKit Agents into a backend architecture within real-world IM\u002Fcalling services.**\n\n## Selection: Why LiveKit instead of Twilio \u002F Agora\n\nFirst, a quick background: I needed to add AI voice assistant capabilities to an existing system that already has IM and WebRTC calling capabilities.\n\nThere are roughly three categories of options available on the market:\n\n| Option | Representative | Review |\n|------|------|------|\n| Fully Managed | Twilio Voice + OpenAI | Simple, but expensive; data is not in your hands, and customization is limited |\n| Custom Pipeline | Build your own STT→LLM→TTS | Flexible, but massive workload; VAD, concurrency, and sentence splitting must be handled manually |\n| Agent Framework | LiveKit Agents, Vocode, Pipecat | Compromise solution; the framework handles low-level details while you write business logic |\n\nThe reason for choosing LiveKit Agents is simple: **it has the best Agent API abstraction layer.**\n\nLiveKit Agents abstracts the action of \"an AI assistant joining a real-time audio\u002Fvideo room\" into a standard Job\u002FWorker model:\n\n```\nRoom Created → LiveKit Server Publishes Job → Worker Consumes Job → Agent Joins Room → Conversation Starts\n```\n\nThis model naturally fits my existing IM calling flow —— the logic for creating call rooms was already written in my system, so now all we need to do is have the AI join as one of the participants after the room is created.\n\n## Architecture Breakdown\n\nThe actual deployment architecture is as follows:\n\n```\nClient (Browser\u002FApp)\n    ↕ WebRTC Audio Stream\nLiveKit Server (Self-hosted)\n    ↕ Audio Stream\nPython Agent Worker (LiveKit Agents)\n    ↕ HTTP (Fetch Context + Push Logs)\nGo Main Backend (Business Services)\n```\n\nThere is only one key design principle: **The Agent does not handle business logic; it only handles the \"Speech ↔ Text\" conversion.**\n\nAll business knowledge, user information, and chat history are fetched from the main backend via HTTP APIs and injected into the LLM as a System Prompt. The text recognized by the Agent and the AI's responses are also written back to the main backend via HTTP for persistence.\n\nThe Agent's entrypoint code is very clear, with the core flow consisting of four steps:\n\n```python\nasync def entrypoint(ctx: JobContext):\n    room_name = ctx.room.name\n\n    # 1. Fetch current room context from business backend\n    room_context = await fetch_room_context(room_name)\n    system_prompt = build_system_prompt(room_context)\n\n    # 2. Connect to the room and wait for user to join\n    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)\n    participant = await ctx.wait_for_participant()\n\n    # 3. Build Agent, inject LLM + VAD + TTS\n    agent = Agent(\n        instructions=system_prompt,\n        llm=google.realtime.RealtimeModel(\n            api_key=google_api_key, voice=\"Aoede\",\n        ),\n        tts=google.beta.GeminiTTS(api_key=google_api_key),\n        vad=ctx.proc.userdata[\"vad\"],\n        allow_interruptions=True,\n    )\n\n    # 4. Start the session\n    session = AgentSession()\n    await session.start(agent, room=ctx.room)\n    await session.say(\"AI assistant has entered the room\")\n```\n\nThe framework encapsulates all low-level details of audio streaming — WebRTC connections, audio codecs, end-to-end latency control — letting developers focus solely on the LLM's instructions and event callbacks.\n\nThree benefits of doing this:\n\n- **Keep the Agent lightweight and replaceable** —— If you want to change STT\u002FTTS providers in the future, or even switch the AI engine from LLM to other models, the Agent code changes will be minimal.\n- **Unified business logic** —— Management of all users, permissions, and history remains in the main backend; the Agent does not need to understand business rules.\n- **Decoupled deployment** —— Agent Workers can scale independently; you can spin up more Workers during high voice traffic without affecting the main backend.\n\n## Full-Duplex vs Half-Duplex: An Underestimated Detail\n\nMost early AI voice solutions were half-duplex —— the user talks, then the AI responds, and whoever finishes speaking \"releases the microphone.\" This is highly unnatural in interaction; in real human conversations, people don't wait for the other person to completely finish speaking before uttering a word.\n\nLiveKit Agents introduced the `allow_interruptions` mechanism in version 1.x, which, paired with the VAD module, achieves true full-duplex conversation. In practice, it accomplishes three things:\n\n1. **Continuous VAD (Silero) Detection** —— Dynamically determines whether the user is speaking and when the user finishes.\n2. **Streaming TTS Support** —— The AI's response is played while being generated, instead of waiting for the entire sentence to complete.\n3. **Interruption Logic** —— Once the user starts speaking, the AI's playback stops immediately, waiting for the user to finish before responding.\n\nThe collaboration of these three mechanisms determines the \"conversational fluidity\" perceived by users. The most common problem I encountered during the tuning phase was: VAD sensitivity being too high, causing frequent interruptions to the AI, or too low, causing the AI to freeze after the user finished speaking.\n\nThe VAD model needs to be pre-loaded into memory when the Worker starts, avoiding downloads for every call:\n\n```python\ndef prewarm(proc: JobProcess):\n    \"\"\"Executed once when the Worker process starts, preloading the Silero VAD model\"\"\"\n    proc.userdata[\"vad\"] = silero.VAD.load()\n```\n\nThen register it in the Worker entrypoint:\n\n```python\nif __name__ == \"__main__\":\n    cli.run_app(\n        WorkerOptions(\n            entrypoint_fnc=entrypoint,\n            prewarm_fnc=prewarm,\n        )\n    )\n```\n\nThe `prewarm` mechanism is a neat design of LiveKit Agents. Loading voice processing models usually takes several hundred milliseconds to a few seconds. If loaded only when each call starts, the user's first voice interaction will experience a noticeable stutter. By preloading at the process level, the very first word can be accurately detected.\n\nThe ultimate parameter experience is: **VAD thresholds and silence timeout values must be tuned according to the speech rate and background noise of the actual usage scenarios.** The default parameters in the documentation only yield a runnable result, which is still a long way from being \"production-ready.\"\n\n## Critical Decision: Choose Multimodal or Split \"Three-Piece\" Architecture\n\nThere is a point worth discussing in the Agent's architectural design: Should the LLM handle audio input and output?\n\n- **Option A (Traditional Split Solution):**  \n  User Audio → STT (Speech-to-Text) → LLM (Generate Text) → TTS (Text-to-Speech) → Playback\n\n- **Option B (Multimodal Solution):**  \n  User Audio → Multimodal LLM (Directly understand audio and generate audio) → Playback\n\nThis decision affects latency, cost, and code complexity.\n\nMy current practice is: using the multimodal RealtimeModel solution. The advantages of doing so are obvious:\n\n- **Drastic reduction in code volume** —— No need to register separate STT and TTS components; the LLM handles input and output all at once.\n- **Lower latency** —— Removes the text serialization and deserialization steps.\n- **Natural support for emotion and tone** —— Multimodal models can perceive user intonation, making the responses feel more \"human.\"\n\nHowever, the multimodal solution has its trade-offs: the choice of models is narrow, and the dependency on cloud service providers is high. If you want to switch to locally deployed models in the future, the multimodal option is currently not very practical.\n\n## Conversation Context Synchronization Issue\n\nThis was the trickiest engineering problem encountered in practice.\n\nWhen the AI speaks in a call, that part of the conversation needs to be synchronized to the IM message records so that users can see the \"history.\" But if a voice call is active between the AI and the user, what should the messages they see on IM look like?\n\nMy final design was:\n\n- **Agent pushes transcribed content to the backend in real-time** —— After each sentence spoken by the user is recognized, and after each response from the AI is generated, it is written back asynchronously to the backend via HTTP.\n- **Backend persists it as \"messages\"** —— It shares the same message table with text messages in IM, flagged with a \"voice transcription\" identifier.\n- **Allows summary processing when the call ends** —— Long calls need to generate summaries, which are pushed to participants as special messages.\n\nIn terms of implementation, this is done via an asynchronous HTTP push in the Agent's event callback:\n\n```python\n@session.on(\"conversation_item_added\")\ndef on_item_added(ev):\n    item = ev.item\n    if not isinstance(item, llm.ChatMessage):\n        return\n\n    content = item.text_content\n    if not content:\n        return\n\n    if item.role == \"user\":\n        # User utterance -> push to business backend for persistence\n        asyncio.create_task(\n            push_utterance(room_name, participant.identity, content)\n        )\n    elif item.role == \"assistant\":\n        # AI response -> also push to business backend for persistence\n        asyncio.create_task(\n            push_utterance(room_name, \"ai_assistant\", content)\n        )\n```\n\nAnd `push_utterance` itself is a stateless HTTP POST:\n\n```python\nasync def push_utterance(room_name: str, identity: str, content: str):\n    url = f\"{backend_url}\u002Fapi\u002Fagent\u002Futterance\"\n    payload = {\n        \"room_name\": room_name,\n        \"speaker_identity\": identity,\n        \"content\": content,\n        \"language\": \"zh-CN\",\n    }\n    async with session.post(url, json=payload, timeout=aiohttp.ClientTimeout(total=5)) as resp:\n        if resp.status != 200:\n            logger.warning(f\"Push failed: {resp.status}\")\n```\n\nLooking at these two snippets alone, it seems simple, but in practice, there are several traps:\n\n- **Fan-out issues in async calls** —— An Agent might process multiple rooms simultaneously. Async tasks in callbacks must have robust exception handling, otherwise, a sentence might be dropped.\n- **Ordering guarantee** —— HTTP requests are asynchronous, and arrival order cannot be guaranteed. However, IM messages need to be ordered, so the backend must re-order based on the frontend timestamps.\n- **Retry and Idempotency** —— Network jitters can cause the same utterance to be pushed multiple times, so the backend must perform deduplication.\n\n## Deployment and O&M Experiences\n\nFor Agent Worker deployment, we use Docker. Setting `network_mode: host` was a critical decision:\n\nEvery hop in a voice stream increases latency. If you use the bridge network mode, audio data passes through NAT translation inside the container, which is non-negligible in bandwidth-sensitive real-time scenarios. The host mode directly shares the host's network stack, reducing round-trip latency by about 5-10ms.\n\nAdditionally, skipping SSL verification for self-signed certificates is worth mentioning.\n\nWhen LiveKit Server is deployed on an intranet, self-signed certificates are almost always used, while Python's aiohttp\u002Furllib validates SSL by default. Passing `ssl=False` in every single request is too tedious. Doing a monkey-patch at the module level is the easiest approach:\n\n```python\nimport ssl\nimport aiohttp\n\n# Skip SSL verification globally (only for intranet self-signed certificate scenarios)\nssl._create_default_https_context = ssl._create_unverified_context\n\n_original_init = aiohttp.TCPConnector.__init__\n\ndef _patched_init(self, *args, **kwargs):\n    kwargs[\"ssl\"] = False\n    _original_init(self, *args, **kwargs)\n\naiohttp.TCPConnector.__init__ = _patched_init\n```\n\nAlthough monkey-patching is not elegant, it is the most practical solution in a trusted intranet.\n\nFor Docker deployment, `network_mode: host` is a crucial parameter:\n\n```yaml\nservices:\n  axiom-agent:\n    build: .\n    env_file:\n      - .env\n    network_mode: \"host\"          # Directly reuse the host's network stack\n    deploy:\n      resources:\n        limits:\n          memory: 1G\n    logging:\n      driver: \"json-file\"\n      options:\n        max-size: \"10m\"\n        max-file: \"3\"\n```\n\nEvery hop in a voice stream increases latency. Under bridge mode, audio data passes through container NAT translation; host mode directly shares the host's network stack, reducing round-trip latency by about 5-10ms. For real-time voice, every millisecond counts.\n\n## Cost Analysis\n\nThe cost structure of using a multimodal model is slightly different from the traditional split solution. Under the traditional scheme, the approximate cost of an active call hour is:\n\n- STT (Speech-to-Text): ~ $0.26\n- LLM (Language Model): ~ $0.50\n- TTS (Text-to-Speech): ~ $0.30  \n  **Total: ~ $1.06\u002Fhour**\n\nUnder the multimodal approach, the costs of STT and TTS are \"bundled\" into the LLM API calls. Although the unit price seems more expensive, it saves the overhead of serializing multiple services and network round-trips, making the actual overall cost potentially lower.\n\nOf course, these numbers are only estimates, depending on conversation density, model version, and API pricing changes. But as a reference, keeping the cost of an AI call hour within one to two dollars is fully acceptable for many B2B scenarios.\n\n## Future Exploration Directions\n\nThe current solution has successfully implemented the basic closed loop, but there are still several areas worth exploring further:\n\n- **More Natural Interruptions** —— While VAD-based interruptions are usable, in multi-person conversation scenarios, the AI doesn't know \"when to chip in\" and currently can only wait passively for the user to finish speaking. If more granular conversation state management can be introduced, the AI's presence will feel much stronger.\n- **TTS Emotional Control** —— Current TTS supports tone adjustment via parameter tuning, but there is no link with the content of the LLM replies yet. The experience would be completely different if the AI could speak more softly when comforting users and more firmly when confirming details.\n- **Long Conversation Token Management** —— For calls exceeding one hour, the context becomes very long, increasing token consumption and model response latency. Currently, simple sliding window strategies lose early information, requiring better summarization and memory mechanisms.\n- **VAD and Background Noise Reduction** —— Background noises (keyboard clicks, ambient chatter) in real-world environments affect VAD accuracy, and can even trick the Agent into thinking it is being \"interrupted.\" Adding a layer of noise reduction preprocessing could significantly improve the experience.\n\n## Finally\n\nLiveKit Agents is an outstanding framework that advances AI voice conversation from a \"lab toy\" to a \"production-ready tool.\"\n\nHowever, it must be acknowledged that the framework only solves the \"how to connect\" problem, not the \"how to make it work well\" problem. What truly determines user experience is the repeated tuning of VAD parameters, how conversation contexts integrate with existing business flows, and every millisecond of latency optimization —— all of which must be refined in specific business scenarios.\n\nIf you are doing similar work, feel free to exchange ideas. AI voice assistants are still in their infancy, and the pitfalls and thoughts in this direction deserve more people to explore together.\n","AI",31,"https:\u002F\u002Fblog4-1316398321.cos.ap-nanjing.myqcloud.com\u002Fblog5\u002F20260621005241__Attack-STAGE3.png",[15,16],"Python","Agent",true,{"id":19,"title":20,"title_en":21},32,"在线平台个性化推荐","Personalized recommendation on online platform",{"id":23,"title":24,"title_en":25},35,"2026 终极指南：手把手带你玩转 Google DeepMind 的 Antigravity 智能编程助手","The Ultimate Guide to 2026: Help you play Google DeepMind's Antigravity Intelligent Programming Assistant","2026-06-04T07:15:47.835343+08:00"]