[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-44":3},{"id":4,"title":5,"title_en":6,"abstract":7,"abstract_en":8,"content":9,"content_en":10,"category":11,"banner_id":12,"banner_path":13,"tags":14,"is_recommend":17,"prev_article":18,"next_article":22,"created_at":26},44,"AI 系统出问题时，我如何用 TraceID 把黑盒拆开","When Your AI System Breaks: How I Used TraceID to Debug a Black Box"," how can I use TraceID to open up the black box","When building AI systems, there is a type of problem that is particularly tormenting:","做 AI 系统时，有一种问题特别折磨人：\n* 用户说：“刚才那次请求很慢”、“语音突然没返回”、“AI 回答到一半断了”、“页面显示成功但实际没有结果”。\n* 你打开日志一看，每个模块都好像没有明显错误：\n  * API 层有请求日志。\n  * 模型调用层有耗时日志。\n  * WebSocket 层有连接日志。\n  * 任务队列里也能看到消费记录。\n\n但这些日志像散落在地上的碎片，很难拼成一条完整链路。\n\n尤其在 AI 系统里，这种问题会被进一步放大。因为一个用户请求往往不是一次简单的 HTTP 调用，而是会穿过多个物理与异步边界：\n\n```mermaid\ngraph TD\n    A[Web API] --> B[鉴权与限流]\n    B --> C[任务队列]\n    C --> D[向量检索]\n    D --> E[模型调用]\n    E --> F[流式响应]\n    F --> G[WebSocket 推送]\n    G --> H[异步回调]\n    H --> I[数据落库]\n```\n\n更麻烦的是，AI 请求天然具有不确定性：\n1. **模型响应时间不稳定**：受网络、负载及 Token 数量影响极大。\n2. **流式输出可能中途失败**：网络抖动或上游限流随时会导致连接中断。\n3. **上下文长度影响延迟**：越长的 Context，首字延迟（TTFT）和总耗时越高。\n4. **多模态输入处理复杂**：音频、图片、文本的接收与处理耗时差异巨大。\n5. **多协程协作**：多个 Goroutine 可能共同处理同一个任务，导致日志顺序错乱。\n\n用户端看到的问题，往往发生在服务端链路的后半段。所以 AI 系统排查最难的地方，不是“有没有日志”，而是：**当一次请求跨过多个模块、多个异步边界、多个 IO 系统后，你还能不能把它重新串起来。**\n\n这就是 TraceID 和全链路日志的价值。它不是为了让日志变多，而是为了让每一条日志都能回答同一个问题：**这条日志，属于哪一次用户请求？**\n\n---\n\n## 一、 为什么 AI 系统比普通业务系统更容易变成黑盒\n\n传统 CRUD 系统的问题链路通常比较短。一次请求进来，查数据库，做业务判断，返回结果。如果出错，大多数时候在 API 日志、SQL 日志或错误堆栈里就能定位。\n\n但 AI 系统不是这样。以一次语音 AI 请求为例，它经历了极其漫长且复杂的链路：\n\n```mermaid\ngraph TD\n    Client[客户端发起请求] --> Gateway[HTTP \u002F WebSocket 接入层]\n    Gateway --> Auth[鉴权、额度、会话校验]\n    Auth --> AudioRecv[音频分片接收]\n    AudioRecv --> ASR[ASR 语音识别]\n    ASR --> ContextBuild[上下文组装]\n    ContextBuild --> VectorSearch[向量检索 \u002F 业务数据查询]\n    VectorSearch --> LLM[LLM 推理]\n    LLM --> TTS[TTS 合成]\n    TTS --> StreamPush[流式返回给客户端]\n    StreamPush --> LogHistory[保存对话记录]\n    LogHistory --> TriggerPost[触发后续任务]\n```\n\n在这条链路里，任何一个环节异常都会导致严重的用户体验问题：\n* **ASR 成功了，但 LLM 超时了**。\n* **LLM 成功了，但 WebSocket 推送失败了**。\n* **WebSocket 推送成功了，但前端没有正确处理最后一个事件**。\n* **模型返回了内容，但落库失败导致历史记录缺失**。\n* **队列任务被消费了，但上下文信息没有正确传递**。\n* **重试机制触发了两次，导致用户收到重复结果**。\n\n这就是黑盒感的来源：**不是系统真的没有日志，而是日志之间缺少共同的坐标系。**\n\nTraceID 做的事情，就是给这条链路建立一个坐标系。\n\n---\n\n## 二、 TraceID 不是一个字段，而是一条请求的生命线\n\n在分布式 AI 系统里，TraceID 更像是一条请求的生命线。它应该从请求进入系统的第一刻开始出现，并且在每一次边界切换时被继续带下去。\n\n这些边界包括：\n* HTTP 请求边界\n* WebSocket 连接边界\n* Goroutine 异步边界\n* 消息队列边界\n* 第三方模型调用边界\n* 数据库写入边界\n* 定时任务或回调边界\n\n> [!IMPORTANT]\n> 只在 API 层生成 TraceID 没有意义。真正有用的是，它能不能穿过系统的每一层。\n\n一个比较理想的链路应该是这样：\n\n```text\ntrace_id = T-001\n\nHTTP request received\n  └─> auth checked\n      └─> quota checked\n          └─> task created\n              └─> queue published\n                  └─> worker consumed\n                      └─> embedding requested\n                          └─> vector search completed\n                              └─> llm stream started\n                                  └─> websocket chunks pushed\n                                      └─> final message persisted\n```\n\n当用户反馈问题时，只要拿到 `trace_id = T-001`，就应该能把这一次请求的完整行为捞出来。\n* 如果你只能查到 HTTP 层日志，查不到 worker 日志，说明 TraceID 在**队列边界**断了。\n* 如果你能查到模型调用日志，查不到 WebSocket 推送日志，说明 TraceID 在**长连接会话**里断了。\n* 如果你能查到异步任务日志，但查不到用户请求入口，说明**异步任务变成了孤岛**。\n\n---\n\n## 三、 在 Go 里，TraceID 应该从 context 开始\n\n在 Go 项目中，最自然的 TraceID 载体是 `context.Context`。\n\n它本来就是为了在请求范围内传递取消信号、超时、元信息而设计的。一个常见的做法是，在接入层生成或读取 TraceID，然后把它放进请求上下文。\n\n```go\nfunc TraceMiddleware(next Handler) Handler {\n    return func(ctx Context, req Request) Response {\n        \u002F\u002F 1. 优先读取上游传来的 TraceID\n        traceID := req.Header.Get(\"X-Trace-ID\")\n        if traceID == \"\" {\n            traceID = NewTraceID() \u002F\u002F 2. 没有时再自动生成\n        }\n\n        ctx = WithTraceID(ctx, traceID)\n\n        resp := next(ctx, req)\n        \n        \u002F\u002F 3. 把 TraceID 写回响应头，方便前端\u002F用户上报\n        resp.Header.Set(\"X-Trace-ID\", traceID)\n\n        return resp\n    }\n}\n```\n\n### 关键细节\n1. **优先读取上游传来的 TraceID**：如果你的系统前面还有网关、BFF 或客户端 SDK，它们可能已经生成了 TraceID。直接覆盖会切断跨服务链路。\n2. **写回响应头**：这一步极具实操价值。用户反馈问题时，前端可以把响应里的 TraceID 一并上报，排查效率会高很多。\n3. **避免在业务代码中直接读写裸字符串 key**：使用强类型包装读写逻辑，防覆盖防冲突。\n\n```go\ntype traceIDKey struct{}\n\nfunc WithTraceID(ctx context.Context, traceID string) context.Context {\n    return context.WithValue(ctx, traceIDKey{}, traceID)\n}\n\nfunc TraceIDFromContext(ctx context.Context) string {\n    if v, ok := ctx.Value(traceIDKey{}).(string); ok {\n        return v\n    }\n    return \"\"\n}\n```\n\n---\n\n## 四、 日志封装：不要让业务代码手动拼 TraceID\n\n如果依赖业务开发者在打日志时手动拼写 TraceID，系统里很快就会出现风格迥异的代码（如 `trace_id=xxx`、`traceId=xxx`、`tid=xxx` 等），导致日志平台无法统一检索。\n\n正确的方案是将 TraceID 的注入作为**日志基础设施**的一部分：\n\n```go\nfunc Info(ctx context.Context, msg string, fields ...Field) {\n    fields = appendTraceFields(ctx, fields...)\n    logger.Info(msg, fields...)\n}\n\nfunc Warn(ctx context.Context, msg string, fields ...Field) {\n    fields = appendTraceFields(ctx, fields...)\n    logger.Warn(msg, fields...)\n}\n\nfunc Error(ctx context.Context, msg string, fields ...Field) {\n    fields = appendTraceFields(ctx, fields...)\n    logger.Error(msg, fields...)\n}\n\nfunc appendTraceFields(ctx context.Context, fields ...Field) []Field {\n    if traceID := TraceIDFromContext(ctx); traceID != \"\" {\n        fields = append(fields, String(\"trace_id\", traceID))\n    }\n    return fields\n}\n```\n\n现在，业务开发编写代码时，只需关注核心业务字段：\n```go\nlog.Info(ctx, \"llm stream started\",\n    String(\"model\", modelName),\n    Int(\"prompt_tokens\", promptTokens),\n)\n```\n\n最终生成的结构化 JSON 日志会自动富集 TraceID：\n```json\n{\n  \"level\": \"info\",\n  \"time\": \"2026-06-23T10:21:03.120+08:00\",\n  \"msg\": \"llm stream started\",\n  \"trace_id\": \"T-001\",\n  \"model\": \"model_x\",\n  \"prompt_tokens\": 1024\n}\n```\n\n---\n\n## 五、 AI 链路中容易折断的三大边界\n\n### 5.1 Goroutine 异步边界\n在 Go 中，我们习惯直接使用 `go func` 开协程处理后台任务：\n```go\n\u002F\u002F 错误示范：上下文断连\ngo func() {\n    processJob(job)\n}()\n```\n这会导致任务与原有请求上下文脱钩。更稳健的做法是显式传递上下文：\n```go\n\u002F\u002F 改进版：传递 Context\ngo func(ctx context.Context, job Job) {\n    processJob(ctx, job)\n}(ctx, job)\n```\n\n> [!WARNING]\n> **警惕连接断开**：原请求的 `ctx` 会随着 HTTP 连接的意外中断而被 Cancel。如果异步后台任务必须继续执行，应该**复制 TraceID 并重建无取消语义的任务级 Context**。\n\n```go\n\u002F\u002F 正确做法：延续追踪，但解绑取消信号\ntaskCtx := NewBackgroundContext()\ntaskCtx = WithTraceID(taskCtx, TraceIDFromContext(ctx))\n\ngo func(ctx context.Context, job Job) {\n    processJob(ctx, job)\n}(taskCtx, job)\n```\n\n### 5.2 消息队列（MQ）边界\n当任务进入队列时，生产者和消费者的进程上下文是彻底断开的。我们必须将 TraceID 写入消息的**元数据元组（Metadata）**中传递：\n\n```go\ntype JobMessage struct {\n    ID       string\n    Payload  Payload\n    Metadata map[string]string \u002F\u002F 存放追踪元数据\n}\n\n\u002F\u002F 生产者发布任务：\nmsg.Metadata[\"trace_id\"] = TraceIDFromContext(ctx)\nqueue.Publish(msg)\n\n\u002F\u002F 消费者消费任务：\nfunc Consume(msg JobMessage) {\n    ctx := context.Background()\n    ctx = WithTraceID(ctx, msg.Metadata[\"trace_id\"]) \u002F\u002F 恢复 TraceID\n    process(ctx, msg.Payload)\n}\n```\n\n### 5.3 WebSocket 长连接边界\nWebSocket 是一条持续连接的通道，中间可能会承载数十次用户的 AI 对话和流式推送。如果在建立连接时只生成一个 ID，会让多轮对话的日志乱作一团。\n\n#### 正确的做法是区分两个概念：\n1. **`connection_id`**：用于标识并跟踪这一条特定的 WebSocket 物理连接。\n2. **`trace_id`**：用于标识每一次具体的请求事件或单轮对话任务。\n\n```text\nconnection_id = C-001, user_id = U-001\n    ├── trace_id = T-101, event = user_audio_started\n    ├── trace_id = T-101, event = llm_chunk_pushed\n    └── trace_id = T-101, event = response_completed\n```\n\n---\n\n## 六、 全链路日志不是把所有东西都打出来\n\n为了避免日志平台存储爆炸、性能下降和噪音泛滥，全链路日志应遵循“**关键节点 + 阶段耗时**”的原则，而非全量记录。\n\n我们将日志结构化为以下三类核心数据：\n\n### 1. 链路事件日志（Where）\n用于精确描述请求走到了哪个核心节点：\n* `request_received` -> `auth_checked` -> `retrieval_started` -> `llm_started` -> `tts_completed` -> `message_persisted`\n\n### 2. 性能指标日志（Why Slow）\n用于定量回答慢请求的瓶颈所在：\n* `http_latency_ms`（接口耗时）\n* `queue_wait_ms`（队列等待）\n* `llm_first_token_latency_ms`（**首字延迟，至关重要**）\n* `llm_total_latency_ms`（模型推理总耗时）\n* `tts_latency_ms`（语音合成耗时）\n\n### 3. 结果状态日志（Success Or Not）\n在 AI 链路中，请求存在“**部分成功**”的灰色状态（例如：模型返回文字成功了，但是 TTS 语音合成失败了）。必须精细化记录状态：\n* `status` = `success` | `failed` | `canceled` | `partial_success`\n* `error_code` = `upstream_timeout`\n* `retry_count` = 2\n\n---\n\n## 七、 哪些内容不应该进入日志（脱敏规范）\n\nAI 输入与输出通常包含用户隐私、敏感的专有知识库碎片或内部 Prompt 设计。未经脱敏全量打入日志属于严重安全漏洞。\n\n| 🟢 建议记录的字段 | 🟡 谨慎记录的字段 | 🔴 默认禁止记录的字段 |\n| :--- | :--- | :--- |\n| `trace_id` | `prompt` 长度（Token数） | 用户原始输入的 Prompt 文本 |\n| `user_id_hash` (哈希值) | `response` 长度 | 模型生成的完整 Response 文本 |\n| `request_type` | 命中的知识库文档 ID | 原始音频\u002F转写出的明文 |\n| `model_alias` (模型别名) | 调用的工具函数名称 (Function) | Token \u002F API Key \u002F 鉴权密钥 |\n| `latency_ms` (各阶段耗时) | | 用户手机号、邮箱等 PII 隐私数据 |\n| `status` & `error_code` | | 内部系统真实服务 IP 和地址 |\n\n---\n\n## 八、 一次真实排查应该怎么用 TraceID\n\n假设用户反馈：“刚才我说完话以后，AI 过了很久才回答，而且回答到一半断了。”\n\n利用 TraceID，排查逻辑会像剥洋葱一样简单：\n\n```mermaid\nsequenceDiagram\n    autonumber\n    actor User as 用户\n    participant Gateway as 接入网关\n    participant ASR as 语音识别\n    participant LLM as 大语言模型\n    participant WS as WebSocket通道\n\n    User->>Gateway: 发起语音请求 (生成\u002F传入 T-101)\n    Note over Gateway: T-101: 校验成功，耗时正常\n    Gateway->>ASR: 推送音频流\n    Note over ASR: T-101: 识别文本成功 (latency: 920ms)\n    ASR->>LLM: 检索知识库并调用模型\n    Note over LLM: T-101: 首字生成极慢 (TTFT: 2600ms)\n    LLM-->>WS: 持续返回 Stream Chunk\n    Note over LLM: T-101: 模型发生超时中断 (upstream_timeout)\n    WS-->>User: 连接意外断开 (connection_closed)\n```\n\n通过链路排查，你可以精准还原现场：\n1. **排除 ASR 故障**：ASR 耗时 920ms，正常。\n2. **锁定首字延迟**：LLM 的首 Token 耗时达到了 2600ms，定位到是模型服务过载或冷启动。\n3. **定位中断原委**：日志中显示 `llm_stream_interrupted`，错误码为 `upstream_timeout`，说明是模型侧的 Stream 突然中断，从而引发了 WebSocket 关闭。\n\n---\n\n## 九、 TraceID、RequestID、SpanID 协同设计\n\n* **`TraceID`**：标识一次端到端的完整业务链路（比如用户发起的一轮多模态对话）。\n* **`RequestID`**：标识链路中的某一次具体网络请求。\n* **`SpanID`**：标识链路中某一个具体的微小执行步骤或子模块调用。\n\n在复杂的大模型工程架构中，建议通过日志规范预留如下字段设计，便于后续无缝对接 Loki、ELK 甚至 OpenTelemetry 分布式链路追踪系统：\n\n```json\n{\n  \"trace_id\": \"T-101\",\n  \"request_id\": \"R-202\",\n  \"span_id\": \"S-303\",\n  \"connection_id\": \"C-001\",\n  \"span_name\": \"vector_search\",\n  \"latency_ms\": 150,\n  \"status\": \"success\"\n}\n```\n\n---\n\n## 十、 落地时最容易踩的六个坑\n\n1. **只有错误日志带 TraceID**：排查时你必须清楚错误发生前系统在做什么。因此，**链路关键点上的 Info、Warn 日志也必须统一带上 TraceID**。\n2. **TraceID 在异步任务（Goroutine\u002FQueue）中丢失**：队列边界未显式解析 Metadata，或开辟新 Goroutine 时直接传递了空上下文。\n3. **一个 WebSocket 长连接从始至终只用一个 TraceID**：必须使用 `connection_id` 标识连接，用 `trace_id` 区分连接里的每轮请求。\n4. **日志字段命名不统一**：`traceId`、`trace_id`、`tid` 混用，导致聚合分析时提取失败。一律规范为 `trace_id`。\n5. **为图方便将 Prompt \u002F Response 原文直接打印到生产环境日志**：数据隐私红线，极易造成数据泄漏。\n6. **只记录了请求的总耗时**：AI 系统必须做阶段耗时拆解（特别是首包时间 TTFT 和推理时间），否则根本查不出是慢在网络、模型推理还是语音合成。\n\n---\n\n## 结语\n\n在构建复杂的 AI 工程时，上下文一旦在某个物理或协程边界发生断连，所有的日志就会从“**连续的生命线**”退化成“**孤立的碎片**”。\n\nTraceID 全链路日志并不是为了消除 AI 系统的不确定性，而是为了当超时、中断或幻觉发生时，能够让复杂性变得**可追踪、可解释、可修复**。\n","# AI System Full-Link Logging Troubleshooting Guide: How to Use TraceID to Track the Lifeline of a Request\n\nWhen building AI systems, there is a type of problem that is particularly tormenting:\n* Users say: \"That request just now was very slow\", \"Voice suddenly didn't return\", \"AI cut off halfway through the answer\", \"Page shows success but there is actually no result\".\n* You open the logs and see that every module seems to have no obvious error:\n  * API layer has request logs.\n  * Model calling layer has latency logs.\n  * WebSocket layer has connection logs.\n  * Task queue also has consumption records.\n\nBut these logs are like fragments scattered on the ground, making it very difficult to piece together a complete workflow.\n\nEspecially in AI systems, this problem is further amplified. Because a user request is usually not a simple HTTP call, but rather passes through multiple physical and asynchronous boundaries:\n\n```mermaid\ngraph TD\n    A[Web API] --> B[Auth & Rate Limiting]\n    B --> C[Task Queue]\n    C --> D[Vector Retrieval]\n    D --> E[Model Calling]\n    E --> F[Streaming Response]\n    F --> G[WebSocket Push]\n    G --> H[Async Callback]\n    H --> I[Database Persistence]\n```\n\nMore troublesome is that AI requests naturally possess uncertainty:\n1. **Unstable model response time**: Highly affected by network, load, and Token quantity.\n2. **Streaming output can fail halfway**: Network jitter or upstream rate limiting can lead to connection interruption at any time.\n3. **Context length affects latency**: The longer the Context, the higher the Time-To-First-Token (TTFT) and total elapsed time.\n4. **Complex multimodal input processing**: Latency differences for processing audio, images, and text are huge.\n5. **Multi-goroutine collaboration**: Multiple Goroutines may process the same task, resulting in out-of-order logs.\n\nThe problems users see usually occur in the latter half of the server-side pipeline. So the most difficult part of debugging AI systems is not \"whether there are logs\", but: **when a request crosses multiple modules, multiple asynchronous boundaries, and multiple IO systems, can you still stitch it back together?**\n\nThis is the value of TraceID and full-link logging. It is not to make logs more voluminous, but to make every log answer the same question: **To which user request does this log belong?**\n\n---\n\n## I. Why AI Systems Turn Into Black Boxes More Easily Than Ordinary Systems\n\nTraditional CRUD systems usually have short problem paths. A request comes in, queries the database, makes a business decision, and returns the result. If an error occurs, it can be located in the API logs, SQL logs, or error stack traces in most cases.\n\nBut AI systems are not like this. Take a voice AI request as an example, it undergoes an extremely long and complex path:\n\n```mermaid\ngraph TD\n    Client[Client Initiates Request] --> Gateway[HTTP \u002F WebSocket Gateway]\n    Gateway --> Auth[Auth, Quota, Session Validation]\n    Auth --> AudioRecv[Audio Chunk Ingestion]\n    AudioRecv --> ASR[ASR Speech Recognition]\n    ASR --> ContextBuild[Context Assembly]\n    ContextBuild --> VectorSearch[Vector Retrieval \u002F Business Data Query]\n    VectorSearch --> LLM[LLM Inference]\n    LLM --> TTS[TTS Synthesis]\n    TTS --> StreamPush[Streaming Response to Client]\n    StreamPush --> LogHistory[Save Conversation History]\n    LogHistory --> TriggerPost[Trigger Subsequent Tasks]\n```\n\nIn this pipeline, any failure in any module will lead to serious user experience issues:\n* **ASR succeeded, but LLM timed out**.\n* **LLM succeeded, but WebSocket push failed**.\n* **WebSocket push succeeded, but frontend failed to handle the final event correctly**.\n* **Model returned content, but persistence failed, leading to lost history records**.\n* **Queue task was consumed, but context was not passed correctly**.\n* **Retry mechanism was triggered twice, leading to duplicate results for the user**.\n\nThis is where the black-box feeling comes from: **it's not that the system really doesn't have logs, but that the logs lack a common coordinate system.**\n\nTraceID establishes a coordinate system for this pipeline.\n\n---\n\n## II. TraceID is Not a Field, But the Lifeline of a Request\n\nIn distributed AI systems, TraceID is like the lifeline of a request. It should appear from the very first moment the request enters the system, and be carried along whenever crossing boundaries.\n\nThese boundaries include:\n* HTTP request boundaries\n* WebSocket connection boundaries\n* Goroutine asynchronous boundaries\n* Message queue boundaries\n* Third-party model call boundaries\n* Database write boundaries\n* Cron task or callback boundaries\n\n> [!IMPORTANT]\n> Generating TraceID only at the API layer is meaningless. What is truly useful is whether it can penetrate every layer of the system.\n\nA more ideal pipeline should look like this:\n\n```text\ntrace_id = T-001\n\nHTTP request received\n  └─> auth checked\n      └─> quota checked\n          └─> task created\n              └─> queue published\n                  └─> worker consumed\n                      └─> embedding requested\n                          └─> vector search completed\n                              └─> llm stream started\n                                  └─> websocket chunks pushed\n                                      └─> final message persisted\n```\n\nWhen a user reports a problem, as long as we get `trace_id = T-001`, we should be able to fetch the complete behavior of this request.\n* If you can only find HTTP layer logs but not worker logs, it means the TraceID broke at the **队列边界 (Queue Boundary)**.\n* If you can find model call logs but not WebSocket push logs, it means the TraceID broke in the **长连接会话 (Long Connection Session)**.\n* If you can find async task logs but not the user request entry, it means **the async task became an isolated island**.\n\n---\n\n## III. In Go, TraceID Should Start from context\n\nIn Go projects, the most natural carrier for TraceID is `context.Context`.\n\nIt was originally designed to pass cancellation signals, timeouts, and metadata within the scope of a request. A common practice is to generate or read the TraceID at the entry layer and inject it into the request context:\n\n```go\nfunc TraceMiddleware(next Handler) Handler {\n    return func(ctx Context, req Request) Response {\n        \u002F\u002F 1. Read upstream TraceID with priority\n        traceID := req.Header.Get(\"X-Trace-ID\")\n        if traceID == \"\" {\n            traceID = NewTraceID() \u002F\u002F 2. Generate if missing\n        }\n\n        ctx = WithTraceID(ctx, traceID)\n\n        resp := next(ctx, req)\n        \n        \u002F\u002F 3. Write TraceID back to response header for frontend\u002Fuser reporting\n        resp.Header.Set(\"X-Trace-ID\", traceID)\n\n        return resp\n    }\n}\n```\n\n### Key Details\n1. **Read upstream TraceID with priority**: If there is a gateway, BFF, or client SDK in front of your system, they may have already generated a TraceID. Overwriting it directly will cut the cross-service link.\n2. **Write back to response header**: This is highly practical. When users report issues, the frontend can report the TraceID in the response together, making troubleshooting much faster.\n3. **Avoid directly reading and writing bare string keys in business code**: Use strong types to wrap the read\u002Fwrite logic to prevent overwriting and conflicts.\n\n```go\ntype traceIDKey struct{}\n\nfunc WithTraceID(ctx context.Context, traceID string) context.Context {\n    return context.WithValue(ctx, traceIDKey{}, traceID)\n}\n\nfunc TraceIDFromContext(ctx context.Context) string {\n    if v, ok := ctx.Value(traceIDKey{}).(string); ok {\n        return v\n    }\n    return \"\"\n}\n```\n\n---\n\n## IV. Log Wrapper: Don't Let Business Code Manually String-Concatenate TraceID\n\nIf you rely on business developers to manually append TraceIDs when writing logs, style discrepancies (such as `trace_id=xxx`, `traceId=xxx`, `tid=xxx`, etc.) will quickly pop up, making unified log index search impossible.\n\nThe correct approach is to inject TraceID as part of the **logging infrastructure**:\n\n```go\nfunc Info(ctx context.Context, msg string, fields ...Field) {\n    fields = appendTraceFields(ctx, fields...)\n    logger.Info(msg, fields...)\n}\n\nfunc Warn(ctx context.Context, msg string, fields ...Field) {\n    fields = appendTraceFields(ctx, fields...)\n    logger.Warn(msg, fields...)\n}\n\nfunc Error(ctx context.Context, msg string, fields ...Field) {\n    fields = appendTraceFields(ctx, fields...)\n    logger.Error(msg, fields...)\n}\n\nfunc appendTraceFields(ctx context.Context, fields ...Field) []Field {\n    if traceID := TraceIDFromContext(ctx); traceID != \"\" {\n        fields = append(fields, String(\"trace_id\", traceID))\n    }\n    return fields\n}\n```\n\nNow, business developers only need to care about core business fields:\n```go\nlog.Info(ctx, \"llm stream started\",\n    String(\"model\", modelName),\n    Int(\"prompt_tokens\", promptTokens),\n)\n```\n\nThe final generated structured JSON log will automatically enrich the TraceID:\n```json\n{\n  \"level\": \"info\",\n  \"time\": \"2026-06-23T10:21:03.120+08:00\",\n  \"msg\": \"llm stream started\",\n  \"trace_id\": \"T-001\",\n  \"model\": \"model_x\",\n  \"prompt_tokens\": 1024\n}\n```\n\n---\n\n## V. Three Fragile Boundaries in AI Pipelines\n\n### 5.1 Goroutine Asynchronous Boundary\nIn Go, we are used to using `go func` directly to handle background tasks:\n```go\n\u002F\u002F Bad practice: context disconnected\ngo func() {\n    processJob(job)\n}()\n```\nThis causes the task to decouple from the original request context. A more robust approach is to explicitly pass the context:\n```go\n\u002F\u002F Improved version: pass Context\ngo func(ctx context.Context, job Job) {\n    processJob(ctx, job)\n}(ctx, job)\n```\n\n> [!WARNING]\n> **Beware of disconnected connections**: The original request's `ctx` will be cancelled if the HTTP connection is interrupted. If the background task must continue running, you should **copy the TraceID and rebuild a task-level Context with no cancellation semantics**.\n\n```go\n\u002F\u002F Correct practice: extend tracing, but decouple cancellation signal\ntaskCtx := NewBackgroundContext()\ntaskCtx = WithTraceID(taskCtx, TraceIDFromContext(ctx))\n\ngo func(ctx context.Context, job Job) {\n    processJob(ctx, job)\n}(taskCtx, job)\n```\n\n### 5.2 Message Queue (MQ) Boundary\nWhen a task enters the queue, the process contexts of the producer and consumer are completely disconnected. We must write the TraceID into the message's **Metadata** to pass it along:\n\n```go\ntype JobMessage struct {\n    ID       string\n    Payload  Payload\n    Metadata map[string]string \u002F\u002F Store tracing metadata\n}\n\n\u002F\u002F Producer publishes task:\nmsg.Metadata[\"trace_id\"] = TraceIDFromContext(ctx)\nqueue.Publish(msg)\n\n\u002F\u002F Consumer consumes task:\nfunc Consume(msg JobMessage) {\n    ctx := context.Background()\n    ctx = WithTraceID(ctx, msg.Metadata[\"trace_id\"]) \u002F\u002F Restore TraceID\n    process(ctx, msg.Payload)\n}\n```\n\n### 5.3 WebSocket Long Connection Boundary\nA WebSocket is a continuously connected channel, which may carry dozens of user AI dialogues and streaming pushes in the middle. If a single ID is generated only when the connection is established, logs of multiple dialogue rounds will be jumbled up.\n\n#### The correct way is to distinguish two concepts:\n1. **`connection_id`**: Identifies and tracks this specific WebSocket physical connection.\n2. **`trace_id`**: Identifies each specific request event or single-round dialogue task.\n\n```text\nconnection_id = C-001, user_id = U-001\n    ├── trace_id = T-101, event = user_audio_started\n    ├── trace_id = T-101, event = llm_chunk_pushed\n    └── trace_id = T-101, event = response_completed\n```\n\n---\n\n## VI. Full-Link Logging is Not Printing Everything\n\nTo avoid log platform storage explosions, performance degradation, and noise flooding, full-link logging should follow the principle of \"**key nodes + stage latency**\" instead of full recording.\n\nWe structure logs into three core categories:\n\n### 1. Link Event Logs (Where)\nUsed to accurately describe which core node the request has reached:\n* `request_received` -> `auth_checked` -> `retrieval_started` -> `llm_started` -> `tts_completed` -> `message_persisted`\n\n### 2. Performance Metric Logs (Why Slow)\nUsed to quantitatively answer where the slow request bottleneck is:\n* `http_latency_ms` (API latency)\n* `queue_wait_ms` (Queue waiting)\n* `llm_first_token_latency_ms` (**Time-To-First-Token, crucial**)\n* `llm_total_latency_ms` (Model inference total time)\n* `tts_latency_ms` (Speech synthesis latency)\n\n### 3. Result Status Logs (Success Or Not)\nIn AI pipelines, there are gray states of \"**partial success**\" (e.g., model returned text successfully, but TTS synthesis failed). We must log status finely:\n* `status` = `success` | `failed` | `canceled` | `partial_success`\n* `error_code` = `upstream_timeout`\n* `retry_count` = 2\n\n---\n\n## VII. What Should Not Go into Logs (Masking Rules)\n\nAI input and output usually contain user privacy, sensitive proprietary knowledge base fragments, or internal Prompts. Logging them without masking is a serious security vulnerability.\n\n| 🟢 Recommended to Log | 🟡 Log with Caution | 🔴 Prohibited to Log by Default |\n| :--- | :--- | :--- |\n| `trace_id` | `prompt` length (Token count) | User's original Prompt text |\n| `user_id_hash` (Hashed value) | `response` length | Complete Response text generated by model |\n| `request_type` | Hit knowledge base document ID | Original audio \u002F transcribed plain text |\n| `model_alias` (Model alias) | Called tool function name (Function) | Token \u002F API Key \u002F Auth secrets |\n| `latency_ms` (Latency of stages) | | User's phone number, email, and other PII |\n| `status` & `error_code` | | Internal system real service IPs and addresses |\n\n---\n\n## VIII. Troubleshooting a Real Issue Using TraceID\n\nSuppose a user reports: \"Just now after I finished speaking, AI took a very long time to answer, and it cut off halfway.\"\n\nUsing TraceID, the troubleshooting logic will be as simple as peeling an onion:\n\n```mermaid\nsequenceDiagram\n    autonumber\n    actor User as User\n    participant Gateway as Gateway\n    participant ASR as ASR\n    participant LLM as LLM\n    participant WS as WebSocket Channel\n\n    User->>Gateway: Initiate voice request (Generate\u002Fpass T-101)\n    Note over Gateway: T-101: Validation success, latency normal\n    Gateway->>ASR: Push audio stream\n    Note over ASR: T-101: Transcription success (latency: 920ms)\n    ASR->>LLM: Retrieve knowledge base & call model\n    Note over LLM: T-101: Extremely slow first token (TTFT: 2600ms)\n    LLM-->>WS: Stream response chunks continuously\n    Note over LLM: T-101: Upstream model timeout (upstream_timeout)\n    WS-->>User: Connection closed unexpectedly (connection_closed)\n```\n\nThrough link troubleshooting, you can accurately restore the scene:\n1. **Exclude ASR fault**: ASR latency is 920ms, normal.\n2. **Lock first token latency**: LLM's first Token took 2600ms, locating the bottleneck at model service overload or cold start.\n3. **Locate interruption cause**: Logs show `llm_stream_interrupted` with `upstream_timeout`, indicating the stream broke on the model side, which then triggered the WebSocket closure.\n\n---\n\n## IX. TraceID, RequestID, and SpanID Collaborative Design\n\n* **`TraceID`**: Identifies a complete end-to-end business link (e.g., a round of multi-modal dialogue initiated by a user).\n* **`RequestID`**: Identifies a specific network request within the link.\n* **`SpanID`**: Identifies a specific micro-execution step or sub-module call.\n\nIn complex large model architectures, it is recommended to reserve the following fields in logging specifications for seamless integration with Loki, ELK, or OpenTelemetry:\n\n```json\n{\n  \"trace_id\": \"T-101\",\n  \"request_id\": \"R-202\",\n  \"span_id\": \"S-303\",\n  \"connection_id\": \"C-001\",\n  \"span_name\": \"vector_search\",\n  \"latency_ms\": 150,\n  \"status\": \"success\"\n}\n```\n\n---\n\n## X. Six Traps Most Easy to Fall Into\n\n1. **Only error logs carry TraceID**: You must know what the system was doing before the error occurred. Therefore, **Info and Warn logs at key nodes must also carry TraceIDs**.\n2. **TraceID lost in async tasks (Goroutine\u002FQueue)**: Queue boundaries failed to parse Metadata, or new Goroutines were opened with empty contexts.\n3. **A WebSocket long connection uses a single TraceID from beginning to end**: Must use `connection_id` to identify the connection, and `trace_id` to distinguish each dialogue round.\n4. **Inconsistent logging field names**: `traceId`, `trace_id`, and `tid` are mixed, causing aggregation analysis queries to fail. Standardize on `trace_id`.\n5. **Print raw Prompt \u002F Response text to production logs for convenience**: Data privacy red line, extremely easy to cause leaks.\n6. **Only log total request latency**: AI systems must break down stage latency (especially first packet time TTFT and inference time), otherwise you cannot determine if it's slow on the network, model inference, or speech synthesis.\n\n---\n\n## Conclusion\n\nIn building complex AI engineering, once context breaks at a physical or coroutine boundary, all logs degrade from a \"**continuous lifeline**\" into \"**isolated fragments**\".\n\nTraceID full-link logging is not to eliminate AI system uncertainty, but to make complexity **traceable, explainable, and repairable** when timeout, interruption, or hallucination occurs.\n","AI",35,"https:\u002F\u002Fblog4-1316398321.cos.ap-nanjing.myqcloud.com\u002Fblog5\u002F20260623050840__二次元少女.png",[15,16],"GO","Agent",false,{"id":19,"title":20,"title_en":21},43,"避坑指南:桌面端 WebRTC 音频兼容性：为什么 AirPods 接上就\"没声音\"？","Pitfall Guide: Desktop WebRTC Audio Compatibility: Why Do AirPods Produce 'No Sound' When Connected?",{"id":23,"title":24,"title_en":25},45,"我是如何用几百行 Go 代码撸出一个生产级 AI 智能体编排引擎的","How I Built a Production-Grade AI Agent Orchestration Engine with Just a Few Hundred Lines of Go Code","2026-06-23T12:55:33.977876+08:00"]