[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-27":3},{"id":4,"title":5,"title_en":6,"abstract":7,"abstract_en":8,"content":9,"content_en":10,"category":11,"banner_id":12,"banner_path":13,"tags":14,"is_recommend":16,"prev_article":17,"next_article":21,"created_at":25},27,"在 Go 后端构建可观测性（Prometheus + Tracing + 日志）","Building observability on the Go backend (Prometheus + Tracing + Logging)","## 本文：在 Go 后端构建可观测性（Prometheus + Tracing + 日志）\n\n##","##This article: Building observability on the Go backend (Prometheus + Tracing + Logging)\n\n##","## 本文：在 Go 后端构建可观测性（Prometheus + Tracing + 日志）\n\n### 一、总体架构建议\n\n为了把可观测性整合到现有项目（config\u002Fcore\u002Fmodels\u002Fservice\u002Frouters\u002Fmiddleware），建议采用三层监控方案：\n\n1. 指标（Metrics）：Prometheus 指标用于量化系统的健康与性能。埋点位置通常在 HTTP 层、数据库访问、外部 API 调用和关键业务逻辑。 \n2. 日志（Logging）：结构化日志（例如 zap）用于记录请求上下文、错误堆栈与重要事件，用于审计与交叉验证指标。 \n3. 分布式追踪（Tracing）：使用 OpenTelemetry\u002FOpenTracing 导出到 Jaeger\u002FZipkin，用于呈现请求调用链、跨服务延迟分布与依赖关系。\n\n把这些能力放在 `middleware`（HTTP 层埋点、请求时长、trace 注入）、`core`（初始化 metrics\u002Ftracer\u002Flogger）与 `service`（业务级指标）中。\n\n### 二、指标（Prometheus）落地要点\n\n- 指标类型：\n  - Counter（计数器）：请求总数、错误数、重试次数。\n  - Gauge（测量值）：当前连接数、队列长度、缓存命中率（百分比）。\n  - Histogram\u002FSummary：请求时延分布、数据库查询时延。Histogram 更适合在 Prometheus 中做聚合。\n\n- Embedding（示例实践要点）：\n  - 在 `core` 初始化一个全局的 registry 或使用默认 registry。 \n  - 在 `middleware` 中埋请求计数与时延：按路由和状态码打标签（method, route, status）。\n  - 在 `service` 层为关键业务打业务指标（例如文章发布耗时、评论写入失败率）。\n\n- 指标命名与标签策略：\n  - 使用一致的前缀如 `appname_http_requests_total`、`appname_db_query_duration_seconds`。\n  - 标签不要过多，避免高基数（例如不要把 user_id 作为 label）。\n\n- 暴露 `\u002Fmetrics` 接口：\n  - 仅在内部网络或通过反向代理保护，切勿将裸 metrics 暴露到公网。\n\n- 示例工具链：prometheus（抓取）、grafana（展示）、alertmanager（告警）。\n\n### 三、结构化日志（zap）与日志关联\n\n- 使用结构化日志（如 zap），约定输出 JSON。字段至少包含：timestamp、level、msg、service、env、trace_id、span_id、请求路径、请求ID。 \n- 在 middleware 中把 trace_id \u002F request_id 注入到上下文，并在每条日志中输出，以便把日志与 tracing 链接。 \n- 日志级别策略：\n  - Debug：开发\u002F调试环境，包含 SQL\u002Fheaders（敏感数据脱敏）。\n  - Info：正常运行信息、关键事件。 \n  - Error：错误与异常，包含栈信息（适度）。\n\n- 日志采样：在高流量场景下对 debug\u002Finfo 做采样，避免日志风暴。推荐在收集端（如 fluentd\u002FLogstash）或 SDK 集成采样器。\n\n### 四、分布式追踪（OpenTelemetry \u002F Jaeger）\n\n- 目的：追踪请求在应用内部与外部依赖间的延迟与调用顺序，快速定位慢端或依赖问题。\n- 关键实践：\n  - 初始化一个 tracer（在 `core`），在 HTTP middleware 中为每个请求创建 root span，并将 trace 信息注入 downstream（例如 RPC\u002FHTTP client）。\n  - 将 trace_id 写入日志与 response header（如 `X-Trace-Id`），便于排查。 \n  - 采样策略：生产环境选择合适的采样率（例如 1% 或基于错误\u002F慢请求采样）。\n\n- 常见工具链：OpenTelemetry Collector -> Jaeger\u002FZipkin 或直接导出至云厂商的 APM 服务。\n\n### 五、Alert 策略与 SLO\n\n- 指标到告警的建议：\n  - 关键业务错误率（5 分钟内 error_rate > threshold）\n  - 请求 P95\u002FP99 延迟异常上升\n  - 主从数据库延迟、连接池耗尽\n  - 磁盘\u002F磁盘使用率、队列积压\n\n- 把 SLO（服务等级目标）写成可测量的指标，例如“99.9% 的请求 P95 时延 \u003C 500ms”。告警按 SLO 违背程度分为 P0\u002FP1\u002F…\n\n### 六、在现有架构中的落地步骤（按优先级）\n\n1. 初始化 logger（结构化）并在所有模块统一使用。\n2. 在 `core` 初始化 Prometheus client 与 tracer SDK，并创建 `\u002Fmetrics` 与 `\u002Fhealthz` 接口。\n3. 在 `middleware` 中埋点（请求计数、时延、状态码、trace 注入）。\n4. 在 `service` 层为关键业务添加业务指标（计数器\u002F直方图）。\n5. 配置 Prometheus 抓取并在 Grafana 上建立基础 dashboard（HTTP QPS、错误率、P95\u002FP99、DB latency）。\n6. 逐步引入 tracing，并在生产选择低采样率先观测关键路径。\n\n### 七、实操示例（高层伪代码，避免泄露真实源码）\n\n- Middleware 埋点思路：\n  1. 从请求 header 中读取或生成 request_id\u002Ftrace_id。\n  2. 在 context 中存储这些 id 并开始一个 span。\n  3. 记录请求开始时的时间，完成后更新 Prometheus histogram 和 counter，同时结束 span。\n\n- Service 指标示例：\n  - `article_publish_duration_seconds` (histogram)\n  - `article_publish_failures_total` (counter)\n\n### 八、运维与成本注意事项\n\n- 指标基数：不要将高基数标签放到指标里（如 user_id），否则会导致 Prometheus 存储成本暴涨。 \n- Tracing 存储成本：采样与保留期需要平衡，完整 trace 存储昂贵，常用采样+索引策略降低成本。 \n- 日志存储：日志长期保留会产生成本，分级存储（热\u002F冷）和日志裁剪策略是必要的。\n\n### 九、常见问题 FAQ\n\nQ：metrics 会暴露敏感信息吗？\nA：避免在指标 label 中放置敏感或 PII 数据，metrics 应该是汇总级别的数据。\n\nQ：如何把旧服务接入可观测性？\nA：先从最容易的开始：结构化日志 -> \u002Fmetrics -> middleware 埋点 -> tracing。分阶段上线并验证。\n\n### 十、结语\n\n可观测性是使系统可维护、可诊断的核心能力。把它作为工程化的一部分，从一开始就设计（而不是事后补救），会大幅降低排障成本与事故影响面。","## This article: Building observability on the Go backend (Prometheus + Tracing + Logging)\n\n### 1. Overall architecture recommendations\n\nIn order to integrate observability into existing projects (config\u002Fcore\u002Fmodels\u002Fservice\u002Frouters\u002Fmiddleware), a three-layer monitoring solution is recommended:\n\n1. Metrics: Prometheus metrics are used to quantify system health and performance. Buried points are usually located at the HTTP layer, database access, external API calls, and critical business logic. \n2. Logging: Structured logs (such as zap) are used to record request contexts, error stacks, and important events, and are used for auditing and cross-verification metrics. \n3. Distributed Tracing: Export to Jaeger\u002FZipkin using OpenTelemetry\u002FOpenTracing to present the request invocation chain, cross-service latency distribution and dependencies.\n\nPut these capabilities in 'middleware'(HTTP layer buried points, request duration, trace injection),'core'(initialization metrics\u002Ftracer\u002Flogger), and 'service'(business-level metrics).\n\n### 2. Key points for the implementation of Prometheus\n\n- Indicator type:\n  - Counter: Total number of requests, number of errors, number of retries.\n  - Gauge (measured value): Current number of connections, queue length, cache hit rate (percentage).\n  - Histogram\u002FSummary: Request latency distribution, database query latency. Histogram is more suitable for aggregation in Prometheus.\n\n- Embedding:\n  - Initialize a global registry at core or use the default registry. \n  - Buried request count and latency in 'middleware': labeling by route and status code (method, route, status).\n  - Set up business indicators for key businesses at the service level (such as time spent publishing articles and failure rate to write comments).\n\n- Indicator naming and labeling strategies:\n  - Use consistent prefixes such as `appname_http_requests_total`,`appname_db_query_duration_seconds`.\n  - Don't have too many tags and avoid high cardinality (for example, don't use user_id as a label).\n\n- Expose the `\u002Fmetrics` interface:\n  - Do not expose naked metrics to the public network only on internal networks or through reverse proxy protection.\n\n- Example tool chains: prometheus, grafana, alertmanager.\n\n### 3. Associating structured logs (zap) with logs\n\n- Using structured logging (such as zap), convention output JSON. The fields contain at least: timestamp, level, msg, service, env, trace_id, span_id, request path, request ID. \n- Inject trace_id \u002F request_id into the context in middleware and output it in each log to link the log with tracing. \n- Log level policy:\n  - Debug: Development\u002Fdebugging environment, including SQL\u002Fheaders (sensitive data masking).\n  - Info: Normal operation information, key events. \n  - Error: Errors and exceptions, including stack information (moderate).\n\n- Log sampling: Sample debug\u002Finfo in high-traffic scenarios to avoid log storms. It is recommended to integrate the sampler on the collection side (such as fluentd\u002FLogstash) or SDK.\n\n### 4. Distributed tracing (OpenTelemetry \u002F Jaeger)\n\n- Purpose: Track the delay and call order of requests between internal and external dependencies of the application, and quickly locate slow-end or dependency issues.\n- Key practices:\n  - Initialize a tracer (in `core`), create a root span in HTTP middleware for each request, and inject trace information into the downstream (for example, RPC\u002FHTTP client).\n  - Write trace_id to the log and response header (such as `X-Trace-Id`) for easy troubleshooting. \n  - Sampling strategy: Select the appropriate sampling rate for the production environment (e.g. 1% or sampling based on erroneous\u002Fslow requests).\n\n- Common tool chains: OpenTelemetry Collector -> Jaeger\u002FZipkin or APM services exported directly to cloud vendors.\n\n### 5. Alert Policy and SLO\n\n- Suggestions from indicators to alarms:\n  - Critical business error rate (error_rate > threshold within 5 minutes)\n  - Request P95\u002FP99 Delay Abnormal Rise\n  - Master-slave database latency, connection pool depletion\n  - Disk\u002Fdisk utilization, queue backlog\n\n- Write SLO (Service Level Objective) as a measurable metric, such as \"99.9% of requests P95 latency \u003C 500ms\". Alarms are divided into P0\u002FP1\u002F…\n\n### 6. Implementation steps in existing architecture (by priority)\n\n1. Initialize the logger and use it uniformly in all modules.\n2. Initialize the Prometheus client and tracer SDK in the `core`, and create the `\u002Fmetrics` and `\u002Fhealthz` interfaces.\n3. Buried points in 'middleware'(request counting, latency, status code, trace injection).\n4. Add business metrics (counters\u002Fhistograms) to critical businesses at the `service` level.\n5. Configure Prometheus scraping and establish basic dashboards (HTTP QPS, error rate, P95\u002FP99, DB latency) on Grafana.\n6. Gradually introduce tracing, and select a low sampling rate in production to observe critical paths first.\n\n### 7. Practical examples (high-level pseudo-code to avoid revealing the real source code)\n\n- Middleware buried some ideas:\n  1. Read or generate request_id\u002Ftrace_id from the request header.\n  2. Store these ids in context and start a span.\n  3. Record the time when the request started, update the Prometheus histogram and counter after completion, and end the span.\n\n- Example of Service metrics:\n  - `article_publish_duration_seconds` (histogram)\n  - `article_publish_failures_total` (counter)\n\n### 8. Precautions for operation, maintenance and cost\n\n- Indicator base: Don't put high base tags in indicators (such as user_id), otherwise Prometheus storage costs will skyrocket. \n- Tracing storage costs: Sampling and retention periods need to be balanced, full trace storage is expensive, and common sampling + indexing strategies reduce costs. \n- Log storage: Long-term log retention incurs costs, and hierarchical storage (hot\u002Fcold) and log clipping strategies are necessary.\n\n### 9. FAQ of Common Questions\n\nQ: Will metrics expose sensitive information?\nA: Avoid placing sensitive or PII data in indicator labels. Metrics should be aggregated level data.\n\nQ: How to connect old services to observability?\nA: Let's start with the easiest one: structured logging-> \u002Fmetrics -> middleware embedding-> tracing. Launch and verify in stages.\n\n### 10. Conclusion\n\nObservability is the core capability that makes systems maintainable and diagnosable. Designing it from the beginning as part of engineering (rather than remediating it afterwards) will significantly reduce the cost of troubleshooting and the impact of the accident.","企业必备",0,"https:\u002F\u002Fblog4-1316398321.cos.ap-nanjing.myqcloud.com\u002Fblog5\u002F20250823013157__暗黑光影.png",[15],"GO",false,{"id":18,"title":19,"title_en":20},26,"Go 项目中稳健使用 GORM 的实践与思考","Practice and Thinking on Robust Use of GORM in Go Projects",{"id":22,"title":23,"title_en":24},28," Go语言性能分析"," Go language performance analysis","2025-08-23T09:17:15+08:00"]