This article: Building observability on the Go backend (Prometheus + Tracing + Logging)

1. Overall architecture recommendations

In order to integrate observability into existing projects (config/core/models/service/routers/middleware), a three-layer monitoring solution is recommended:

1. Metrics: Prometheus metrics are used to quantify system health and performance. Buried points are usually located at the HTTP layer, database access, external API calls, and critical business logic.
2. Logging: Structured logs (such as zap) are used to record request contexts, error stacks, and important events, and are used for auditing and cross-verification metrics.
3. Distributed Tracing: Export to Jaeger/Zipkin using OpenTelemetry/OpenTracing to present the request invocation chain, cross-service latency distribution and dependencies.

Put these capabilities in 'middleware'(HTTP layer buried points, request duration, trace injection),'core'(initialization metrics/tracer/logger), and 'service'(business-level metrics).

2. Key points for the implementation of Prometheus

• Indicator type:

  • Counter: Total number of requests, number of errors, number of retries.
  • Gauge (measured value): Current number of connections, queue length, cache hit rate (percentage).
  • Histogram/Summary: Request latency distribution, database query latency. Histogram is more suitable for aggregation in Prometheus.

• Embedding:

  • Initialize a global registry at core or use the default registry.
  • Buried request count and latency in 'middleware': labeling by route and status code (method, route, status).
  • Set up business indicators for key businesses at the service level (such as time spent publishing articles and failure rate to write comments).

• Indicator naming and labeling strategies:

  • Use consistent prefixes such as appname_http_requests_total,appname_db_query_duration_seconds.
  • Don't have too many tags and avoid high cardinality (for example, don't use user_id as a label).

• Expose the /metrics interface:

  • Do not expose naked metrics to the public network only on internal networks or through reverse proxy protection.

• Example tool chains: prometheus, grafana, alertmanager.

3. Associating structured logs (zap) with logs

• Using structured logging (such as zap), convention output JSON. The fields contain at least: timestamp, level, msg, service, env, trace_id, span_id, request path, request ID.

• Inject trace_id / request_id into the context in middleware and output it in each log to link the log with tracing.

• Log level policy:

  • Debug: Development/debugging environment, including SQL/headers (sensitive data masking).
  • Info: Normal operation information, key events.
  • Error: Errors and exceptions, including stack information (moderate).

• Log sampling: Sample debug/info in high-traffic scenarios to avoid log storms. It is recommended to integrate the sampler on the collection side (such as fluentd/Logstash) or SDK.

4. Distributed tracing (OpenTelemetry / Jaeger)

• Purpose: Track the delay and call order of requests between internal and external dependencies of the application, and quickly locate slow-end or dependency issues.

• Key practices:

  • Initialize a tracer (in core), create a root span in HTTP middleware for each request, and inject trace information into the downstream (for example, RPC/HTTP client).
  • Write trace_id to the log and response header (such as X-Trace-Id) for easy troubleshooting.
  • Sampling strategy: Select the appropriate sampling rate for the production environment (e.g. 1% or sampling based on erroneous/slow requests).

• Common tool chains: OpenTelemetry Collector -> Jaeger/Zipkin or APM services exported directly to cloud vendors.

5. Alert Policy and SLO

• Suggestions from indicators to alarms:

  • Critical business error rate (error_rate > threshold within 5 minutes)
  • Request P95/P99 Delay Abnormal Rise
  • Master-slave database latency, connection pool depletion
  • Disk/disk utilization, queue backlog

• Write SLO (Service Level Objective) as a measurable metric, such as "99.9% of requests P95 latency < 500ms". Alarms are divided into P0/P1/…

6. Implementation steps in existing architecture (by priority)

1. Initialize the logger and use it uniformly in all modules.
2. Initialize the Prometheus client and tracer SDK in the core, and create the /metrics and /healthz interfaces.
3. Buried points in 'middleware'(request counting, latency, status code, trace injection).
4. Add business metrics (counters/histograms) to critical businesses at the service level.
5. Configure Prometheus scraping and establish basic dashboards (HTTP QPS, error rate, P95/P99, DB latency) on Grafana.
6. Gradually introduce tracing, and select a low sampling rate in production to observe critical paths first.

7. Practical examples (high-level pseudo-code to avoid revealing the real source code)

• Middleware buried some ideas:

  1. Read or generate request_id/trace_id from the request header.
  2. Store these ids in context and start a span.
  3. Record the time when the request started, update the Prometheus histogram and counter after completion, and end the span.

• Example of Service metrics:

  • article_publish_duration_seconds (histogram)
  • article_publish_failures_total (counter)

8. Precautions for operation, maintenance and cost

• Indicator base: Don't put high base tags in indicators (such as user_id), otherwise Prometheus storage costs will skyrocket.
• Tracing storage costs: Sampling and retention periods need to be balanced, full trace storage is expensive, and common sampling + indexing strategies reduce costs.
• Log storage: Long-term log retention incurs costs, and hierarchical storage (hot/cold) and log clipping strategies are necessary.

9. FAQ of Common Questions

Q: Will metrics expose sensitive information?
A: Avoid placing sensitive or PII data in indicator labels. Metrics should be aggregated level data.

Q: How to connect old services to observability?
A: Let's start with the easiest one: structured logging-> /metrics -> middleware embedding-> tracing. Launch and verify in stages.

10. Conclusion

Observability is the core capability that makes systems maintainable and diagnosable. Designing it from the beginning as part of engineering (rather than remediating it afterwards) will significantly reduce the cost of troubleshooting and the impact of the accident.