[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-57":3},{"id":4,"title":5,"title_en":6,"abstract":7,"abstract_en":8,"content":9,"content_en":10,"category":11,"banner_id":12,"banner_path":13,"tags":14,"is_recommend":17,"prev_article":18,"next_article":22,"created_at":26},57,"用 Go + Redis ZSET 实现滑动窗口告警引擎与 Gemini AI 根因分析","Building a Sliding Window Alerting Engine in Go with Redis ZSET and Gemini AI Root Cause Analysis\n","做实时日志系统的时候，告警功能几乎是标配。\n\n最开始我的实现很简单粗暴：日志进来，逐条匹配规则，满足","When designing a real-time data or logging pipeline, alerting is a fundamental requirement.","做实时日志系统的时候，告警功能几乎是标配。\n\n最开始我的实现很简单粗暴：日志进来，逐条匹配规则，满足条件就发告警。\n\n结果线上跑了一段时间，有一次某个服务疯狂报错，告警系统在两分钟内发出了 200 多条通知。运维的手机被叮个不停，最后直接把通知关掉了——反而错过了真正需要关注的窗口期。\n\n这就是**告警风暴（Alert Storm）**，也是大部分自研告警引擎最容易踩的坑。\n\n这篇文章讲三件事：\n\n1. 用 Redis ZSET 实现**滑动窗口计数**，取代\"每条日志都触发告警\"的简单计数\n2. 用 Redis 冷却锁**防止同一规则在短时间内重复告警**\n3. 告警触发后，异步调用 **Gemini 大模型**自动完成根因分析，生成 Markdown 诊断报告\n\n\n---\n\n## 一、为什么不能用简单计数？\n\n假设你有一条告警规则：\n\n> 在 60 秒内，日志里出现 `connection refused` 超过 10 次，就触发告警。\n\n最直觉的实现是用 Redis INCR 计一个固定的计数器，每 60 秒重置一次。\n\n问题在哪？**固定窗口有边界效应。**\n\n比如计数器在第 59 秒重置，而错误恰好集中在第 58 秒到第 61 秒之间发生了 20 次：\n\n```\n|--- 窗口1 ---|--- 窗口2 ---|\n... 10次 | 10次 ...\n```\n\n窗口 1 里 10 次，窗口 2 里 10 次，两个窗口都没超过阈值，**告警压根没触发**——但实际上短短 3 秒内出现了 20 次错误，已经是严重事故了。\n\n**滑动窗口**解决的就是这个问题：它不是以固定时间点作为窗口边界，而是以\"**当前时刻往前推 N 秒**\"作为窗口，窗口随时间滑动，没有边界效应。\n\n---\n\n## 二、用 Redis ZSET 实现滑动窗口\n\nRedis ZSET（有序集合）是实现滑动窗口的最优解：\n\n- **Key**：唯一标识这条告警规则的计数桶\n- **Member**：每次命中时写入一条记录（用 UUID 保证唯一性）\n- **Score**：写入时的**时间戳（毫秒）**\n\n有了这个结构，\"窗口内有多少命中\"这个问题就变成了：\n\n> 在 ZSET 里，Score 在 `(now - windowMs, now]` 范围内的元素有多少个？\n\n```\nZSET key: alert:rule:42:hits\n│\n├── member: uuid-aaa  score: 1720000001000  (01:00:01)\n├── member: uuid-bbb  score: 1720000003500  (01:00:03)\n├── member: uuid-ccc  score: 1720000058000  (01:00:58) ← 过期，需清理\n└── member: uuid-ddd  score: 1720000061000  (01:01:01)\n```\n\n代码实现：\n\n```go\n\u002F\u002F SlideWindowCount 基于 Redis ZSET 实现滑动窗口计数\n\u002F\u002F ruleID: 规则的唯一 ID\n\u002F\u002F windowSec: 时间窗口大小（秒）\n\u002F\u002F 返回：当前窗口内的命中次数\nfunc SlideWindowCount(ctx context.Context, ruleID int, windowSec int) (int64, error) {\n    key := fmt.Sprintf(\"alert:rule:%d:hits\", ruleID)\n    \n    now := time.Now()\n    nowMs := now.UnixNano() \u002F int64(time.Millisecond) \u002F\u002F 当前时间戳（毫秒）\n    windowMs := int64(windowSec) * 1000               \u002F\u002F 窗口大小（毫秒）\n\n    \u002F\u002F 用 Pipeline 把三个操作打包，减少网络往返，同时保证操作的整体顺序\n    pipe := rdb.Pipeline()\n\n    \u002F\u002F 1. 写入本次命中（Score = 当前时间戳，Member = UUID 保证唯一）\n    pipe.ZAdd(ctx, key, redis.Z{\n        Score:  float64(nowMs),\n        Member: uuid.New().String(),\n    })\n\n    \u002F\u002F 2. 清理已经滑出窗口的历史数据（Score \u003C cutoff 的全部删掉）\n    cutoff := float64(nowMs - windowMs)\n    pipe.ZRemRangeByScore(ctx, key, \"-inf\", fmt.Sprintf(\"%v\", cutoff))\n\n    \u002F\u002F 3. 统计当前窗口内的元素数量\n    cardCmd := pipe.ZCard(ctx, key)\n\n    \u002F\u002F 执行 Pipeline\n    _, err := pipe.Exec(ctx)\n    if err != nil {\n        return 0, err\n    }\n\n    return cardCmd.Val(), nil\n}\n```\n\n这里有几个细节值得拿出来说：\n\n**为什么 Member 用 UUID，不用时间戳？**\n\n如果用时间戳作 Member，在同一毫秒内有两条日志同时命中，第二条的 ZAdd 会覆盖第一条（因为 Member 相同），导致少计一次。UUID 保证每次命中都是独立的 Member，不会相互覆盖。\n\n**为什么用 Pipeline，不用事务（MULTI\u002FEXEC）？**\n\nPipeline 把命令批量发送到 Redis，减少网络 RTT，性能更好。而这三个操作（ZAdd、ZRemRange、ZCard）不需要强事务保证——即使中途有别的命令插进来，也只是多了几条记录，不影响窗口计数的最终正确性。\n\n**ZRemRangeByScore 的必要性**\n\n如果不清理历史数据，ZSET 会无限增长，长期运行下去必然 OOM。每次命中时顺手清理掉窗口外的数据，Key 的大小就永远不会超过\"窗口内最大命中次数\"，内存可控。\n\n---\n\n## 三、防止告警风暴：冷却锁\n\n滑动窗口解决了\"何时触发告警\"的问题，但还没解决\"触发了之后疯狂重复告警\"的问题。\n\n想象一下：一个 60 秒窗口、阈值 10 次的规则被触发了。接下来的每一条新日志进来，只要窗口内依然有 10 条以上，就会再次触发——也就是说，只要故障持续，告警会无限发。\n\n解决方案很简单：**告警触发后，用 Redis 设一把带过期时间的\"冷却锁\"**。在锁的有效期内，这条规则不再重复发告警。\n\n```go\n\u002F\u002F IsAlertCooldown 检查这条规则当前是否在冷却期\nfunc IsAlertCooldown(ctx context.Context, ruleID int) bool {\n    key := fmt.Sprintf(\"alert:rule:%d:cooldown\", ruleID)\n    exists, err := rdb.Exists(ctx, key).Result()\n    if err != nil {\n        return false \u002F\u002F Redis 故障时，降级放行（宁可多发一条，不能漏告警）\n    }\n    return exists > 0\n}\n\n\u002F\u002F SetAlertCooldown 设置冷却锁，有效期内该规则不再重复告警\nfunc SetAlertCooldown(ctx context.Context, ruleID int, duration time.Duration) error {\n    key := fmt.Sprintf(\"alert:rule:%d:cooldown\", ruleID)\n    return rdb.Set(ctx, key, \"locked\", duration).Err()\n}\n```\n\n注意 `IsAlertCooldown` 里的降级处理：**Redis 出故障时返回 `false`（放行）**，而不是 `true`（拦截）。\n\n这是一个关键的设计决策：冷却锁是为了减少噪音，但如果因为 Redis 故障就完全屏蔽告警，万一这时候线上真的出了严重问题，你反而什么都收不到。**宁可多发几条重复告警，不能因为降级把真正的告警也干掉。**\n\n---\n\n## 四、规则缓存：不能在日志热路径上查数据库\n\n告警规则存在数据库里，但告警引擎是在日志处理的**热路径**上异步执行的——每一条进来的日志都可能触发 `checkAlert`，如果每次都去数据库查规则，QPS 高的时候数据库直接被打挂。\n\n解决方案是在内存里维护一个**本地规则缓存**：\n\n```go\n\u002F\u002F CachedRule 包含一条告警规则，以及它预编译好的正则表达式\ntype CachedRule struct {\n    ID         int\n    Name       string\n    Pattern    string          \u002F\u002F 原始正则字符串\n    Regexp     *regexp.Regexp  \u002F\u002F ← 预编译好的正则句柄，直接复用\n    Source     string          \u002F\u002F 来源过滤，空字符串表示不过滤\n    WindowSec  int             \u002F\u002F 时间窗口（秒）\n    Threshold  int             \u002F\u002F 触发阈值\n}\n\n\u002F\u002F ruleCache 以 channelID 为 Key 的本地规则缓存\ntype ruleCache struct {\n    sync.RWMutex\n    data map[uint64][]*CachedRule\n}\n\nvar cache = &ruleCache{\n    data: make(map[uint64][]*CachedRule),\n}\n```\n\n**为什么把 `*regexp.Regexp` 缓存起来？**\n\n`regexp.Compile` 是有开销的，如果每条日志都重新编译一遍正则，CPU 消耗非常可观。缓存预编译好的 `*regexp.Regexp` 句柄，匹配时直接调用 `r.Regexp.MatchString()`，性能提升显著——而且 `*regexp.Regexp` 是并发安全的，多个 goroutine 同时读没有问题。\n\n查规则时，先走缓存，未命中再回源数据库：\n\n```go\nfunc getRulesFromCache(ctx context.Context, channelID uint64) ([]*CachedRule, error) {\n    \u002F\u002F 先加读锁查缓存\n    cache.RLock()\n    rules, exists := cache.data[channelID]\n    cache.RUnlock()\n\n    if exists {\n        return rules, nil \u002F\u002F 命中缓存，直接返回\n    }\n\n    \u002F\u002F 缓存未命中，查数据库（此处替换为你的查询逻辑）\n    dbRules, err := queryEnabledRules(ctx, channelID)\n    if err != nil {\n        return nil, err\n    }\n\n    \u002F\u002F 预编译正则，组装 CachedRule\n    cached := make([]*CachedRule, 0, len(dbRules))\n    for _, r := range dbRules {\n        reg, err := regexp.Compile(r.Pattern)\n        if err != nil {\n            \u002F\u002F 正则编译失败，记录日志跳过，不能因为一条脏数据崩掉整个引擎\n            log.Error(\"正则编译失败\", \"rule_id\", r.ID, \"pattern\", r.Pattern, \"err\", err)\n            continue\n        }\n        cached = append(cached, &CachedRule{\n            ID:        r.ID,\n            Name:      r.Name,\n            Pattern:   r.Pattern,\n            Regexp:    reg,\n            Source:    r.Source,\n            WindowSec: r.WindowSec,\n            Threshold: r.Threshold,\n        })\n    }\n\n    \u002F\u002F 写入缓存（加写锁）\n    cache.Lock()\n    cache.data[channelID] = cached\n    cache.Unlock()\n\n    return cached, nil\n}\n```\n\n**规则变更时的缓存失效**\n\n当用户新增、修改、删除了一条告警规则，对应频道的缓存就必须失效，否则新规则不会生效、老规则不会退出。在规则的 CRUD Service 里调用这个函数即可：\n\n```go\n\u002F\u002F InvalidateCache 主动失效某个频道的规则缓存（在规则 CRUD 后调用）\nfunc InvalidateCache(channelID uint64) {\n    cache.Lock()\n    defer cache.Unlock()\n    delete(cache.data, channelID)\n}\n```\n\n---\n\n## 五、把它们串在一起\n\n整个告警引擎的判定流程：\n\n```go\n\u002F\u002F checkAlert 实时告警核心逻辑（在日志处理 Worker 中异步拉起，不阻塞主流程）\nfunc checkAlert(ctx context.Context, channelID uint64, source, message string) {\n    \u002F\u002F 1. 从缓存获取该频道的所有启用规则\n    rules, err := getRulesFromCache(ctx, channelID)\n    if err != nil {\n        log.Error(\"获取告警规则失败\", \"err\", err)\n        return\n    }\n\n    for _, r := range rules {\n        \u002F\u002F 2. 来源过滤（规则可以配置只匹配特定来源的日志）\n        if r.Source != \"\" && r.Source != source {\n            continue\n        }\n\n        \u002F\u002F 3. 正则匹配（直接使用预编译好的句柄）\n        if r.Regexp == nil || !r.Regexp.MatchString(message) {\n            continue\n        }\n\n        \u002F\u002F 4. 滑动窗口计数：在时间窗口内这条规则被命中了多少次？\n        count, err := SlideWindowCount(ctx, r.ID, r.WindowSec)\n        if err != nil {\n            log.Error(\"滑动窗口计数失败\", \"rule_id\", r.ID, \"err\", err)\n            continue\n        }\n\n        \u002F\u002F 5. 判断是否达到阈值\n        if count \u003C int64(r.Threshold) {\n            continue \u002F\u002F 还没到，继续等待\n        }\n\n        \u002F\u002F 6. 检查冷却锁，防止告警风暴\n        if IsAlertCooldown(ctx, r.ID) {\n            continue \u002F\u002F 还在冷却期，跳过\n        }\n\n        \u002F\u002F 7. 设置冷却锁（3 分钟内不再重复告警）\n        _ = SetAlertCooldown(ctx, r.ID, 3*time.Minute)\n\n        \u002F\u002F 8. 写入告警事件记录，并异步拉起 AI 根因分析\n        event := fireAlert(ctx, r, channelID, message, count)\n        go asyncAnalyzeAndBackfill(event.ID, channelID, r) \u002F\u002F AI 分析，下一章详述\n    }\n}\n```\n\n整个流程的数据流：\n\n```\n日志进来\n    │\n    ▼\n来源过滤（Source 字段）\n    │\n    ▼\n正则匹配（预编译 Regexp）── 不匹配 → 跳过\n    │\n    ▼\nSlideWindowCount (Redis ZSET)\n    │\n    ▼\n命中数 >= 阈值？── 否 → 跳过\n    │\n    ▼\nIsAlertCooldown (Redis Key)── 是 → 跳过（冷却中）\n    │\n    ▼\nSetAlertCooldown（锁定 3 分钟）\n    │\n    ▼\n写入 AlertEvent 记录\n    │\n    ├── （同步）通知运维\n    │\n    └── （异步 goroutine）Gemini AI 根因分析 → 回填诊断报告\n```\n\n\n---\n\n## 六、AI 根因分析：告警触发后让大模型帮你定位问题\n\n告警发出去了，运维开始排查。下一步通常是：翻日志、找时间段、猜是哪里出了问题。\n\n这个过程完全可以让大模型来做。告警触发的那一刻，我们已经知道：**哪条规则被触发了、时间窗口是多少、用什么正则匹配到了什么**——把这些信息连同触发窗口内的原始日志一起喂给模型，它能直接给出一份专业的 SRE 根因分析报告。\n\n### 6.1 整体设计\n\n告警触发 → 写入 AlertEvent 记录（此时 `ai_analysis` 字段为空）→ 异步拉起 AI 分析协程 → 等待日志 flush → 查询触发窗口内的日志 → 调用 Gemini → 回填 `ai_analysis` 字段。\n\n整个分析过程完全异步，不阻塞告警主流程，用户打开告警详情时，报告可能已经生成好了。\n\n### 6.2 一个必须有的等待：`time.Sleep(1 * time.Second)`\n\n```go\nfunc asyncAnalyzeAndBackfill(eventID int, channelID uint64, rule *CachedRule) {\n    \u002F\u002F 必须等待 1 秒！\n    \u002F\u002F 日志处理采用批量写入策略（攒够 N 条或每 500ms flush 一次）\n    \u002F\u002F 如果 AI 分析协程立即查数据库，触发告警的那批日志可能还在内存 Buffer 里\n    \u002F\u002F 等 1 秒确保这批日志已经落盘，查到的上下文才是完整的\n    time.Sleep(1 * time.Second)\n\n    ctx := context.Background() \u002F\u002F 见下方解释，不能用请求的 ctx\n\n    \u002F\u002F 查询时间窗口内最近 20 条日志作为 AI 分析的上下文\n    startTime := time.Now().Add(-time.Duration(rule.WindowSec) * time.Second)\n    triggerLogs, err := queryRecentLogs(ctx, channelID, startTime, 20)\n    if err != nil {\n        log.Error(\"AI 分析：查询上下文日志失败\", \"err\", err)\n        return\n    }\n\n    \u002F\u002F 调用 Gemini 生成根因分析报告\n    report, err := analyzeWithGemini(ctx, rule, triggerLogs)\n    if err != nil {\n        log.Error(\"AI 分析：生成报告失败\", \"err\", err)\n        return\n    }\n\n    \u002F\u002F 回填到 AlertEvent 记录\n    if err := backfillReport(ctx, eventID, report); err != nil {\n        log.Error(\"AI 分析：回填失败\", \"err\", err)\n    }\n}\n```\n\n**为什么用 `context.Background()` 而不是传入的 `ctx`？**\n\n告警引擎是从日志处理 Worker 里异步拉起的。如果这个 Worker 的 `ctx` 和某个 HTTP 请求或上报通道绑定，请求结束时 `ctx` 就会被取消，AI 分析协程的数据库查询和 Gemini 调用会被立即中断。\n\nAI 分析调用 Gemini 可能需要几秒，必须用独立的 `context.Background()` 保证协程的生命周期不受外部影响。\n\n### 6.3 为什么只取 20 条日志？\n\n两个原因：\n\n**精准度**：触发告警的关键日志通常就密集出现在时间窗口末尾，取太多反而把模型的注意力分散到无关日志上，分析结论会变模糊。20 条对根因定位来说已经足够。\n\n**Token 成本**：每次 AI 调用都是真金白银。如果不限制日志数量，一次高频告警可能把几百条日志全塞进 prompt，Token 费用会非常可观。限制 20 条是在**分析质量和成本**之间取的平衡点。\n\n### 6.4 Prompt 工程：让模型以 SRE 视角分析\n\n模型输出的质量，90% 取决于 prompt 写得好不好。对于日志根因分析，推荐这样构建 prompt：\n\n```go\nfunc buildPrompt(rule *CachedRule, logs []LogEntry) string {\n    var logText strings.Builder\n    for i, l := range logs {\n        logText.WriteString(fmt.Sprintf(\n            \"[%d] [%s] [%s] %s | source: %s\\n\",\n            i+1, l.CreatedAt.Format(\"15:04:05\"), l.Level, l.Message, l.Source,\n        ))\n    }\n\n    return fmt.Sprintf(`你是一名经验丰富的 SRE 工程师，请分析以下告警事件的根本原因并给出处置建议。\n\n## 告警规则\n- 规则名称：%s\n- 匹配模式（正则）：%s\n- 触发条件：%d 秒内匹配 %d 次\n\n## 触发窗口内的原始日志（最近 %d 条）\n%s\n\n## 请输出以下内容（Markdown 格式）：\n1. **根本原因**：用一句话概括\n2. **详细分析**：结合日志内容展开，指出关键错误模式\n3. **影响范围**：推断受影响的服务或组件\n4. **处置建议**：给出 2-3 条具体可操作的排查步骤`,\n        rule.Name, rule.Pattern, rule.WindowSec, rule.Threshold, len(logs), logText.String(),\n    )\n}\n```\n\n### 6.5 Temperature = 0.2：调低随机性\n\nGemini 的 `Temperature` 参数默认值是 1.0。对于创意写作，高随机性是好事；但对于日志根因分析，我们要的是**稳定、准确**的技术输出，不需要模型发挥想象力。\n\n设置 `Temperature = 0.2` 让模型更专注于日志内容，减少\"幻觉（Hallucination）\"，输出更贴近实际问题：\n\n```go\nmodelInstance := client.GenerativeModel(\"gemini-2.0-flash\")\nvar temperature float32 = 0.2\nmodelInstance.Temperature = &temperature\n\nresp, err := modelInstance.GenerateContent(ctx, genai.Text(prompt))\n```\n\n### 6.6 实际效果\n\n告警触发后大约 3-5 秒，AlertEvent 记录里的 `ai_analysis` 字段就会填入一份 Markdown 报告：\n\n```markdown\n## 根本原因\n数据库连接池耗尽，导致所有需要数据库操作的请求超时失败。\n\n## 详细分析\n从日志时间线来看，错误首次出现在 01:03:21，初始错误为 `connection refused`，\n随后在 12 秒内迅速蔓延至全部 Worker...\n\n## 影响范围\n受影响服务：user-service、order-service\n错误高峰期：01:03:21 - 01:05:47\n\n## 处置建议\n1. 立即检查数据库连接池配置，适当增大 max_open_conns\n2. 查看数据库服务器负载，确认是否存在慢查询堆积\n3. 检查最近是否有流量突增或批量任务并发执行\n```\n\n运维打开告警详情的时候，报告已经准备好了，可以直接按建议排查，不需要再手动翻日志。\n\n---\n\n## 七、几个容易被忽略的细节\n\n**1. 为什么 `checkAlert` 要异步调用？**\n\n告警引擎里有数据库查询和多次 Redis 操作，是有 I\u002FO 开销的。如果同步执行，会阻塞日志处理的主流程，影响日志写入的吞吐量。用 `go checkAlert(...)` 异步拉起，日志处理主流程立即返回，告警判定在后台并发执行。\n\n代价是：告警判定的时序和日志实际到达的时序可能有轻微偏差，但对于告警场景来说，几十毫秒的偏差完全可以接受。\n\n**2. 正则编译失败时，为什么要跳过而不是报错退出？**\n\n如果数据库里有一条历史脏数据（正则写错了），编译失败时直接 `panic` 或者返回 `error` 中断整个流程，会导致这个频道的**所有规则**都失效。\n\n正确的做法是跳过这条坏规则、记录日志告警、继续处理剩余规则。让一条烂规则影响全局是不可接受的。\n\n**3. ZSET Key 为什么不需要单独设过期时间？**\n\n每次 `SlideWindowCount` 都会用 `ZRemRangeByScore` 清理过期的旧数据。在没有命中时，Key 里没有任何元素（或者元素被清空），这个 Key 对内存的占用就是零（Redis 的空 ZSET 会自动被 GC）。所以不需要额外的过期时间管理。\n\n---\n\n## 总结\n\n| 组件 | 实现方式 | 解决的问题 |\n|------|---------|---------|\n| 滑动窗口计数 | Redis ZSET（Score = 时间戳，UUID Member） | 固定窗口的边界效应，精准统计任意时间段内的命中次数 |\n| 内存规则缓存 | `sync.RWMutex` + `map` + 预编译正则 | 日志热路径不查 DB，规则变更时主动失效 |\n| 告警冷却锁 | Redis Key + TTL | 防止同一规则在故障持续期间无限重复告警 |\n| Redis 降级策略 | 故障时放行 | Redis 挂了不能把真实告警也屏蔽掉 |\n| AI 根因分析 | Gemini + 异步协程 + 回填 | 告警触发后自动生成诊断报告，减少人工排查时间 |\n\n整套系统的核心理念是**分层处理**：正则做第一层过滤（极低开销）→ 滑动窗口精准计数（防误报）→ 冷却锁降噪（防风暴）→ AI 智能诊断（提效）。每一层各司其职，可以根据需要单独替换或升级。\n\n希望对你有帮助，有问题欢迎在评论区讨论！\n\n","When designing a real-time data or logging pipeline, alerting is a fundamental requirement.\n\nMy first attempt was straightforward: match incoming logs against alert rules line-by-line, and trigger an alert immediately when a match occurred.\n\nHowever, once it ran in production, a downstream service suffered an outage and began throwing errors continuously. Our alert engine fired over 200 notifications in under two minutes. The SRE team's phones kept ringing until they eventually muted notifications—causing them to miss the actual critical alerts that followed.\n\nThis is the classic **Alert Storm**, a common pitfall when building self-rolled alerting engines.\n\nIn this article, we'll solve this by implementing:\n\n1. **Sliding Window Counting** using Redis ZSET (replacing naive threshold checks).\n2. **Cooldown Locks** using Redis to prevent repetitive alert firing.\n3. **Automated Root Cause Analysis (RCA)** triggered asynchronously via the **Gemini API** to produce Markdown incident reports.\n\n---\n\n## 1. Why Naive Counting Fails: The Boundary Effect\n\nSuppose you define the following alerting rule:\n\n> Trigger an alert if `connection refused` occurs more than 10 times within a 60-second window.\n\nA simple implementation would use Redis `INCR` with a fixed 60-second expiration. \n\nHowever, fixed windows suffer from the **boundary effect**. If the counter resets at the 59th second, and a cluster of errors occurs between the 58th and 61st seconds:\n\n```\n|--- Window 1 ---|--- Window 2 ---|\n... 10 hits      | 10 hits ...\n```\n\nBoth windows record 10 hits, neither crossing the threshold of 10. Consequently, **no alert is triggered**—despite the system receiving 20 critical errors in just 3 seconds.\n\nA **sliding window** addresses this. Instead of dividing time into fixed buckets, it evaluates the window relative to the current timestamp (e.g., `now - 60s`). As time advances, the window slides forward seamlessly, eliminating boundary issues.\n\n---\n\n## 2. Implementing Sliding Windows with Redis ZSET\n\nRedis ZSET (Sorted Set) is the ideal data structure for sliding windows:\n\n- **Key**: A unique identifier for the alerting rule.\n- **Member**: A unique string for each hit (we use a UUID to prevent duplicate overrides).\n- **Score**: The **millisecond timestamp** of when the hit occurred.\n\nCalculating active hits in the current window simplifies to:\n\n> Count elements in the ZSET where the Score falls in the range `(now - windowMs, now]`.\n\n```\nZSET key: alert:rule:42:hits\n│\n├── member: uuid-aaa  score: 1720000001000  (01:00:01)\n├── member: uuid-bbb  score: 1720000003500  (01:00:03)\n├── member: uuid-ccc  score: 1720000058000  (01:00:58) ← Expired, needs cleanup\n└── member: uuid-ddd  score: 1720000061000  (01:01:01)\n```\n\nHere is the Go implementation:\n\n```go\n\u002F\u002F SlideWindowCount implements sliding window counting using Redis ZSET\nfunc SlideWindowCount(ctx context.Context, ruleID int, windowSec int) (int64, error) {\n    key := fmt.Sprintf(\"alert:rule:%d:hits\", ruleID)\n    \n    now := time.Now()\n    nowMs := now.UnixNano() \u002F int64(time.Millisecond) \u002F\u002F Current timestamp in ms\n    windowMs := int64(windowSec) * 1000               \u002F\u002F Window size in ms\n\n    \u002F\u002F Use a Redis Pipeline to bundle commands and minimize RTT\n    pipe := rdb.Pipeline()\n\n    \u002F\u002F 1. Add current hit (Score = timestamp, Member = UUID)\n    pipe.ZAdd(ctx, key, redis.Z{\n        Score:  float64(nowMs),\n        Member: uuid.New().String(),\n    })\n\n    \u002F\u002F 2. Remove stale hits outside the sliding window\n    cutoff := float64(nowMs - windowMs)\n    pipe.ZRemRangeByScore(ctx, key, \"-inf\", fmt.Sprintf(\"%v\", cutoff))\n\n    \u002F\u002F 3. Count elements remaining in the ZSET\n    cardCmd := pipe.ZCard(ctx, key)\n\n    \u002F\u002F Execute pipeline\n    _, err := pipe.Exec(ctx)\n    if err != nil {\n        return 0, err\n    }\n\n    return cardCmd.Val(), nil\n}\n```\n\n### Critical Details:\n\n* **UUID as Member**: If we used the timestamp as the member, concurrent hits occurring in the exact same millisecond would overwrite each other. Using a UUID ensures every hit is uniquely registered.\n* **Pipeline vs. Multi\u002FExec**: We use a pipeline for performance. Since these operations don't strictly require isolation from other parallel commands, we avoid the overhead of full Redis transactions.\n* **ZRemRangeByScore**: Without clearing historical data, the ZSET will grow indefinitely, eventually exhausting memory. Clearing expired members on every hit keeps memory usage bounded by the maximum potential hits in a single window.\n\n---\n\n## 3. Mitigating Alert Storms: Cooldown Locks\n\nSliding windows determine *when* to alert, but they don't prevent repetitive alerts when a fault persists. If a threshold of 10 hits in 60 seconds is met, every subsequent log entry will re-trigger the alert as long as the ZSET contains at least 10 elements.\n\nTo prevent this, we introduce a **cooldown lock** using a Redis key with a Time-To-Live (TTL):\n\n```go\n\u002F\u002F IsAlertCooldown checks if the alerting rule is in its cooldown period\nfunc IsAlertCooldown(ctx context.Context, ruleID int) bool {\n    key := fmt.Sprintf(\"alert:rule:%d:cooldown\", ruleID)\n    exists, err := rdb.Exists(ctx, key).Result()\n    if err != nil {\n        return false \u002F\u002F Fail-open: if Redis fails, proceed with the alert\n    }\n    return exists > 0\n}\n\n\u002F\u002F SetAlertCooldown activates the cooldown lock for a specified duration\nfunc SetAlertCooldown(ctx context.Context, ruleID int, duration time.Duration) error {\n    key := fmt.Sprintf(\"alert:rule:%d:cooldown\", ruleID)\n    return rdb.Set(ctx, key, \"locked\", duration).Err()\n}\n```\n\n> 💡 **Design Pattern (Fail-Open)**: If Redis is unavailable, `IsAlertCooldown` returns `false`. It is better to receive duplicate alerts during a cache outage than to suppress a real incident.\n\n---\n\n## 4. Performance Optimization: Local Rule Caching\n\nAlert evaluation happens in the hot path of the log processor. Performing database lookups for rules on every log line would easily saturate the database.\n\nInstead, cache compiled rules in memory:\n\n```go\ntype CachedRule struct {\n    ID         int\n    Name       string\n    Pattern    string          \u002F\u002F Regex pattern\n    Regexp     *regexp.Regexp  \u002F\u002F Compiled regex handle, reused across goroutines\n    Source     string          \n    WindowSec  int             \n    Threshold  int             \n}\n\ntype ruleCache struct {\n    sync.RWMutex\n    data map[uint64][]*CachedRule\n}\n\nvar cache = &ruleCache{\n    data: make(map[uint64][]*CachedRule),\n}\n```\n\n* **Caching Compiled Regex**: Recompiling patterns via `regexp.Compile` is CPU-heavy. Reusing thread-safe `*regexp.Regexp` pointers drastically reduces CPU load in high-throughput pipelines.\n\nWhen rules are modified in your CRUD operations, invalidate the cache:\n\n```go\nfunc InvalidateCache(channelID uint64) {\n    cache.Lock()\n    defer cache.Unlock()\n    delete(cache.data, channelID)\n}\n```\n\n---\n\n## 5. Automated AI Root Cause Analysis (RCA) with Gemini\n\nOnce an alert fires, engineers must manually inspect log lines, correlate timestamps, and diagnose the root cause. This step can be automated by querying the context window and sending it to **Gemini**.\n\n### 5.1 Architecture Workflow\n\n```\nAlert Triggered ──► Write Incident Event (Report: empty)\n     │\n     └──► Asynchronous Goroutine (Independent Context)\n               │\n               ▼\n         Sleep 1 Second (Allow bulk log writer to flush memory buffer)\n               │\n               ▼\n         Fetch context logs (Last 20 entries)\n               │\n               ▼\n         Request Gemini API (Temperature = 0.2)\n               │\n               ▼\n         Backfill Markdown report into DB\n```\n\n### 5.2 Implementation\n\n```go\nfunc asyncAnalyzeAndBackfill(eventID int, channelID uint64, rule *CachedRule) {\n    \u002F\u002F 1. Wait 1 second to ensure recent logs in the bulk-write buffer\n    \u002F\u002F    have flushed and are queryable in the database.\n    time.Sleep(1 * time.Second)\n\n    \u002F\u002F 2. Use context.Background() so the operation is independent \n    \u002F\u002F    of the request context that triggered the alert.\n    ctx := context.Background() \n\n    \u002F\u002F 3. Fetch context logs surrounding the alert window\n    startTime := time.Now().Add(-time.Duration(rule.WindowSec) * time.Second)\n    triggerLogs, err := queryRecentLogs(ctx, channelID, startTime, 20)\n    if err != nil {\n        log.Error(\"AI Analysis: failed to fetch log context\", \"err\", err)\n        return\n    }\n\n    \u002F\u002F 4. Generate report via Gemini API\n    report, err := analyzeWithGemini(ctx, rule, triggerLogs)\n    if err != nil {\n        log.Error(\"AI Analysis: model generation failed\", \"err\", err)\n        return\n    }\n\n    \u002F\u002F 5. Backfill the Markdown analysis to the incident record\n    _ = backfillReport(ctx, eventID, report)\n}\n```\n\n### 5.3 Optimizing Model Inputs: Why 20 Logs?\n\n* **Accuracy**: Excess context dilutes attention. Feeding the exact log lines surrounding the threshold breach guides the model to the primary error pattern.\n* **Cost Efficiency**: High-frequency alerting pipelines can generate significant API usage. Limiting inputs to 20 lines balances context quality and Token consumption.\n\n### 5.4 Temperature Tuning\n\nFor code debugging or SRE work, set the model's `Temperature` parameter to `0.2` (low). This suppresses creative extrapolation (\"hallucinations\") and forces the model to focus strictly on factual log entries.\n\n```go\nmodelInstance := client.GenerativeModel(\"gemini-2.0-flash\")\nvar temp float32 = 0.2\nmodelInstance.Temperature = &temp\n```\n\n---\n\n## 6. Symmetrical Execution Pipeline\n\nPutting the components together:\n\n```go\nfunc checkAlert(ctx context.Context, channelID uint64, source, message string) {\n    rules, err := getRulesFromCache(ctx, channelID)\n    if err != nil {\n        return\n    }\n\n    for _, r := range rules {\n        if r.Source != \"\" && r.Source != source {\n            continue\n        }\n        if r.Regexp == nil || !r.Regexp.MatchString(message) {\n            continue\n        }\n\n        \u002F\u002F Sliding window check\n        count, err := SlideWindowCount(ctx, r.ID, r.WindowSec)\n        if err != nil || count \u003C int64(r.Threshold) {\n            continue\n        }\n\n        \u002F\u002F Cooldown check\n        if IsAlertCooldown(ctx, r.ID) {\n            continue\n        }\n        _ = SetAlertCooldown(ctx, r.ID, 3*time.Minute)\n\n        \u002F\u002F Write event and kick off AI diagnostic run asynchronously\n        event := fireAlert(ctx, r, channelID, message, count)\n        go asyncAnalyzeAndBackfill(event.ID, channelID, r)\n    }\n}\n```\n\n---\n\n## 7. Summary\n\n| Component | Implementation | Purpose |\n|------|---------|---------|\n| **Sliding Window** | Redis ZSET (Score: timestamp, Member: UUID) | Avoids boundary errors, tracks hits precisely over rolling time |\n| **Rule Cache** | `sync.RWMutex` + Compiled regex pointers | Bypasses database queries in the hot path |\n| **Cooldown Lock** | Redis Key + TTL | Dampens duplicate alert noise |\n| **Fail-Open Policy** | Redis errors return `false` | Prevents cache outages from swallowing critical alerts |\n| **AI RCA** | Gemini API + Asynchronous worker | Generates Markdown diagnostics automatically on alert trigger |\n\nThis architecture decouples filtering, rate-limiting, and analysis. Each layer acts independently, providing a robust SRE pipeline for high-throughput Go services.\n","Go",14,"https:\u002F\u002Fblog4-1316398321.cos.ap-nanjing.myqcloud.com\u002Fblog5\u002F20250716162537__暗夜雏菊.png",[15,16],"GO","Gemini",false,{"id":19,"title":20,"title_en":21},56,"用 Go 从零搭建 WebTransport 实时推送服务","Building a WebTransport Real-Time Push Service from Scratch in Go",{"id":23,"title":24,"title_en":25},0,"已经是最后一篇了","It's already the last one","2026-07-17T10:59:04.611034+08:00"]