Building a Sliding Window Alerting Engine in Go with Redis ZSET and Gemini AI Root Cause Analysis

When designing a real-time data or logging pipeline, alerting is a fundamental requirement.

My first attempt was straightforward: match incoming logs against alert rules line-by-line, and trigger an alert immediately when a match occurred.

However, once it ran in production, a downstream service suffered an outage and began throwing errors continuously. Our alert engine fired over 200 notifications in under two minutes. The SRE team's phones kept ringing until they eventually muted notifications—causing them to miss the actual critical alerts that followed.

This is the classic Alert Storm, a common pitfall when building self-rolled alerting engines.

In this article, we'll solve this by implementing:

Sliding Window Counting using Redis ZSET (replacing naive threshold checks).
Cooldown Locks using Redis to prevent repetitive alert firing.
Automated Root Cause Analysis (RCA) triggered asynchronously via the Gemini API to produce Markdown incident reports.

1. Why Naive Counting Fails: The Boundary Effect

Suppose you define the following alerting rule:

Trigger an alert if connection refused occurs more than 10 times within a 60-second window.

A simple implementation would use Redis INCR with a fixed 60-second expiration.

However, fixed windows suffer from the boundary effect. If the counter resets at the 59th second, and a cluster of errors occurs between the 58th and 61st seconds:

Copy

|--- Window 1 ---|--- Window 2 ---|
... 10 hits      | 10 hits ...

Both windows record 10 hits, neither crossing the threshold of 10. Consequently, no alert is triggered—despite the system receiving 20 critical errors in just 3 seconds.

A sliding window addresses this. Instead of dividing time into fixed buckets, it evaluates the window relative to the current timestamp (e.g., now - 60s). As time advances, the window slides forward seamlessly, eliminating boundary issues.

2. Implementing Sliding Windows with Redis ZSET

Redis ZSET (Sorted Set) is the ideal data structure for sliding windows:

Key: A unique identifier for the alerting rule.
Member: A unique string for each hit (we use a UUID to prevent duplicate overrides).
Score: The millisecond timestamp of when the hit occurred.

Calculating active hits in the current window simplifies to:

Count elements in the ZSET where the Score falls in the range (now - windowMs, now].

Copy

ZSET key: alert:rule:42:hits
│
├── member: uuid-aaa  score: 1720000001000  (01:00:01)
├── member: uuid-bbb  score: 1720000003500  (01:00:03)
├── member: uuid-ccc  score: 1720000058000  (01:00:58) ← Expired, needs cleanup
└── member: uuid-ddd  score: 1720000061000  (01:01:01)

Here is the Go implementation:

go Copy

// SlideWindowCount implements sliding window counting using Redis ZSET
func SlideWindowCount(ctx context.Context, ruleID int, windowSec int) (int64, error) {
    key := fmt.Sprintf("alert:rule:%d:hits", ruleID)
    
    now := time.Now()
    nowMs := now.UnixNano() / int64(time.Millisecond) // Current timestamp in ms
    windowMs := int64(windowSec) * 1000               // Window size in ms

    // Use a Redis Pipeline to bundle commands and minimize RTT
    pipe := rdb.Pipeline()

    // 1. Add current hit (Score = timestamp, Member = UUID)
    pipe.ZAdd(ctx, key, redis.Z{
        Score:  float64(nowMs),
        Member: uuid.New().String(),
    })

    // 2. Remove stale hits outside the sliding window
    cutoff := float64(nowMs - windowMs)
    pipe.ZRemRangeByScore(ctx, key, "-inf", fmt.Sprintf("%v", cutoff))

    // 3. Count elements remaining in the ZSET
    cardCmd := pipe.ZCard(ctx, key)

    // Execute pipeline
    _, err := pipe.Exec(ctx)
    if err != nil {
        return 0, err
    }

    return cardCmd.Val(), nil
}

Critical Details:

UUID as Member: If we used the timestamp as the member, concurrent hits occurring in the exact same millisecond would overwrite each other. Using a UUID ensures every hit is uniquely registered.
Pipeline vs. Multi/Exec: We use a pipeline for performance. Since these operations don't strictly require isolation from other parallel commands, we avoid the overhead of full Redis transactions.
ZRemRangeByScore: Without clearing historical data, the ZSET will grow indefinitely, eventually exhausting memory. Clearing expired members on every hit keeps memory usage bounded by the maximum potential hits in a single window.

3. Mitigating Alert Storms: Cooldown Locks

Sliding windows determine when to alert, but they don't prevent repetitive alerts when a fault persists. If a threshold of 10 hits in 60 seconds is met, every subsequent log entry will re-trigger the alert as long as the ZSET contains at least 10 elements.

To prevent this, we introduce a cooldown lock using a Redis key with a Time-To-Live (TTL):

go Copy

// IsAlertCooldown checks if the alerting rule is in its cooldown period
func IsAlertCooldown(ctx context.Context, ruleID int) bool {
    key := fmt.Sprintf("alert:rule:%d:cooldown", ruleID)
    exists, err := rdb.Exists(ctx, key).Result()
    if err != nil {
        return false // Fail-open: if Redis fails, proceed with the alert
    }
    return exists > 0
}

// SetAlertCooldown activates the cooldown lock for a specified duration
func SetAlertCooldown(ctx context.Context, ruleID int, duration time.Duration) error {
    key := fmt.Sprintf("alert:rule:%d:cooldown", ruleID)
    return rdb.Set(ctx, key, "locked", duration).Err()
}

💡 Design Pattern (Fail-Open): If Redis is unavailable, IsAlertCooldown returns false. It is better to receive duplicate alerts during a cache outage than to suppress a real incident.

4. Performance Optimization: Local Rule Caching

Alert evaluation happens in the hot path of the log processor. Performing database lookups for rules on every log line would easily saturate the database.

Instead, cache compiled rules in memory:

go Copy

type CachedRule struct {
    ID         int
    Name       string
    Pattern    string          // Regex pattern
    Regexp     *regexp.Regexp  // Compiled regex handle, reused across goroutines
    Source     string          
    WindowSec  int             
    Threshold  int             
}

type ruleCache struct {
    sync.RWMutex
    data map[uint64][]*CachedRule
}

var cache = &ruleCache{
    data: make(map[uint64][]*CachedRule),
}

Caching Compiled Regex: Recompiling patterns via regexp.Compile is CPU-heavy. Reusing thread-safe *regexp.Regexp pointers drastically reduces CPU load in high-throughput pipelines.

When rules are modified in your CRUD operations, invalidate the cache:

go Copy

func InvalidateCache(channelID uint64) {
    cache.Lock()
    defer cache.Unlock()
    delete(cache.data, channelID)
}

5. Automated AI Root Cause Analysis (RCA) with Gemini

Once an alert fires, engineers must manually inspect log lines, correlate timestamps, and diagnose the root cause. This step can be automated by querying the context window and sending it to Gemini.

5.1 Architecture Workflow

Copy

Alert Triggered ──► Write Incident Event (Report: empty)
     │
     └──► Asynchronous Goroutine (Independent Context)
               │
               ▼
         Sleep 1 Second (Allow bulk log writer to flush memory buffer)
               │
               ▼
         Fetch context logs (Last 20 entries)
               │
               ▼
         Request Gemini API (Temperature = 0.2)
               │
               ▼
         Backfill Markdown report into DB

5.2 Implementation

go Copy

func asyncAnalyzeAndBackfill(eventID int, channelID uint64, rule *CachedRule) {
    // 1. Wait 1 second to ensure recent logs in the bulk-write buffer
    //    have flushed and are queryable in the database.
    time.Sleep(1 * time.Second)

    // 2. Use context.Background() so the operation is independent 
    //    of the request context that triggered the alert.
    ctx := context.Background() 

    // 3. Fetch context logs surrounding the alert window
    startTime := time.Now().Add(-time.Duration(rule.WindowSec) * time.Second)
    triggerLogs, err := queryRecentLogs(ctx, channelID, startTime, 20)
    if err != nil {
        log.Error("AI Analysis: failed to fetch log context", "err", err)
        return
    }

    // 4. Generate report via Gemini API
    report, err := analyzeWithGemini(ctx, rule, triggerLogs)
    if err != nil {
        log.Error("AI Analysis: model generation failed", "err", err)
        return
    }

    // 5. Backfill the Markdown analysis to the incident record
    _ = backfillReport(ctx, eventID, report)
}

5.3 Optimizing Model Inputs: Why 20 Logs?

Accuracy: Excess context dilutes attention. Feeding the exact log lines surrounding the threshold breach guides the model to the primary error pattern.
Cost Efficiency: High-frequency alerting pipelines can generate significant API usage. Limiting inputs to 20 lines balances context quality and Token consumption.

5.4 Temperature Tuning

For code debugging or SRE work, set the model's Temperature parameter to 0.2 (low). This suppresses creative extrapolation ("hallucinations") and forces the model to focus strictly on factual log entries.

go Copy

modelInstance := client.GenerativeModel("gemini-2.0-flash")
var temp float32 = 0.2
modelInstance.Temperature = &temp

6. Symmetrical Execution Pipeline

Putting the components together:

go Copy

func checkAlert(ctx context.Context, channelID uint64, source, message string) {
    rules, err := getRulesFromCache(ctx, channelID)
    if err != nil {
        return
    }

    for _, r := range rules {
        if r.Source != "" && r.Source != source {
            continue
        }
        if r.Regexp == nil || !r.Regexp.MatchString(message) {
            continue
        }

        // Sliding window check
        count, err := SlideWindowCount(ctx, r.ID, r.WindowSec)
        if err != nil || count < int64(r.Threshold) {
            continue
        }

        // Cooldown check
        if IsAlertCooldown(ctx, r.ID) {
            continue
        }
        _ = SetAlertCooldown(ctx, r.ID, 3*time.Minute)

        // Write event and kick off AI diagnostic run asynchronously
        event := fireAlert(ctx, r, channelID, message, count)
        go asyncAnalyzeAndBackfill(event.ID, channelID, r)
    }
}

7. Summary

Component	Implementation	Purpose
Sliding Window	Redis ZSET (Score: timestamp, Member: UUID)	Avoids boundary errors, tracks hits precisely over rolling time
Rule Cache	`sync.RWMutex` + Compiled regex pointers	Bypasses database queries in the hot path
Cooldown Lock	Redis Key + TTL	Dampens duplicate alert noise
Fail-Open Policy	Redis errors return `false`	Prevents cache outages from swallowing critical alerts
AI RCA	Gemini API + Asynchronous worker	Generates Markdown diagnostics automatically on alert trigger

This architecture decouples filtering, rate-limiting, and analysis. Each layer acts independently, providing a robust SRE pipeline for high-throughput Go services.