• Target Keywords: AI Agent Memory System, Golang AI Agent, Gemini Embeddings, pgvector PostgreSQL, Semantic Search Go, GORM pgvector, Vector Database Go.
• Meta Description: A practical guide on building a resilient, long-term memory system for AI agents using Go, Gemini Embeddings, and pgvector. Learn how to implement semantic search, handle rate limits, and design for graceful degradation.

When people talk about giving AI agents "memory," their first instinct is often to dump all past conversation history directly into the LLM prompt. While this naive approach works for simple demos with a single agent, it quickly falls apart in multi-agent, long-running systems. The context window overflows, API costs skyrocket, and the model struggles to find relevant information amidst a mountain of irrelevant logs.

To solve this, we built a production-grade long-term memory system for our AI agent simulation platform. In this post, we’ll dive deep into our engineering architecture using Golang, Gemini Embeddings (gemini-embedding-001), and pgvector (PostgreSQL), and discuss the key engineering decisions we made along the way.

1. The Core Problem: Why Do Agents Need "Real Memory"?

Imagine this scenario in a virtual agent town:

At Tick 42, Agent Alice talks to Agent Bob. Bob mentions, "The market nearby is selling fresh fish." At Tick 200, Alice passes by the market. Does she remember what Bob said? Does she decide to buy fish based on that past interaction?

Without a dedicated memory system, she won't. When making decisions, Alice only receives observations from the current tick. Everything that happened at Tick 42 is lost to her.

This is the essence of the memory problem in multi-agent systems: agents must selectively retain and retrieve critical information over long-term timelines.

The word "selectively" is key. Not every event is worth remembering. If Alice decides to take one step to the right at Tick 100, storing that observation is useless noise. However, her conversation with Bob at Tick 42 is a high-value memory that must be preserved.

2. System Architecture: Decoupling Writes from Retrieval

To keep our execution loop highly responsive, we decoupled memory writes from vector retrieval:

End of Current Tick
        │
        ▼
┌───────────────────────┐
│ Synchronous Write     │
│ - Grade Importance    │
│ - Store Text to DB    │
│   (Skip Embeddings)   │
└──────────┬────────────┘
           │ Asynchronous Background Job
           ▼
┌───────────────────────┐
│ Vectorization Pipeline│
│ - Call Gemini API     │
│ - Backfill pgvector   │
└───────────────────────┘

Start of Next Tick
        │
        ▼
┌───────────────────────┐
│ Memory Retrieval      │
│ - Vectorize Query     │
│ - Cosine Distance Search│
│ - Inject into Prompt  │
└───────────────────────┘

This decoupled architecture is the most critical design decision we made to maintain high performance. Let's break down why and how we implemented each stage.

3. Designing the Memory Data Model

Let's look at how memory is structured in our database using GORM:

type MemoryModel struct {
    NPCID      uint            `gorm:"index;not null" json:"npc_id"`
    Type       string          `gorm:"size:32;not null" json:"type"`      // "observation" or "conversation"
    Content    string          `gorm:"type:text;not null" json:"content"` // Raw memory text
    Embedding  pgvector.Vector `gorm:"type:vector(1536)" json:"-"`        // 1536-dimensional vector
    Importance int             `json:"importance"`                        // Importance rating (1-10)
    Tick       int64           `json:"tick"`                              // World tick when generated
    IsCore     bool            `json:"is_core"`                           // Core memory flag
}

Why we rate memory Importance

Initially, we considered using an LLM to dynamically grade the importance of every event. While conceptually sound, this introduced significant downsides: extra LLM API calls (adding latency and costs) and severe rate-limiting pressures.

Instead, we switched to rule-based static weights:

Filtering out stay events is crucial. If we stored every "I decided to rest" event, our database would be flooded with low-value records, degrading semantic search quality. Maintaining a high signal-to-noise ratio in your vector database is far more important than storing everything.

4. Writing Memory: Text First, Vectorize Later

At the end of a world tick, the engine converts agent decisions into natural language memories and writes them to PostgreSQL:

func RecordNPCActionMemory(npc *models.NPCModel, decision *llm_ser.NPCDecision, tick int64) {
    if decision.Action == "talk" {
        memory := models.MemoryModel{
            NPCID:      npc.ID,
            Type:       "conversation",
            Content:    fmt.Sprintf("I said to everyone: %s", decision.Speak),
            Importance: 8,
            Tick:       tick,
        }
        // Omit the "Embedding" field—we do not vectorize synchronously
        global.DB.Omit("Embedding").Create(&memory)
    } else if decision.Action == "move" {
        memory := models.MemoryModel{
            NPCID:      npc.ID,
            Type:       "observation",
            Content:    fmt.Sprintf("I decided to move to (%d, %d)", decision.TargetX, decision.TargetY),
            Importance: 4,
            Tick:       tick,
        }
        global.DB.Omit("Embedding").Create(&memory)
    }
}

Notice the use of global.DB.Omit("Embedding").Create(&memory). We intentionally skip generating embeddings synchronously.

Calling the Gemini Embedding API over the network introduces latency (typically 200–500ms). If we waited for this call during the main tick loop, the entire simulation engine would stall. Instead, a background cron worker periodically queries records where embedding IS NULL, batch-vectorizes them, and updates the database.

5. Vectorization: Running Gemini Embeddings in Go

To generate vectors, we call the Google Gemini Embedding model (gemini-embedding-001) via an OpenAI-compatible client wrapper:

func GetEmbeddings(texts []string) ([][]float32, error) {
    if len(texts) == 0 {
        return nil, nil
    }

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    cfg := global.Config.LLM.GetConfigByName(global.Config.LLM.EmbeddingModel)
    client := llm.GetOpenAIClient(cfg.ApiKey, cfg.BaseURL)

    req := openai.EmbeddingRequest{
        Input: texts,
        Model: openai.EmbeddingModel("gemini-embedding-001"),
    }

    res, err := client.CreateEmbeddings(ctx, req)
    if err != nil {
        return nil, fmt.Errorf("batch embedding generation failed: %v", err)
    }

    var embeddings [][]float32
    for _, item := range res.Data {
        val := item.Embedding
        // Truncate to 1536 dimensions to match database schema
        if len(val) > 1536 {
            val = val[:1536]
        }
        embeddings = append(embeddings, val)
    }

    return embeddings, nil
}

Engineering Trade-offs & Techniques:

1. Dimension Alignment: The gemini-embedding-001 model can output dimensions higher than 1536. We define our database column as vector(1536) and truncate the array accordingly. 1536 dimensions provide the perfect balance between semantic precision and database query performance.
2. Batch Processing: The GetEmbeddings function takes a slice of strings ([]string). Our background job processes multiple memories in a single API call, reducing HTTP round-trip overhead and mitigating rate-limit exhaustion.
3. Text Chunking for Long Inputs: For long-form text, we implement a sliding window chunking algorithm using Go's rune type to prevent character corruption when cutting UTF-8 strings:

func ChunkText(text string, chunkSize int, overlap int) []string {
    runes := []rune(text)
    var chunks []string
    for i := 0; i < len(runes); {
        end := i + chunkSize
        if end > len(runes) {
            end = len(runes)
        }
        chunks = append(chunks, string(runes[i:end]))
        if end == len(runes) {
            break
        }
        i = end - overlap // Slide back to preserve context across chunks
    }
    return chunks
}

6. Memory Retrieval: Cosine Distance Search via pgvector

When an agent needs to make a decision, we take their current observation (their immediate sensory input), vectorize it, and search the pgvector database for the top 5 most semantically similar memories:

func RetrieveRelevantMemories(npcID uint, observation string) string {
    // 1. Vectorize the current observation query
    embeddings, err := GetEmbeddings([]string{observation})
    if err != nil || len(embeddings) == 0 {
        global.Log.Sugar().Errorf("Failed to generate embedding for query: %v", err)
        return "" // Graceful degradation: return empty if embedding fails
    }

    vector := pgvector.NewVector(embeddings[0])

    // 2. Perform Cosine Distance search via pgvector
    var memories []models.MemoryModel
    err = global.DB.Where("npc_id = ? AND embedding IS NOT NULL", npcID).
        Order(clause.OrderBy{
            Expression: clause.Expr{
                SQL:  "embedding <=> ?",
                Vars: []any{vector},
            },
        }).Limit(5).Find(&memories).Error

    if err != nil || len(memories) == 0 {
        return ""
    }

    // 3. Format results to inject into the LLM Prompt
    result := "\n[You recall the following relevant memories from the past]:\n"
    for _, m := range memories {
        result += fmt.Sprintf("- %s\n", m.Content)
    }

    return result
}

• The <=> Operator: This is pgvector’s operator for Cosine Distance (1 minus Cosine Similarity). A smaller value means the memory is more semantically relevant. By combining this query with an HNSW or IVFFlat index on the embedding column in PostgreSQL, we get sub-millisecond search times.
• The embedding IS NOT NULL Filter: This is critical. Freshly created memories that haven't been processed by the background vectorization worker have NULL embeddings. Filtering them out avoids query exceptions and ensures we only search valid vectors.

7. Resilience: Designing for Graceful Degradation

A key engineering lesson when working with external AI APIs is that everything will eventually fail. The embedding API might time out, the database connection might saturate, or you might hit rate limits.

Our memory retrieval system is designed with a fail-safe mechanism:

• If the embedding generation fails, we log the error but do not halt the simulation tick.
• RetrieveRelevantMemories simply returns an empty string. The agent's prompt compiles without historical memory and executes normally.
• If the background vectorization worker fails, it retries on the next run cycle without affecting the live simulation thread.

Memory should always be treated as a progressive enhancement to agent behavior, never as a single point of failure.

8. Summary & Key Takeaways

Building this memory system taught us that managing AI agent memory is, at its core, a classic engineering problem of data indexing, performance optimization, and fault tolerance:

1. Decouple CPU/IO-bound work: Generate embeddings asynchronously via background tasks to keep your simulation ticks fast and deterministic.
2. Prioritize Signal over Noise: Use smart rules or lightweight heuristics to filter out low-value events before saving them to your vector database.
3. Graceful Degradation is Mandatory: Never let a failed embedding API call bring down your entire application logic.

By choosing Go for performance, Gemini for embeddings, and pgvector for native relational/vector integration, we created a robust, easy-to-scale architecture that lets our agents reference past events naturally while keeping system latency minimal.

Are you building a memory system for your AI agents? Let us know your thoughts or any optimization tricks you use in the comments below!