Previously, I wrote an introductory article about Firebase Genkit for Go, discussing the basics of the AI framework launched by Google specifically for Go developers. However, getting knowledge from books is shallow; writing Hello World officially is completely different from building a real private butler system that runs in the background and is used heavily every day.

This note is a complete practical review of building my exclusive digital butler Iris from scratch, based on the Go language, Google Gemini Live native multimodal protocol, Genkit, PostgreSQL (pgvector), and Redis. This article avoids pretentious PR buzzwords; all code has been generalized and desensitized, striving to explain the core design, communication links, and bloody lessons learned from pitfalls in plain and simple terms.

0. Why I Don't Want an "Ordinary Chatbot"

Most open-source AI assistants encountered currently are still built around the classic "Web dialogue box" paradigm:
Receiving text input via HTTP or SSE, returning content like a streaming typewriter; if schedule reminders are included, they mostly rely on scheduled tasks that periodically scan database tables.

This classic interaction mode is extremely practical when writing long articles or looking up information, but if I want to use it as a "private butler" that resides permanently in the background and integrates into my daily desktop workflow, I will encounter several obvious user experience bottlenecks in interaction details:

1. "Walkie-talkie-style" Half-Duplex Experience:
   Normal human chatting is Full-Duplex—you can listen to me speak while nodding in response; when I am rambling, you can interrupt me at any time. Traditional Chatbots are typical "walkie-talkies": you press a button, finish a sentence, wait a few seconds, watch it spit out a long text, and usually cannot casually interrupt while it is speaking.
2. Lack of Visual Presence:
   When encountering errors while writing code, or wanting the AI to help me look at physical objects or blueprints on the desk, the traditional approach usually is: screenshot or take a photo -> save -> transfer file -> type description. This process is somewhat tedious and easily interrupts the current train of thought.
3. Memory as Short as a Fish (7 Seconds):
   Every time a new conversation starts, I have to re-explain my background habits. Once the context is reset, it is difficult for the AI to automatically continue the previous tacit understanding.
4. Passive Response Struggles to Fulfill a "Butler" Role:
   A true butler needs to possess real-time awareness and proactively walk over to remind you when the time comes; whereas ordinary dialogue assistants usually can only "wait passively for calls" and will take no action without proactive messaging.

To explore these interaction details closer to natural communication, I initiated Iris.

I. The Soul of Sound: Full-Duplex Intercom and Local Barge-In

1. Why Not Adopt the Three-Stage Pipeline of "Speech-to-Text -> Call Large Model -> Text-to-Speech"?

When building a voice assistant, many people intuitively build such a pipeline:

\text{Microphone Recording} \xrightarrow{1} \text{Whisper ASR} \xrightarrow{2} \text{LLM Text Generation} \xrightarrow{3} \text{TTS Synthesis} \xrightarrow{4} \text{Speaker Playback}

Every station in this chain accumulates network and computational latency:

• Audio recording chunking takes time (at least 500ms);
• ASR recognition takes time (300ms ~ 800ms);
• LLM generating the first batch of text takes time (500ms ~ 1000ms);
• TTS synthesizing full-sentence audio slices takes another time (500ms).

The whole process usually results in an end-to-end latency of 2 to 4 seconds. The walkie-talkie feel is extremely obvious, completely failing to form the "flow" of a real conversation.

Iris's Solution: Direct Connection to Gemini Live Pure Native Bidirectional Stream.
We let the browser microphone directly capture raw audio frames, continuously pouring them into the Go backend via WebSocket like a water flow, and the backend directly passes them through unmodified to the bidirectional stream channel of Gemini Live. The large model directly perceives sound amplitude and pitch, and directly spits out human voice audio slices in real-time inside the model (using the Aoede voice, complete with natural human breathing pauses and accents). End-to-end latency is directly reduced to the sub-hundred-millisecond level.

[Client]                                  [Go Backend]                          [Gemini Live API]
Mic Capture (16kHz PCM) ──(WebSocket)──>  Access Layer Hub ──(Bidi Streaming)──> Model Native Audio Understanding
Speaker Queue (24kHz PCM) <──(WebSocket)── Dispatch Layer  <──(Audio Chunks)──── Model Native Voice Output

2. Core Code: 0ms Local Instant Barge-In

In voice interaction, Barge-In is the deciding factor for quality.

If the interruption logic is placed on the server side (waiting for the server-side VAD to detect your speech, then sending a command to notify the browser to stop playback), a round-trip network time (RTT) takes a hundred milliseconds or so. The actual experience is: You have already shouted "Stop", and the AI in your headphones stubbornly speaks another half sentence, which is a terrible experience.

My approach is: Deploy the first line of defense for interruption directly at the client microphone capture layer.

When the client captures each audio frame, it locally calculates the Root Mean Square (RMS) of the voice energy. As long as the AI is currently speaking in the speaker, and the user's microphone volume suddenly exceeds the set energy threshold, the frontend instantly cuts off local audio playback in 0 milliseconds, while asynchronously sending an interruption beacon upstream:

// Client Web Audio API real-time audio processing node
micProcessor.onaudioprocess = (e) => {
    const inputData = e.inputBuffer.getChannelData(0);

    // 1. Locally calculate the Root Mean Square (RMS) of audio energy
    let sumSquares = 0;
    for (let i = 0; i < inputData.length; i++) {
        sumSquares += inputData[i] * inputData[i];
    }
    const rms = Math.sqrt(sumSquares / inputData.length);

    // 2. Core interruption logic: If the AI is speaking in the headphones and the microphone detects the user suddenly starting to speak (exceeding threshold 0.025)
    if (rms > 0.025 && activeAudioSources.length > 0) {
        // Instantly clear all queued AudioBuffer playback sources in the browser (0ms local mute)
        stopAllLocalAudio();
        
        // Asynchronously send an interruption signal to the upstream Go backend, telling the large model to truncate this round of output immediately
        ws.send(JSON.stringify({ type: "interrupt" }));
    }

    // 3. Convert 32-bit floating-point audio into standard 16kHz 16-bit linear PCM
    const pcm16 = new Int16Array(inputData.length);
    for (let i = 0; i < inputData.length; i++) {
        const s = Math.max(-1, Math.min(1, inputData[i]));
        pcm16[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
    }

    // 4. Base64 encode the audio slice and continuously push it via WebSocket
    ws.send(JSON.stringify({
        type: "audio_chunk",
        data: { pcm: arrayBufferToBase64(pcm16.buffer) }
    }));
};

Layman's Explanation: This is like listening to someone speak with headphones on; once your own vocal cords vibrate, your brain immediately and proactively cuts off the sound in the headphones, rather than waiting for the other person to hear you, react, and stop.

II. The Secret of Vision: Why Frame Extraction Instead of Video Streaming?

When I first wanted to add "vision" to Iris, my first thought was to push an H.264 video stream or pull WebRTC.

However, after hitting pitfalls, I knew this is an absolute anti-pattern in current AI interaction:

• Video encoding heavily consumes the CPU/GPU computing power of mobile devices and computers;
• The underlying large model does not watch dynamic video frame by frame; it disassembles images into two-dimensional Patches (visual slices) and understands them as special image Tokens.

Industrial-Grade Elegant Solution: Continuous High-Frequency Image Frame Stream (1 FPS JPEG).

We don't need to push 60 frames like playing a game. For the large model to understand code on the screen or objects on the desk, capturing 1 image per second (1 FPS) is enough for the model to capture all dynamic changes, and the network overhead is so low it can be ignored:

javascript
// Frontend captures a frame from screen sharing or camera every second (1000ms)
setInterval(() => {
if (!videoElement.videoWidth) return;

// Dynamically draw to a hidden Canvas, with dimensions proportionally scaled to 720p (1280x720)
canvas.width = Math.min(1280, videoElement.videoWidth);
canvas.height = Math.round(canvas.width * (videoElement.videoHeight / videoElement.videoWidth));
ctx.drawImage(videoElement, 0, 0, canvas.width, canvas.height);

// Compress to a lightweight JPEG image with 0.65 quality
const dataUrl = canvas.toDataURL("image/jpeg", 0.65);
const b64Jpeg = dataUrl.split(",")[1];

// Send to Go backend
ws.send(JSON.stringify({
    type: "screen_frame", // or video_frame (camera)
    data: { data: b64Jpeg, ts: Date.now() }
}));

}, 1000);

On the Go server side, we just need to feed this JPEG byte stream directly to the multimodal video frame channel of the Live API:

```go
// internal/genkit/live.go (generalized illustration)
func (ls *LiveSession) SendImage(jpegBytes []byte) error {
    return ls.session.SendRealtimeInput(genai.LiveSendRealtimeInputParameters{
        Video: &genai.Blob{
            MIMEType: "image/jpeg",
            Data:     jpegBytes,
        },
    })
}

Layman's Effect: You share your IDE while speaking: "Iris, take a look at what the error on line 50 means." In the same long connection, the large model listens to your PCM voice with its left ear while watching the 720p screenshots you send every second with its right eye, instantly pointing out that line 50 of your code is missing a parenthesis.

III. The Intelligent Center: Genkit Go Tool System and Domain Decomposition

During initial development, I crammed all tool functions (check time, query database, search Baidu, take notes) into a giant tools.go file. As a result, later in the project, a single file had nearly two thousand lines, various modules intertwined with each other, and maintenance became extremely painful.

Later, I performed Domain Splitting, and the overall architecture became exceptionally clean:

internal/genkit/
├── genai.go         # Genkit runtime unified initialization (G instance and global tools mounting)
├── live.go          # Gemini Live bidirectional PCM real-time communication session management
├── memory.go        # pgvector 1536-dimensional vector retrieval and persistence encapsulation
├── tool_system.go   # Domain 1: Physical system clock (provides time physical anchor)
├── tool_user.go     # Domain 2: User state and Casbin permissions
├── tool_memory.go   # Domain 3: Read and write tools for long-term memory
├── tool_reminder.go # Domain 4: CRUD for schedule reminders
├── tool_search.go   # Domain 5: Full-network DuckDuckGo dual-channel search
└── tools.go         # Unified registration center (registerTools)

Paradigmatic Code for Writing Tools (Taking the System Reference Clock as an Example)

The easiest mistake for a large model to make is "forgetting what year and month it is." If a user asks "remind me to have a meeting tomorrow at 3 PM," without a real-time clock reference, the model can only make things up. Therefore, providing an authoritative system clock is the first step:

// internal/genkit/tool_system.go
package genkit

import (
    "fmt"
    "time"
    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
)

// timeTool gets the backend server's current precise time and timezone
func timeTool() ai.ToolRef {
    return genkit.DefineTool(G, "getServerTime", "Get the backend server's current precise time and timezone; this tool must be called first when calculating relative time",
        func(ctx *ai.ToolContext, _ struct{}) (string, error) {
            nowStr := time.Now().Format("2006-01-02 15:04:05 MST")
            return fmt.Sprintf("The current absolute server time is: %s", nowStr), nil
        },
    )
}

In tools.go, append and register all sub-tools at once:

// internal/genkit/tools.go
package genkit

func registerTools() {
    Tools = append(Tools,
        timeTool(),          // System clock
        saveMemoryTool(),    // Memory insertion
        searchMemoryTool(),  // Memory retrieval
        setReminderTool(),   // Countdown reminder setting
        listRemindersTool(), // To-do list query
        searchWebTool(),     // Full-network real-time search
    )
}

Through the reflection function buildLiveTools(), this tool set defined by Genkit standards can be automatically converted into the FunctionDeclaration required by Gemini Live. Whether it is voice intercom or subsequent background Flows, write the tool once and reuse it globally anywhere.

IV. The Foundation of Memory: pgvector Semantic Retrieval and Native Pitfall Avoidance

A human butler won't let you repeat your background life story a second time. To make Iris understand me better the more it is used, it must have a high-precision long-term memory system.

1. Why Not Use Traditional Relational Database Fuzzy Queries (LIKE)?

If you ask the AI: "What is the lowest price President Li can accept?"
If the sentence stored in the database is: "President Li stated during negotiations that an 85% discount is his limit concession".
Traditional SQL WHERE content LIKE '%lowest price%' has a hit rate of 0, because the words "lowest price" do not exist in the text at all.

Vector Embeddings must be used for semantic retrieval.

2. Vector Ingestion and pgvector Pitfall Practice

Iris adopts the official PostgreSQL extension pgvector, paired with Google's latest gemini-embedding-2 (1536-dimensional feature space).

When executing vector cosine distance queries in Go, many online codes directly write embedding <=> $1, which triggers a PostgreSQL underlying error operator does not exist: vector <=> text. Explicit type casting must be applied to incoming parameters:

// internal/genkit/memory.go (generalized and core logic)
package genkit

import (
    "context"
    "fmt"
    "github.com/pgvector/pgvector-go"
)

// SearchMemory retrieves historically related facts and preferences from PostgreSQL based on semantic vectors
func SearchMemory(ctx context.Context, userID int, query string, limit int) ([]*MemorySearchResult, error) {
    // 1. Call the Embedding model to convert the user's question into a 1536-dimensional floating-point vector
    queryVector, err := GetEmbedding(ctx, query)
    if err != nil {
        return nil, fmt.Errorf("failed to generate retrieval vector: %w", err)
    }

    // 2. Core SQL: Use 1 - (embedding <=> $1::vector) to calculate standard cosine similarity (0.0 ~ 1.0)
    // <=> is pgvector's cosine distance operator; smaller distance means higher similarity; hence 1 minus distance gives the similarity score
    sqlQuery := `
        SELECT id, title, content, category, importance,
               1 - (embedding <=> $1::vector) AS similarity,
               created_at
        FROM memories
        WHERE user_id = $2
        ORDER BY similarity DESC
        LIMIT $3;
    `

    rows, err := dbPool.Query(ctx, sqlQuery, pgvector.NewVector(queryVector), userID, limit)
    if err != nil {
        return nil, err
    }
    defer rows.Close()

    var results []*MemorySearchResult
    for rows.Next() {
        var m MemorySearchResult
        if err := rows.Scan(&m.ID, &m.Title, &m.Content, &m.Category, &m.Importance, &m.Similarity, &m.CreatedAt); err != nil {
            continue
        }
        results = append(results, &m)
    }

    return results, nil
}

Layman's Explanation: Vectorization is like projecting all human language into a 1536-dimensional three-dimensional universe. Although "limit concession" and "lowest price" have different literal spellings, their coordinates in the universe are almost next to each other. By calculating the angle distance between two coordinates, the AI can instantly dig out details you once said from tens of thousands of historical records.

V. The Proactive Butler: Saying Goodbye to Infinite Polling Loops with Redis TTL Event-Driven Architecture

This is a design solution in the Iris system that I consider relatively lightweight and effective.

1. Hard Flaws of Traditional Alarm Polling

How do many AI assistants implement "remind me to drink water in 10 minutes"?
The backend writes an infinite loop:

// Traditional inefficient approach: scan the database every minute
for range time.Tick(1 * time.Minute) {
    db.Query("SELECT * FROM reminders WHERE remind_at <= NOW() AND is_sent = false")
}

This approach not only adds pointless I/O burdens to the database every minute, but also carries a sluggish error margin of up to 60 seconds for expiration times, completely lacking a tech feel.

2. My Design: Redis TTL Key-Expired Event-Driven Architecture

Iris completely abandoned polling, adopting an Event-Driven precise countdown architecture:

User says: "Remind me to drink water in 10 minutes"
   │
   ├──> 1. DB writes persistent record (for backup, is_sent = false)
   ├──> 2. Redis writes metadata snapshot Hash (reminder:data:ID)
   └──> 3. Redis writes an empty key with a precise TTL:
           SETEX reminder:timer:42 600 "1"
           
           [The entire system then enters a complete sleep state with 0 CPU and 0 database I/O]
           
Redis timer is precise to the exact 600th second, and the Key instantly evaporates!
   │
   └──> Redis internal PubSub automatically broadcasts event: __keyevent@*__:expired
           │
           └──> A single event listening goroutine in the Go backend instantly captures the event!
                   │
                   ├── Read metadata from Redis Hash at millisecond level
                   ├── Update DB status is_sent = true
                   ├── Push pop-up notification to frontend via WebSocket
                   └── Core strike: Wake up the online Gemini Live voice channel!

3. Core Implementation Code

① Countdown Registration (AddReminder)

// internal/reminder/scheduler.go (core snippet)
func AddReminder(ctx context.Context, userID int, content string, remindAt time.Time) error {
    ttl := time.Until(remindAt)
    if ttl <= 0 {
        return fmt.Errorf("reminder time must be in the future")
    }

    // 1. Database persistence
    rem, _ := db.Client.Reminder.Create().
        SetUserID(userID).
        SetContent(content).
        SetRemindAt(remindAt).
        Save(ctx)

    // 2. Redis Hash stores metadata snapshot (for millisecond direct read upon expiration, no DB query needed)
    payload, _ := json.Marshal(map[string]any{
        "id": rem.ID, "user_id": userID, "content": content,
    })
    _ = db.Redis.HSet(ctx, "reminder:data", rem.ID, payload).Err()

    // 3. Set precise Redis TTL countdown key (automatically disappears upon expiration)
    timerKey := fmt.Sprintf("reminder:timer:%d", rem.ID)
    _ = db.Redis.Set(ctx, timerKey, "1", ttl).Err()

    return nil
}

② Listening for Redis Expiration Events (StartReminderListener)

// Start listening for Redis keyspace expiration notifications
func StartReminderListener() {
    ctx := context.Background()
    // Listen to key expiration events across all databases
    pubsub := db.Redis.PSubscribe(ctx, "__keyevent@*__:expired")
    
    for msg := range pubsub.Channel() {
        // Match prefix format: reminder:timer:42
        if strings.HasPrefix(msg.Payload, "reminder:timer:") {
            idStr := strings.TrimPrefix(msg.Payload, "reminder:timer:")
            reminderID, _ := strconv.Atoi(idStr)
            
            // Asynchronously trigger processing without blocking the listening channel
            go handleReminderTrigger(reminderID)
        }
    }
}

③ Ultimate Quality: Driving Gemini Live to Proactively "Speak Out"

When a reminder expires, if it only pops up a window on the phone or rings a "ding", it is still just an alarm clock.
Iris's approach is: Proactively wake up the currently online Gemini Live session to speak by injecting system events with TurnComplete: true!

// internal/genkit/live.go
func (ls *LiveSession) NotifyReminder(content string) error {
    // Build system prompt for verbal reporting
    prompt := fmt.Sprintf("[System Event: Scheduled Reminder Time Has Reached] The reminder item '%s' set by the user has now expired. As the intelligent butler Iris, please immediately and proactively broadcast this reminder to the user in a calm and natural Mandarin Chinese tone, stating the reminder content directly. Keep the tone natural, kind, and concise, without explaining that this is a system notification.", content)

    // Send ClientContent and force TurnComplete = true
    return ls.session.SendClientContent(genai.LiveClientContentInput{
        Turns: []*genai.Content{
            {
                Role: "user",
                Parts: []*genai.Part{{Text: prompt}},
            },
        },
        TurnComplete: new(true), // Key: Tell the model this round of input is complete, the model will immediately start reasoning and speak directly!
    })
}

Actual Effect: You are writing code with headphones on. When the 10-minute mark arrives, without touching any keyboard or screen, Iris directly and calmly speaks in your headphones: "Sir, the hydration time you set has arrived."

VI. Engineering Architecture Pitfalls and Trade-off Philosophy

During the process of kneading various modules together, I gained two very profound architectural insights:

1. The "Callback Decoupling Method" to Avoid Go Import Cycles

When implementing proactive reminders, I encountered an extremely disgusting package circular dependency problem:

• The reminder package needs to call the ws package to push streams after expiration;
• The ws package manages active clients and internally needs to call the genkit package to drive voice broadcasting;
• Tools inside the genkit package (tool_reminder.go) need to import the reminder package to create new reminders.

The Go compiler turns hostile on the spot:
imports reminder -> imports ws -> imports genkit -> imports reminder: import cycle not allowed

Many beginners encountering this problem like to compromise by building a messy global common package or stacking all code together.

The best solution is: The lower-level package defines function signatures, and the upper-level package injects callbacks unilaterally upon startup (Notifier Pattern):

// 1. In the underlying internal/reminder/scheduler.go, only define the callback type and global variables
type NotifierFunc func(userID uint, reminderID int, content, remindAt string)
var Notifier NotifierFunc

func SetNotifier(fn NotifierFunc) {
    Notifier = fn
}

// Upon expiration, just trigger the callback without importing any ws-related code at all
if Notifier != nil {
    Notifier(uint(payload.UserID), payload.ID, payload.Content, payload.RemindAt)
}

When launching the upper-layer internal/ws/hub.go, the ws package is responsible for mounting its own push actions:

// internal/ws/hub.go
func (m *Manager) Start() {
    // Stuff the upper-layer actual business logic into the lower layer as a callback function
    reminder.SetNotifier(func(userID uint, reminderID int, content, remindAt string) {
        // Execute WebSocket push and Gemini Live voice proactive calling...
    })
    // ...
}

The lower layer does not know who the upper layer is, and the upper layer is responsible for assembling logic. Dependency loops are instantly resolved, and each package maintains a high degree of autonomy.

2. Rejecting Over-Engineering (Adhering to the YAGNI Principle)

While refining the expiration trigger deduplication logic, I once thought about: Should I use Redis distributed exclusive locks? Should I implement strict atomic state contention in PostgreSQL (UPDATE ... WHERE is_sent = false)?

Sitting down late at night to think: What I am building is my personal digital assistant, running as a single instance on my local server.
Stand-alone Redis Key-Expired events naturally arrive as single events. As long as Where(reminder.IsSent(false)) protection is added to the database query fallback, the system is already impregnable.

If I pursued seemingly "high-end" distributed strong consistency, adding several database row locks and network communications on top of the originally microsecond-level memory pipeline would both slow down interaction response and increase troubleshooting complexity.

"Don't pay for imagined millions of concurrent requests (YAGNI - You Aren't Gonna Need It)"; when designing personal systems, always put code readability, rapid response, and architectural simplicity first.

VII. Current Status and TODOs

Fiddling around until now, this system runs reasonably stably on my own machine. Full-duplex voice intercom, screen sharing to view code, memory retrieval, and proactive voice reminders upon expiration core requirements are basically smooth. Hanging it in the background daily indeed helps me save a lot of trouble switching windows and typing.

Of course, the system still has plenty of rough edges left for gradual refinement:

1. Long-Connection Reconnection Fault Tolerance: When hung in the background for a long time, WebSocket fluctuations occasionally happen. The current automatic reconnection is not smooth enough, sometimes requiring manual page refreshes;
2. Hands-Free Operation (Wake Word): Intercom currently requires clicking a button on the webpage; plans are underway to add a local lightweight wake-word detector on the frontend to respond instantly to calls;
3. Local System Control: Currently it can only "see" and "speak"; next, I plan to consider giving it some capabilities to directly control the computer via restricted local scripts (such as querying local system load, opening designated projects, controlling music playback, etc.).

Coding is like this: first get the core trunk pipeline running, then gradually fill in the remaining edge experiences step by step.