Target Audience: Backend engineers with a solid foundation in Go who want to get started with large language model (LLM) application development but don't know where to begin
Reading Time: About 20 minutes
Official Repository: github.com/firebase/genkit/go

I recently integrated an LLM into a Go backend project, and after hitting quite a few roadblocks, I finally figured out a smooth approach. This article summarizes the entire process and mindset to share with friends struggling with the same things.

Table of Contents

1. Introduction: Why is it so hard for Go developers to integrate AI?
2. What is Genkit, and what specific problems does it solve?
3. Ecosystem Overview: Beyond Gemini, what models and storage are supported?
4. Four Core Concepts: Once you understand them, you know how to use it
5. Writing Your First Complete Example (Including Tool Calling)
6. Dev UI: Debugging LLM applications without guessing in the dark
7. Dotprompt: Freeing prompts from your codebase
8. Tips and Precautions for Integrating into Existing Go Projects
9. Conclusion

1. Introduction: Why is it so hard for Go developers to integrate AI?

Anyone who has done this knows that the simplest "calling an LLM API" isn't actually difficult—dozens of lines of HTTP code can get it done. The hard parts are the following:

First, Tool Calling is extremely tedious. When you want the LLM to call functions you wrote yourself—like querying a database or calling a third-party API—you have to manually describe your function's input parameters as a JSON Schema and pass it to the model. Then, you parse the invocation instructions returned by the model, execute the local function, and stitch the result back into the messages to request the model again. The entire process is completely manual, lengthy, and prone to bugs.

Second, multi-turn conversation state management is messy. As users chat back and forth with the AI, you have to maintain the message history yourself. If you aren't careful, you'll exceed the model's context window, or the context order will get messed up.

Third, it's a complete black box, making debugging extremely difficult. When the model gives a wrong answer, you don't know if the prompt was written poorly, if the tool returned incorrect data, or if the context was truncated. A massive blob of JSON in the logs is completely unreadable.

Fourth, using Gemini today and switching to DeepSeek tomorrow requires major code overhauls. Without a unified abstraction, changing models means rewriting your code.

Genkit was created to systematically solve these problems.

2. What is Genkit, and what specific problems does it solve?

Google Genkit is an open-source framework for developing LLM applications, with official support for both Go and TypeScript. Its core philosophy is: Establish a structured intermediate layer between your business code and the LLMs, so you don't have to deal with repetitive and error-prone engineering problems from scratch every time.

In one sentence: Genkit is an LLM application scaffolding in the Go ecosystem, not just an SDK wrapper.

What problems does it specifically solve:

3. Ecosystem Overview: Beyond Gemini, what models and storage are supported?

Many people see "Google Genkit" and assume it only works with Google's own models. This is a common misconception. Genkit's plugin architecture is completely open, supporting a wide range of mainstream models and vector stores.

3.1 Model Support

Official Built-in Plugins:

• plugins/googlegenai — Integrates with Google AI Studio and Vertex AI, supporting the entire Gemini family (including Gemini 3.8 Flash, Gemini 3.1 Pro) and Imagen image generation models.
• plugins/ollama — Integrates with local Ollama, supporting open-source models like DeepSeek-R1/V3, Qwen2.5, Llama 3, and Gemma 3, keeping data entirely local.
• plugins/compat_oai — OpenAI protocol compatibility layer. Any service implementing OpenAI's API standard can be integrated, including GPT-4o, o3, and private proxy gateways built with vLLM, OneAPI, or LiteLLM.

Community Plugins (such as BloomLabsInc/genkit-plugins):

• Anthropic Claude (3.5 / 3.7 / Sonnet)
• Groq (ultra-fast inference, supports Llama, Mixtral)
• Mistral
• Azure OpenAI

Switching models is completely transparent to the upper-layer code; changing models only requires updating the initialization:

// Using Gemini 3.8 Flash
g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithDefaultModel("googleai/gemini-3.8-flash"),
)

// Switch to local DeepSeek, keeping the rest of the code completely unchanged
g := genkit.Init(ctx,
    genkit.WithPlugins(&ollama.Plugin{ServerAddress: "http://localhost:11434"}),
    genkit.WithDefaultModel("ollama/deepseek-r1:latest"),
)

// Switch to OpenAI
g := genkit.Init(ctx,
    genkit.WithPlugins(&compat_oai.Plugin{
        APIKey:  os.Getenv("OPENAI_API_KEY"),
        BaseURL: "https://api.openai.com/v1",
    }),
    genkit.WithDefaultModel("openai/gpt-4o"),
)

3.2 Vector Storage (for RAG / Knowledge Base Retrieval)

Genkit defines unified Indexer (ingestion) and Retriever (retrieval) interfaces, supporting:

• pgvector (PostgreSQL vector extension, the top choice for Go projects)
• Pinecone (cloud-native vector database)
• Milvus / Weaviate / Qdrant / Chroma
• In-memory retriever (for local development and unit testing with zero external dependencies)

3.3 Quick Reference for Current Mainstream Models

4. Four Core Concepts: Once you understand them, you know how to use it

Genkit has a relatively gentle learning curve. Mastering these four concepts allows you to handle the vast majority of use cases.

4.1 Plugin — The Model Integration Mechanism

Plugins are responsible for bringing in specific LLMs or vector stores. You register plugins during initialization, and thereafter use unified interfaces everywhere else, without needing to know which underlying model is being used.

g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
)

4.2 Tool — Giving Models Action Capabilities

This is the most interesting part of Genkit. LLMs can naturally only generate text; they don't know "what date it is today" nor can they query your database. Tools allow you to expose arbitrary Go functions to the model, enabling the model to autonomously decide during inference: "I need to call this function to answer the user's question."

// Define a Tool: Query order status in the database
orderTool := genkit.DefineTool(
    g,
    "getOrderStatus",              // Tool name, used by the model to identify it
    "Query the current status of a specified order number",      // Tool description, used by the model to determine when to call it
    func(ctx *ai.ToolContext, input struct {
        OrderID string `json:"order_id"`
    }) (string, error) {
        // Write actual database query logic here
        status := db.QueryOrderStatus(input.OrderID)
        return status, nil
    },
)

Note: The model automatically completes the entire loop of "deciding which tool to call → framework executes the Go function → returns the result to the model → model generates a response based on the result." You do not need to write any polling or coordination code.

4.3 Flow — Testable and Traceable Execution Units

Flows encapsulate your AI business logic. They act like functions with type signatures, support Trace tracking, and can be tested individually in the Dev UI.

chatFlow := genkit.DefineFlow(
    g,
    "chatFlow",
    func(ctx context.Context, userMessage string) (string, error) {
        return genkit.GenerateText(ctx, g,
            ai.WithPrompt(userMessage),
            ai.WithTools(orderTool),
        )
    },
)

// Invoke
result, err := chatFlow.Run(ctx, "Help me check the status of order ORD-12345")

4.4 Generate — Triggering Model Inference

This is where model inference is actually triggered. Here you can combine prompts, specify models, attach tools, and set output formats.

// Generate plain text
text, err := genkit.GenerateText(ctx, g,
    ai.WithPrompt("Introduce the Go language in one sentence"),
)

// Generate structured data (model output will be strictly validated and deserialized)
type ProductInfo struct {
    Name  string  `json:"name"`
    Price float64 `json:"price"`
}
resp, err := genkit.GenerateData[ProductInfo](ctx, g,
    ai.WithPrompt("Describe a laptop, returning its name and price"),
)
fmt.Println(resp.Output.Name, resp.Output.Price)

5. Writing Your First Complete Example (Including Tool Calling)

Without further ado, let's jump straight into a runnable, complete example. This example contains two tools: getting the current time and querying city weather (mocked), clearly demonstrating the complete workflow of Tool Calling.

Step 1: Initialize the Project

mkdir my-genkit-app && cd my-genkit-app
go mod init my-genkit-app
go get github.com/firebase/genkit/go@latest
go get github.com/firebase/genkit/go/plugins/googlegenai@latest

Get your API Key (free): https://aistudio.google.com/

export GEMINI_API_KEY="Your API Key"

Step 2: Write the Code

Create main.go:

package main

import (
	"context"
	"fmt"
	"log"
	"strings"
	"time"

	"github.com/firebase/genkit/go/ai"
	"github.com/firebase/genkit/go/genkit"
	"github.com/firebase/genkit/go/plugins/googlegenai"
)

// Tool input type; Genkit automatically converts structs to JSON Schema for the model
type WeatherInput struct {
	City string `json:"city" description:"City name to query, e.g., Beijing, Shenzhen"`
}

type WeatherResult struct {
	City        string `json:"city"`
	Temperature string `json:"temperature"`
	Description string `json:"description"`
}

func main() {
	ctx := context.Background()

	// Initialize Genkit, with plugins handling Google Gemini integration
	// The GEMINI_API_KEY environment variable will be automatically read
	g := genkit.Init(ctx,
		genkit.WithPlugins(&googlegenai.GoogleAI{}),
		genkit.WithDefaultModel("googleai/gemini-3.8-flash"),
	)

	// Tool 1: Get current system time
	// LLMs do not have a real-time clock; this tool allows it to answer questions like "What time is it?"
	timeTool := genkit.DefineTool(g,
		"getCurrentTime",
		"Get the current precise time of the server",
		func(ctx *ai.ToolContext, _ struct{}) (string, error) {
			return time.Now().Format("2006-01-02 15:04:05 MST"), nil
		},
	)

	// Tool 2: Query city weather (using mock data here; replace with real APIs or databases in actual projects)
	weatherTool := genkit.DefineTool(g,
		"getCityWeather",
		"Query real-time weather conditions for a specified city",
		func(ctx *ai.ToolContext, input WeatherInput) (*WeatherResult, error) {
			// Mock weather data, replace with actual API calls in reality
			data := map[string]WeatherResult{
				"Beijing": {City: "Beijing", Temperature: "28°C", Description: "Sunny, light breeze"},
				"Shanghai": {City: "Shanghai", Temperature: "23°C", Description: "Cloudy to light rain"},
				"Shenzhen": {City: "Shenzhen", Temperature: "32°C", Description: "Hot, thundershowers"},
			}
			for key, result := range data {
				if strings.Contains(input.City, key) {
					r := result
					return &r, nil
				}
			}
			return &WeatherResult{
				City:        input.City,
				Temperature: "Unknown",
				Description: "No data for this city",
			}, nil
		},
	)

	// Define a Flow and attach the tools
	// Flows appear in the Dev UI and can be tested directly in the browser
	assistantFlow := genkit.DefineFlow(g,
		"assistantFlow",
		func(ctx context.Context, question string) (string, error) {
			return genkit.GenerateText(ctx, g,
				ai.WithPrompt(question),
				ai.WithTools(timeTool, weatherTool),
			)
		},
	)

	// Test several questions to observe how the model autonomously decides which tool to call
	questions := []string{
		"What time is it?",
		"What's the weather like in Beijing today? Do I need to bring an umbrella?",
		"Which city is hotter today, Shanghai or Shenzhen?",
	}

	for _, q := range questions {
		fmt.Printf("\n🙋 Question: %s\n", q)
		answer, err := assistantFlow.Run(ctx, q)
		if err != nil {
			log.Printf("Error: %v", err)
			continue
		}
		fmt.Printf("🤖 Answer: %s\n", answer)
		fmt.Println("────────────────────────────────────────")
	}
}

Step 3: Run

go run main.go

You will observe that:

• Asking "What time is it?" → The model calls getCurrentTime and answers with the real returned time.
• Asking about Beijing's weather → The model extracts the city parameter, calls getCityWeather, and gives travel advice based on the weather data.
• Asking which is hotter between Shanghai and Shenzhen → The model calls getCityWeather twice in succession, compares the data, and draws a conclusion.

Throughout the process, you did not write any coordination logic; the Tool Calling loop is entirely handled by the framework.

6. Dev UI: Debugging LLM applications without guessing in the dark

This is the feature that surprised me the most about Genkit.

A common pain point in developing LLM applications is that you never quite know what the model "saw," whether the tool was called, what parameters were passed during the call, or what was returned. When things go wrong, you are left guessing.

Genkit's Dev UI directly visualizes this information.

How to Launch

First, install the CLI globally (only needs to be done once):

npm install -g genkit-cli

Then launch your Go service using Genkit:

genkit start -- go run main.go

Your browser will automatically open http://localhost:4000.

What You Can Do in the Dev UI

Test Flows directly in the browser: Find assistantFlow, input any question, and click run—no Postman or test code required.

View complete call chain Traces: Every run generates a Trace record displayed as a waterfall chart. You can precisely see:

• Step 1: What request the model received (complete messages JSON)
• Step 2: Which tool the model decided to call and what parameters were passed
• Step 3: What data the tool returned
• Step 4: How the model generated the final reply based on the tool's results
• Millisecond-precision latency for every step.

Token consumption statistics: Input tokens, output tokens, and cache hits for every call, helping you evaluate costs and optimize prompts.

7. Dotprompt: Freeing prompts from your codebase

As project complexity increases, you will increasingly dislike hardcoding prompts into Go strings—changing phrasing requires recompilation, and prompt modifications get mixed up with code logic during team collaboration, making them hard to manage.

Genkit provides Dotprompt, allowing prompts to exist as independent files.

Format Example

Create a file named prompts/support.prompt:

---
model: googleai/gemini-3.8-flash
config:
  temperature: 0.5
  maxOutputTokens: 1024
input:
  schema:
    userName: string
    issue: string
---
You are a professional customer service assistant who is patient and concise.

User Name: {{userName}}
User Feedback Issue: {{issue}}

Please analyze potential causes of the issue and provide 2-3 solutions, with each solution explaining the steps in one sentence.

Loading and Executing in Go Code

// Specify the prompt file directory when initializing Genkit
g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithPromptDir("./prompts"),
)

// Load and execute without recompiling Go code
prompt := genkit.LookupPrompt(g, "support")
resp, err := prompt.Execute(ctx, map[string]any{
    "userName": "John Doe",
    "issue":    "App prompts network error during login, but my network is fine",
})
fmt.Println(resp.Text())

The practical benefits of this are real: Product managers or algorithm engineers can directly modify .prompt files without knowing Go or changing business code; they can just submit a PR to release.

8. Tips and Precautions for Integrating into Existing Go Projects

8.1 Where to Place Initialization

The *genkit.Genkit instance returned by genkit.Init should be initialized once when the service starts, and then reused across various handlers. You can store it in a package-level variable, a global configuration struct, or pass it via dependency injection, depending on your project's existing style.

// Example: Initialize in a standalone ai package
package genai

import (
    "context"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/googlegenai"
)

var G *genkit.Genkit

func Init(ctx context.Context, apiKey, model string) {
    G = genkit.Init(ctx,
        genkit.WithPlugins(&googlegenai.GoogleAI{APIKey: apiKey}),
        genkit.WithDefaultModel("googleai/" + model),
    )
    registerTools() // Register all tools uniformly here
}

8.2 Tool Security Boundaries

Pay special attention to permission controls for tools provided to LLMs:

• Read-only query operations: Database tools should only permit SELECT statements; never expose DELETE/UPDATE directly to the model.
• Limit result sizes: Add limits to query results to prevent large amounts of data from filling up the context window and wasting tokens.
• High-risk operations require manual confirmation: Operations involving money, sending messages, or modifying critical configurations should have the model return an "operation plan" to be confirmed by the user before execution, rather than executing directly.

8.3 Handling LLM Response Latency

LLM inference typically takes 1–5 seconds, which can feel quite slow to users. Two common solutions exist:

Streaming: Return data as it's generated so users see a "typewriter effect," dramatically shortening the Time to First Token (TTFT).

err = genkit.Generate(ctx, g,
    ai.WithPrompt(userMessage),
    ai.WithStreaming(func(ctx context.Context, chunk *ai.ModelResponseChunk) error {
        // Push each generated chunk to the frontend (via SSE or WebSocket)
        fmt.Print(chunk.Text())
        return nil
    }),
)

Asynchronous processing: For scenarios that don't require real-time returns (such as integrating with Lark/DingTalk robots), return a 200 status code immediately upon receiving the HTTP message, process it asynchronously using a goroutine, and proactively send messages via the robot API to avoid platform timeout retries.

8.4 Context and Multi-Turn Conversations

Genkit itself does not store conversation history for you; this is something your business logic needs to handle. The common practice is to cache the last N rounds of messages in Redis keyed by user or session ID, and include them with every request:

history := sessionStore.Get(sessionID) // Fetch history messages from Redis

resp, err := genkit.GenerateText(ctx, g,
    ai.WithMessages(history...),       // Inject history context
    ai.WithPrompt(newUserMessage),
)

sessionStore.Append(sessionID, newUserMessage, resp) // Store them back

9. Conclusion

Genkit is not a toy built to showcase demos; it is a framework built for serious AI application engineering. From the plug-and-play design of model plugins and the automated loop of tool calling, to the debugging experience brought by the Dev UI, you can feel that this is a product forged from real-world engineering pain points.

For Go developers, its biggest value might be: You don't need to reinvent the wheel or learn a whole Python ecosystem. Within the language and engineering stack you already know, you can properly integrate, utilize, and maintain LLM capabilities.

If you want to start trying it out, the simplest path is:

1. Run go get github.com/firebase/genkit/go@latest to install dependencies.
2. Go to aistudio.google.com to get a free Gemini API Key.
3. Run the code from Section 5 of this article.
4. Use genkit start -- go run main.go to open the Dev UI and check it out.

Then you'll start wondering—which feature currently in my hands can I add AI capabilities to using this approach?

References

• Official Documentation: genkit.dev
• Community Plugin Collection: github.com/BloomLabsInc/genkit-plugins
• Model Application (Free): Google AI Studio