• Target Keywords: Golang multi-LLM client, OpenAI compatible API Go, DeepSeek Go API, Gemini Go API, Go OpenAI client pool, LLM provider fallback Golang, go-openai multi-provider.
• Meta Description: Learn how to build a unified, multi-provider LLM client in Golang. We cover configuration abstraction, connection pooling with sync.Mutex, fallback mechanisms, and how to seamlessly switch between OpenAI, Gemini, DeepSeek, and other compatible APIs.

Over the past two years, the number of Large Language Model (LLM) providers has exploded. DeepSeek, Qwen, Gemini, Claude, Doubao... almost every tech giant now has their own API. If you write separate integration code for each provider, your maintenance costs will skyrocket, and adding a new provider will require massive code refactoring.

But there is good news: the vast majority of mainstream LLM providers now offer OpenAI-compatible APIs.

This means you only need to write one set of code. By simply swapping the base_url and api_key, you can connect to dozens of different LLM providers simultaneously.

In this article, we'll share the complete implementation strategy we use in production: How to design a universal multi-provider LLM client in Go. We'll cover dynamic model switching at runtime, connection pooling for performance, and fallback mechanisms for fault tolerance.

1. Why is Everyone Compatible with the OpenAI Protocol?

As a pioneer in the LLM space, OpenAI's Chat Completion API has become the de facto industry standard. The core interface looks something like this:

POST https://api.openai.com/v1/chat/completions

{
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "Hello"}]
}

Most domestic and international providers now offer interfaces compatible with this protocol. The only differences are usually the base_url and the model name:

As long as the base_url and api_key are different, the rest of the code is identical. This is the biggest advantage of protocol compatibility: you only need to maintain one set of calling logic.

In the Go ecosystem, the highly popular github.com/sashabaranov/go-openai library natively supports custom base_urls, making it the perfect foundation for a multi-provider integration.

2. A Unified Configuration Abstraction

The first step is to design a unified configuration struct to abstract away the differences between providers.

// PlatformConfig represents the unified abstraction for an LLM interface.
// Regardless of the provider, you only need these three fields.
type PlatformConfig struct {
    ApiKey  string `yaml:"api_key"`
    Model   string `yaml:"model"`
    BaseURL string `yaml:"base_url"`
}

// LLMConfig manages the configuration collection for all providers.
type LLMConfig struct {
    EmbeddingModel string                    `yaml:"embedding_model"` // Dedicated provider for embeddings
    Default        string                    `yaml:"default"`         // Fallback default provider name
    Provider       map[string]PlatformConfig `yaml:"provider"`        // Map of Provider Name -> Config
}

The corresponding YAML configuration file is very straightforward:

llm:
  embedding_model: "gemini"   # Which provider is used specifically for embeddings
  default: "deepseek"         # Fallback provider if a specified one is not found

  provider:
    gemini:
      api_key: "your-gemini-api-key"
      model: "gemini-2.0-flash"
      base_url: "https://generativelanguage.googleapis.com/v1beta/openai/"

    deepseek:
      api_key: "your-deepseek-api-key"
      model: "deepseek-chat"
      base_url: "https://api.deepseek.com"

    qwen:
      api_key: "your-qwen-api-key"
      model: "qwen-plus"
      base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1"

There are two key design points here:

1. Provider is a map, not a list of hardcoded fields. This means if you want to add a new provider, you just add a few lines to your YAML file. Zero code changes required.
2. Default provides a fallback mechanism. If a specific entity requests a provider that cannot be found (due to a typo in the config, or the provider being deprecated), the system automatically falls back to the default provider instead of crashing.

3. Configuration Lookup with Graceful Fallback

Based on the struct above, we implement a method to fetch the configuration by provider name:

// GetConfigByName fetches the configuration based on the provider's name.
// If the specified name is not found, or if the name is empty, it returns the Default fallback.
func (l *LLMConfig) GetConfigByName(name string) PlatformConfig {
    if cfg, ok := l.Provider[name]; ok && name != "" {
        return cfg
    }
    // Fallback: return the Default configuration
    return l.Provider[l.Default]
}

Although this function is only five lines long, its design philosophy is crucial:

• name != "": Prevents an empty string from being treated as a valid key and returning a zero-value struct.
• Fallback to Default: The system does not interrupt execution due to a minor configuration error. This is graceful degradation.
• Returns by value (not pointer): The caller gets an independent copy of the struct, preventing accidental concurrent modification issues.

Using it looks like this:

// Each business entity can independently specify which provider it uses
providerName := entity.Provider  // e.g., "gemini" or "deepseek"
cfg := config.LLM.GetConfigByName(providerName)

// Once you have the apiKey, model, and baseURL, you can make the call
result, err := callLLM(ctx, cfg.ApiKey, cfg.BaseURL, cfg.Model, prompt)

4. Client Connection Pooling: Solving the Initialization Bottleneck

Once we have the configuration, the next step is creating the HTTP client. However, there is an easily overlooked performance issue:

If you create a new openai.Client for every LLM call, you will incur massive overhead from connection establishment and TLS handshakes.

In high-concurrency scenarios (e.g., dozens of AI agents calling LLMs simultaneously), this overhead becomes a severe bottleneck.

The solution is to maintain a global client connection pool: for any unique (baseURL, apiKey) combination, we always reuse the same client instance.

var (
    mu      sync.Mutex
    clients = make(map[string]*openai.Client)
)

// GetOpenAIClient retrieves an existing client or creates a new singleton client.
// It guarantees that the same instance is returned for the same baseURL + apiKey combination.
func GetOpenAIClient(apiKey string, baseURL string) *openai.Client {
    mu.Lock()
    defer mu.Unlock()

    // Create a unique key by combining baseURL and apiKey
    key := baseURL + "|" + apiKey

    // If the client for this provider already exists in the pool, reuse it
    if cli, ok := clients[key]; ok {
        return cli
    }

    // If not, create a new one and cache it in the pool
    config := openai.DefaultConfig(apiKey)
    if baseURL != "" {
        config.BaseURL = baseURL
    }

    cli := openai.NewClientWithConfig(config)
    clients[key] = cli

    return cli
}

Design Decision Analysis:

Why use baseURL + "|" + apiKey as the key, rather than just the provider name?
Because you might have multiple accounts for the same provider (e.g., you are using two DeepSeek API keys for load balancing). A composite key precisely distinguishes them.

Why use sync.Mutex instead of sync.RWMutex?
After initialization, the clients in the pool are only read, not written to. Theoretically, a read lock from RWMutex seems more appropriate. However, in practice, client creation only happens during cold starts (the first time a provider is requested). After that, it's entirely read operations. The lock contention from a standard Mutex in this specific scenario is completely negligible. Keeping things simple is the better choice here.

5. The Universal Chat Completion Function

With the connection pool in place, wrapping a universal calling function is natural:

// CallChat is a universal LLM chat interface, compatible with all OpenAI-protocol providers.
func CallChat(ctx context.Context, apiKey, baseURL, model, prompt string) (string, error) {
    client := GetOpenAIClient(apiKey, baseURL)

    req := openai.ChatCompletionRequest{
        Model: model,
        // Force the return format to JSON (Note: not all providers support this, requires testing)
        ResponseFormat: &openai.ChatCompletionResponseFormat{
            Type: openai.ChatCompletionResponseFormatTypeJSONObject,
        },
        Messages: []openai.ChatCompletionMessage{
            {
                Role:    openai.ChatMessageRoleUser,
                Content: prompt,
            },
        },
    }

    resp, err := client.CreateChatCompletion(ctx, req)
    if err != nil {
        return "", fmt.Errorf("LLM API call failed: %w", err)
    }

    if len(resp.Choices) == 0 {
        return "", fmt.Errorf("LLM returned no choices")
    }

    return resp.Choices[0].Message.Content, nil
}

Putting these layers together, the complete execution chain looks like this:

YAML Configuration
    ↓
GetConfigByName("deepseek")  →  Returns PlatformConfig{ApiKey, Model, BaseURL}
    ↓
GetOpenAIClient(apiKey, baseURL)  →  Reuses or creates the client
    ↓
CallChat(ctx, apiKey, baseURL, model, prompt)  →  Returns the string response

Any entity in your application only needs to hold a providerName string to route its request to the correct LLM model.

6. Real-World Gotchas and Pitfalls

Pitfall 1: Not All Providers Support ResponseFormat: JSON

ResponseFormatTypeJSONObject is an OpenAI feature. While the protocol is compatible, some providers have incomplete support for this parameter. Some will silently ignore it, while others will throw an error.

Solution: Add a SupportsJSONMode bool field to your PlatformConfig. Decide whether to enable this parameter based on the provider's configuration. For providers that don't support it, rely on strict Prompt Engineering to enforce JSON output as a fallback.

Pitfall 2: Inconsistent Error Codes Across Providers

Although the request format is compatible, different providers often return varying error codes and message fields. Some return standard OpenAI error formats, while others use custom structures.

Solution: Unify error handling within the CallChat function. Catch errors and wrap them using fmt.Errorf("... %w", err) to bubble them up. Avoid attempting provider-specific error parsing at this abstraction layer unless absolutely necessary.

Pitfall 3: Context Leaks in Concurrent Scenarios

When a large number of concurrent calls are dispatched, if you use the same top-level context.Context for all of them, a timeout in one call might accidentally cancel other concurrently running calls depending on how the context tree is structured.

Solution: Always create an independent, timeout-bound child context for every individual CallChat invocation, rather than directly passing down the parent context:

func CallChat(ctx context.Context, ...) (string, error) {
    // Create an independent timeout for this specific call, isolating it from other concurrent calls
    callCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
    defer cancel()

    resp, err := client.CreateChatCompletion(callCtx, req)
    // ...
}

7. Future Extensions: Where Can We Take This?

The implementation above covers 80% of use cases, but there are several directions for further extension:

1. Load Balancing: Configure multiple API Keys for a single provider and randomly select one per call to distribute request pressure and mitigate rate limits.
2. Health Checks & Auto-Switching: If a provider fails continuously beyond a certain threshold (e.g., 3 consecutive errors), automatically mark it as "unavailable" and route subsequent requests to the fallback default provider.
3. Cost Tracking: Extract the consumed token count from resp.Usage inside CallChat to perform cost analysis on a per-provider basis.
4. Dynamic Hot-Swapping: Integrate with a configuration center (like Consul or etcd) to update API keys or switch default models at runtime without restarting the service.

Summary

The core philosophy of this entire architecture can be summarized in one sentence:

Encapsulate all provider differences within the configuration, and expose a unified interface to your business logic.

In terms of code volume, the core logic for this multi-provider client is less than 80 lines of Go code. But the benefits are long-lasting: adding a new provider only requires changing the config file, switching a specific entity's model takes a one-field update, and fallback mechanisms happen automatically during system failures.

The widespread adoption of the OpenAI-compatible protocol has drastically lowered the engineering cost of this approach. In the Go ecosystem, leveraging the go-openai library allows you to build a production-ready, multi-provider LLM gateway with just a few dozen lines of code.

If your project requires integrating multiple LLM providers simultaneously, we hope this article provides a concrete, referenceable blueprint for your implementation.