Article Background: In a multi-agent system project, we had different agents connect to large models from various vendors and run simultaneously. The concurrency orchestration layer was driven by my open-source Go multi-agent library, GopherGraph. This article discusses two topics: first, the interesting phenomena that emerged during this “heterogeneous experiment”; second, the problems related to rate limits when concurrently calling APIs from multiple providers, and how we overcame these challenges step by step.

I. The starting point of the experiment: The same problem, presented to different “brains”

This idea actually comes from a concept in chemistry: isomers.

Isomers refer to compounds that have the same molecular formula (for example, both are C₄H₁₀), but their atoms are arranged differently in space. As a result, their chemical properties are completely different. Butane and isobutane may look “the same,” but their boiling points and reactivity differ significantly.

Our experiment was similar: we fed the same Prompt template to different Agents (the same “molecular formula”), but used different large models behind the scenes (different “structural frameworks”): DeepSeek used one, Qwen used another, and Gemini used yet another.

Observe what different behaviors emerge when they’re faced with the same situation.

The significance of this experiment isn’t merely “comparing which model is smarter.” A more valuable question is: When multiple heterogeneous agents are placed in the same environment, what kind of collective behavior will emerge from them?

II. Experimental Design: The Same Rules, Different “Souls”

Principles of Prompt Design

To ensure the validity of the experimental results, we imposed strict constraints on the design of the Prompt:

All agents share the same set of Prompt templates. The templates include:

1. Identity Settings: The Agent’s name and personality traits (e.g., “cautious and introverted”, “enthusiastic and extroverted”)
2. Environmental Observation: Information that the Agent can perceive at the current moment (who’s nearby, how far away they are, what the other party is doing)
3. Historical Memory: Relevant historical snippets retrieved from the vector database (e.g., who was spoken to and what was said last time)
4. Output Format Requirements: A strict JSON structure must be returned, including action instructions, target location, and spoken content.

Templates themselves are neutral and unbiased. In theory, if all large models have identical capabilities, their outputs should be consistent.

But reality is nothing like that at all.

The differences we observed

After dozens of runs, Agents driven by different models exhibited distinct “personalities”:

Social-oriented vs. Introverted

Some models prefer that the Agent take the initiative to initiate conversations. As long as there are other Agents within its field of vision, it will make its character go over and talk to them. The messages are usually long and enthusiastic, filled with introductory greetings.

Other models, on the other hand, behave completely differently. Even when there are agents nearby, they continue to move toward their destination. They only occasionally say a few words or remain silent altogether.

How to Handle “No-Man’s Land” Areas

When there’s nothing within the Agent’s field of vision, the reactions of different models vary greatly:

• Some models cause the Agent to wait in place, displaying statuses like “Observing surroundings”.
• Some models cause the Agent to actively explore certain coordinates, essentially giving itself something to do.
• In some cases, certain models may occasionally produce unstable output formats, with additional “self-talk” elements mixed into the structured JSON.

Interpretations of the Same Event

This is the most interesting part. When two Agents are both within sight at the same time, and one of them is speaking, different models will react completely differently: one might cause the Agent to respond to the speech, another might ignore it completely and keep walking, and yet another might make the Agent stop and “watch” without saying anything.

These differences aren’t inherently good or bad, but they make the entire environment surprisingly dynamic. In one word: emergence. No one dictates that “things should behave like real society,” yet that’s exactly how it evolves.

III. The Cost of Concurrency: API Rate Limit Exceeded Warnings

The experiments described above sound promising. However, in practical engineering applications, we quickly encountered a very difficult problem.

Where did the problem come from?

To enable all agents to make decisions simultaneously (rather than waiting in line), we’ve adopted a concurrent architecture, with the underlying logic handled by the open-source library GopherGraph. This is a multi-agent workflow engine I’ve written in Go. Its core functionality is to allow all agents to process tasks concurrently, collect their results, and then process them together. This is achieved through a declarative directed graph API, eliminating the need to manually manage goroutines and WaitGroups.

Specifically, with each world tick, GopherGraph expands the decision nodes of all agents into concurrent branches and initiates LLM calls simultaneously. After all calls are completed, the results are merged back into the main state using a Merger function. To the caller, the entire concurrent process remains transparent.

10 agents sending requests simultaneously doesn’t seem like a problem at all.

But among these 10 agents right now: 3 use DeepSeek, 3 use Qwen, 2 use Gemini, and 2 use OpenAI. Each company’s API has its own rate limit.

Taking free quotas or low-cost packages as examples, the common restrictions are roughly as follows (varies greatly depending on the account and package; for reference only):

When we send 10 requests concurrently in the same millisecond, from the server’s perspective, this represents a massive traffic spike in a very short period of time. Even if the total volume doesn’t exceed the limit, the peak value alone can trigger rate limiting.

As a result, some agents received a “429 Too Many Requests” response from the server. This meant that the decision-making data for that tick was incomplete, causing the corresponding agents to become “stuck”.

The Essence of the Problem

The essence of frequency control issues is: the conflict between your concurrency model (requests sent simultaneously) and the API provider’s expectations (requests received smoothly).

Service providers want your requests to come in steadily, like a gentle stream. What we provide, however, is a sudden, massive influx of requests at once.

To solve this problem, there are several approaches. We’ve gone through the following stages of development.

IV. Evolution of Solutions

Phase 1: The simplest approach – simply retry

The first reaction is: Just retry when encountering a 429 error.

go
// Pseudocode illustration
func callLLMWithRetry(ctx context.Context, prompt string) (string, error) {}
maxRetries := 3
for i := 0; i < maxRetries; i++ {
result, err := callLLM(ctx, prompt)
if err == nil {
return result, nil
}
if isRateLimitError(err) {
time.Sleep(time.Second * 2) // Wait 2 seconds before trying again
Continue
}
return "", err // Return any other errors directly
}
return "", fmt.Errorf("Maximum number of retries exceeded")
}

This solution is effective, but there are two problems:
1. **Retry timing is fixed**: All failed requests wait for 2 seconds, then are retried together. This can lead to another surge in requests, resulting in further 429 errors and creating a cycle.
2. **Delay Accumulation**: If multiple Agents trigger retries within a single Tick, the waiting time will significantly prolong the overall execution time of that Tick.

### Phase Two: Exponential Backoff – Smarter Retries

Exponential Backoff is a classic strategy for handling server-side rate limiting: wait 1 second after the first failure, 2 seconds after the second failure, 4 seconds after the third failure, and so on. The waiting time doubles with each attempt, preventing a retry storm.

At the same time, random jitter is added, so that the waiting times aren’t fixed at 1, 2, or 4 seconds. Instead, they range from 1±0.5, 2±1, to 4±2 seconds. This ensures that even when multiple agents attempt to retry at the same time, their retry times will be staggered naturally.

go
// Pseudocode illustration
func exponentialBackoff(attempt int) time.Duration {
base := time.Duration(1<<attempt) * time.Second // 1s, 2s, 4s, 8s…
// Add random jitter: Random values within ±50% of the base time
jitter := time.Duration(rand.Int63n(int64(base)))
return base + jitter
}

This solution works very well when the total amount doesn’t exceed the limit. The success rate of retries is significantly improved.

But we quickly realized that retrying is essentially a form of passive defense—the problem has already occurred before we can address it. Can’t we prevent problems from happening in the first place?

Phase 3: Proactive Peak-Shifting – Request Timing Control

We’ve noticed that the main reason for frequency-controlled triggering is the issuance of a large number of requests at the same time.

In that case, can we proactively spread out concurrent requests along the timeline before they’re actually sent?

We’ve introduced a simple yet effective mechanism: each Agent waits for a random amount of time before initiating a request.

go
// Each Agent “sleeps randomly” for a moment before thinking.
sleepDuration := time.Duration(rand.Intn(3000)) * time.Millisecond
time.Sleep(sleepDuration)

// Then send another request
decision, err := callLLM(ctx, prompt)

This 3000-millisecond (0 to 3 seconds) random window causes the 10 requests that were originally concentrated within one single millisecond to be distributed evenly over the 3-second time period.

From the perspective of API providers, what they receive is a steady stream of requests, not sudden surges.

**This change involves only one line of code, but the effect is immediate: the frequency control trigger rate drops to near zero.**

Of course, the downside of this approach is that it increases the tick time by 0–3 seconds in the best-case scenario. However, since LLM calls themselves take several seconds, this delay is barely noticeable. From the user’s perspective, there’s almost no impact at all.

### Phase Four: Throttling by Manufacturer Group – More Precise Control

The random jitter mentioned above is a global policy that applies equally to all Agents. In reality, however, different vendors have varying frequency control strategies: Gemini’s free quota is extremely limited, while DeepSeek’s is relatively more generous.

A more refined approach is to **maintain a separate token bucket or semaphore for each LLM vendor, thereby limiting the number of concurrent requests sent to that vendor at any given time.**

go
// Pseudocode illustration
type ProviderLimiter struct {}
semaphore chan struct{} // Semaphore for controlling concurrency
}

// Initialization: Gemini allows up to 2 concurrent requests at a time, while DeepSeek allows up to 5.
limiters := map[string]*ProviderLimiter{}
"gemini": {semaphore: make(chan struct{}, 2)},
"deepseek": {semaphore: make(chan struct{}, 5)}
}

func (l *ProviderLimiter) Acquire() {}
l.semaphore <- struct{}{} // Occupies one slot; if full, blocks and waits
}

func (l *ProviderLimiter) Release() {}
<-l.semaphore // Release the slot
}

// Use it
limiter := limiters[npc.Provider]
limiter.Acquire()
defer limiter.Release()
result, err := callLLM(ctx, prompt)

This solution ensures that no manufacturer’s request concurrency ever exceeds the preset limit, transforming “random fluctuations dependent on luck” into “deterministic concurrency control”.

The two approaches aren’t mutually exclusive. We’re using both random jittering (the first line of defense, to prevent traffic spikes) and semaphore throttling (the second line of defense, to control the maximum concurrency level).

V. Unexpected Benefits: Frequency Control Led to a Better Architecture

In dealing with frequency control issues, an unexpected side effect occurred: we began to seriously consider request priorities.

In a multi-agent system, not all LLM calls are equally important. If there are 10 requests waiting in the system at the same time, which ones should be sent first?

For example:

• A certain Agent has just reached a crucial plot point. His decisions will affect multiple surrounding Agents.
• The other agents were just wandering around aimlessly in open areas.

The former clearly has a higher priority. But in the initial design, all requests were completely equal, with no concept of priority at all.

The frequency control issue forced us to introduce priority queues. This, in turn, made the behavior of the entire multi-agent system more “focused,” reducing the waste of tokens on less important decisions.

This is a typical example of a good design that results from being forced by constraints.

VI. Some Reflections on Heterogeneous Experiments

Back to the isomerism experiment at the beginning.

At the engineering level, running models from multiple different vendors simultaneously has clear advantages:

1. Avoiding single-vendor risks: If a vendor’s API experiences sudden fluctuations or downtime, it will only affect certain agents, while the overall system remains operational.
2. Cost Optimization: Simple, high-frequency decisions can be handled by cheaper models, while complex, critical decisions should be entrusted to higher-quality models.
3. Experimental approach: When you’re unsure which model is better for a particular task, you can compare them in the same environment using A/B testing, and use the resulting data to determine the best option.

However, the downside of this architecture is increased operational complexity: You need to manage API Keys and configurations from multiple vendors. Each vendor has different error codes and rate-limiting rules, requiring separate customization for each one.

Whether this level of complexity is worthwhile depends on the scale of your system and your specific requirements. For us, the interesting behavioral differences that arise make it all worthwhile.

Conclusion

The two themes of this article—heterogeneous multi-model experiments and API rate limiting—are essentially two sides of the same issue in our project.

Since heterogeneous experiments require concurrency, frequency control becomes an issue. To address this, it’s necessary to properly manage request priorities and concurrency control.

That’s how engineering problems work: one problem often leads to another. But solving them also helps us gain a deeper understanding of the entire system.

If you’re also working on similar LLM applications, I hope the lessons learned from this article will be useful to you. Feel free to ask questions in the comments section.