[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-49":3},{"id":4,"title":5,"title_en":6,"abstract":7,"abstract_en":8,"content":9,"content_en":10,"category":11,"banner_id":12,"banner_path":13,"tags":14,"is_recommend":17,"prev_article":18,"next_article":22,"created_at":26},49,"Go + Gemini 多模型并发：让不同大脑同台竞技，以及差点打爆 API 的那段经历","Go + Gemini Multi-Model Concurrency: Putting Different “Brains” Head-to-Head—and the Time We Almost Crashed the API","> **文章背景**：在一个多智能体系统项目里，我们让不同的 Agent 分别接入不同厂商的大模型同","Article Background**: In a multi-agent system project, we had different agents connect to large models from various vendors and run simultaneously. The concurrency orchestration layer was driven by my open-source Go multi-agent library,","> **文章背景**：在一个多智能体系统项目里，我们让不同的 Agent 分别接入不同厂商的大模型同时运行。并发编排层由我开源的 Go 多智能体库 **[GopherGraph](https:\u002F\u002Fgithub.com\u002Funclesam-ly\u002FGopherGraph)** 驱动。这篇文章聊两件事：第一，这个\"异构实验\"里涌现出了什么有趣的现象；第二，并发调用多家 API 时遭遇的频率限制（Rate Limit）噩梦，以及我们是怎么一步步解决的。\n\n---\n\n## 一、实验的起点：同一个问题，交给不同的\"大脑\"\n\n这个想法其实来自化学里的一个概念——**同分异构体**。\n\n同分异构体是指：分子式完全相同（比如都是 C₄H₁₀），但原子在空间里的排列方式不同，因此化学性质截然不同。丁烷和异丁烷，看起来\"一样\"，但沸点、反应性完全不同。\n\n我们的实验也是如此：给不同的 Agent 输入**完全相同的 Prompt 模板**（相同的\"分子式\"），但背后调用的大模型不同（不同的\"空间结构\"）——DeepSeek 用一个，Qwen 用一个，Gemini 用一个。\n\n观察它们面对同样处境时，涌现出什么样的差异化行为。\n\n这个实验的意义不仅仅是\"对比哪个模型更聪明\"，更有价值的问题是：**当多个异构 Agent 被放入同一个环境里，他们的群体行为会涌现出什么？**\n\n---\n\n## 二、实验设计：同一套规则，不同的灵魂\n\n### Prompt 的设计原则\n\n为了让实验结果有意义，我们对 Prompt 的设计做了严格约束：\n\n**所有 Agent 共享同一套 Prompt 模板**，模板里包含：\n1. **身份设定**：Agent 的姓名、性格标签（比如\"谨慎内向\"、\"热情外向\"）\n2. **环境观察**：当前时刻 Agent 能感知到的信息（周边有谁、距离多远、对方在做什么）\n3. **历史记忆**：从向量数据库里检索出来的相关历史片段（上次和谁说了什么）\n4. **输出格式约束**：要求返回严格的 JSON 结构，包含行动指令、目标位置、说话内容\n\n模板本身是中性的、没有偏向性的。理论上，如果所有大模型能力完全相同，它们的输出应该趋于一致。\n\n但现实完全不是这样。\n\n### 我们观察到的差异\n\n经过数十轮的运行，不同模型驱动的 Agent 表现出了非常鲜明的\"个性\"：\n\n**倾向社交 vs 倾向独处**\n\n有些模型特别喜欢让 Agent 主动发起对话。只要视野范围内有其他 Agent，它就会让自己的角色走过去搭话，而且说话内容往往比较长、比较热情，充满开场白式的问候语。\n\n另一些模型则截然相反，即使周边有 Agent，也会选择继续向某个目的地移动，偶尔才说一两句简短的话，或者干脆保持沉默。\n\n**对\"无人区\"的处理方式**\n\n当 Agent 的视野范围内什么都没有时，不同模型的反应差异极大：\n\n- 有的模型会让 Agent 原地等待，输出\"正在观察周围\"之类的状态\n- 有的模型会让 Agent 主动去探索某个坐标，给自己找事做\n- 有的模型在这种情况下偶尔会出现输出格式不稳定的情况，结构化 JSON 里混入一些额外的自言自语\n\n**对同一事件的\"解读\"**\n\n这是最有趣的部分。当两个 Agent 同时在视野范围内，并且其中一个正在说话，不同模型会给出截然不同的反应：一个可能让 Agent 回应这句话，另一个可能完全无视并继续走路，还有一个可能会让 Agent 停下来\"围观\"，但不发言。\n\n这些差异没有好坏之分，但它们让整个环境变得**出乎意料地生动**。用一个词形容：涌现性（Emergence）——没有人规定\"你们要表现得像真实社会\"，但它自己演化出来了。\n\n---\n\n## 三、并发的代价：API 频控打爆警告\n\n上面的实验听起来很美好。但在工程实践上，我们很快遇到了一个极其棘手的问题。\n\n### 问题是怎么来的\n\n为了让所有 Agent 同时做决策（而不是排队等），我们使用了并发架构——底层由开源库 **[GopherGraph](https:\u002F\u002Fgithub.com\u002Funclesam-ly\u002FGopherGraph)** 编排。它是我用 Go 写的一个多智能体工作流引擎，核心能力是把\"让所有 Agent 并发思考，收齐结果再统一处理\"这件事，用声明式的有向图 API 表达出来，不需要手写一堆 goroutine 和 WaitGroup。\n\n具体来说，每次世界心跳（Tick）时，GopherGraph 会把所有 Agent 的决策节点展开为并发分支同时发起 LLM 调用，全部完成后再通过 Merger 函数把结果汇聚回主状态。对调用方来说，整个并发过程是透明的。\n\n10 个 Agent，同时发请求，听起来没什么问题。\n\n但现在这 10 个 Agent 里：3 个用 DeepSeek，3 个用 Qwen，2 个用 Gemini，2 个用 OpenAI。每家的 API 都有自己的**频率限制（Rate Limit）**。\n\n以免费额度或低价套餐为例，常见的限制大概是这样的（不同账号和套餐差异很大，仅供参考）：\n\n| 限制维度 | 含义 | 常见数值范围 |\n|----------|------|-------------|\n| RPM（每分钟请求数） | 每分钟最多发多少次请求 | 几十次 ～ 几百次不等 |\n| TPM（每分钟 Token 数） | 每分钟最多处理多少 Token | 几万 ～ 几十万不等 |\n| RPD（每天请求数） | 每天的总请求上限 | 几百次 ～ 几千次不等 |\n\n当我们在同一毫秒内并发发出 10 个请求，在服务端看来，这是在极短的时间内产生了一个巨大的流量尖峰。即使总量没超，峰值也可能触发限流。\n\n结果就是：部分 Agent 的请求被服务器返回 `429 Too Many Requests`，这次 Tick 的决策数据不完整，对应的 Agent 就\"卡住\"了。\n\n### 问题的本质\n\n频控问题的本质是：**你的并发模型（同时发请求）和 API 服务商的期望（平滑地接收请求）之间的矛盾。**\n\n服务商希望你的请求像涓涓细流一样均匀地流进来，而我们给的是一次性的洪峰。\n\n要解决这个问题，有几种思路，我们经历了以下几个演进阶段。\n\n---\n\n## 四、解决方案的演进\n\n### 阶段一：最朴素的方案——直接重试\n\n第一反应是：遇到 429 就重试呗。\n\n```go\n\u002F\u002F 伪代码示意\nfunc callLLMWithRetry(ctx context.Context, prompt string) (string, error) {\n    maxRetries := 3\n    for i := 0; i \u003C maxRetries; i++ {\n        result, err := callLLM(ctx, prompt)\n        if err == nil {\n            return result, nil\n        }\n        if isRateLimitError(err) {\n            time.Sleep(time.Second * 2) \u002F\u002F 等 2 秒再试\n            continue\n        }\n        return \"\", err \u002F\u002F 其他错误直接返回\n    }\n    return \"\", fmt.Errorf(\"超过最大重试次数\")\n}\n```\n\n这个方案有效，但有两个问题：\n1. **重试时间是固定的**：所有失败的请求都等 2 秒，然后一起重试，可能再次产生洪峰，继续 429，陷入循环。\n2. **延迟叠加**：如果一次 Tick 里有多个 Agent 触发重试，等待时间会把整个 Tick 的执行时间拖得很长。\n\n### 阶段二：指数退避——更聪明的重试\n\n指数退避（Exponential Backoff）是处理服务端限流的经典策略：第一次失败等 1 秒，第二次失败等 2 秒，第三次等 4 秒……每次等待时间翻倍，避免重试风暴。\n\n同时，加入随机抖动（Jitter），让等待时间不是固定的 1、2、4 秒，而是 1±0.5、2±1、4±2 这样带随机范围的值。这样即使多个 Agent 同时触发重试，它们的重试时间也会自然错开。\n\n```go\n\u002F\u002F 伪代码示意\nfunc exponentialBackoff(attempt int) time.Duration {\n    base := time.Duration(1\u003C\u003Cattempt) * time.Second \u002F\u002F 1s, 2s, 4s, 8s...\n    \u002F\u002F 加入随机抖动：在基础时间的 ±50% 范围内随机\n    jitter := time.Duration(rand.Int63n(int64(base)))\n    return base + jitter\n}\n```\n\n这个方案在总量没超限的情况下效果很好，重试成功率大幅提升。\n\n但我们很快发现，重试本质上是一种**被动防御**——问题已经发生了，再去处理。能不能在问题发生之前就避免它？\n\n### 阶段三：主动错峰——请求时序控制\n\n我们注意到，频控触发的核心原因是**同一时刻发出大量请求**。\n\n既然如此，能不能在并发请求发出之前，就主动把它们在时间轴上分散开？\n\n我们引入了一个简单但非常有效的机制：**在每个 Agent 发起请求之前，随机等待一段时间**。\n\n```go\n\u002F\u002F 每个 Agent 在思考之前先\"随机睡一会儿\"\nsleepDuration := time.Duration(rand.Intn(3000)) * time.Millisecond\ntime.Sleep(sleepDuration)\n\n\u002F\u002F 然后再发请求\ndecision, err := callLLM(ctx, prompt)\n```\n\n这 3000 毫秒（0 ～ 3 秒）的随机窗口，让原本集中在同一毫秒的 10 个请求，自然地散布在了 3 秒的时间线上。\n\n从 API 服务商的角度来看，收到的就是平滑的请求流，而不是洪峰。\n\n**这个改动只有一行，但效果立竿见影：频控触发率降到接近零。**\n\n当然，这个方案的代价是把最快的情况下的 Tick 时间增加了 0-3 秒。但由于 LLM 调用本身就需要几秒，这个延迟几乎被掩盖了，从用户感知的角度来说几乎没有影响。\n\n### 阶段四：按厂商分组限流——更精细的控制\n\n上面的随机抖动是一个全局策略，对所有 Agent 一视同仁。但实际上，不同厂商的频控策略是不同的：Gemini 的免费额度极其有限，DeepSeek 相对宽松。\n\n更精细的做法是：**为每个 LLM 厂商维护一个独立的令牌桶（Token Bucket）或信号量（Semaphore），限制同时向这家厂商发请求的并发数。**\n\n```go\n\u002F\u002F 伪代码示意\ntype ProviderLimiter struct {\n    semaphore chan struct{} \u002F\u002F 信号量，控制并发\n}\n\n\u002F\u002F 初始化：Gemini 最多同时 2 个并发请求，DeepSeek 最多 5 个\nlimiters := map[string]*ProviderLimiter{\n    \"gemini\":   {semaphore: make(chan struct{}, 2)},\n    \"deepseek\": {semaphore: make(chan struct{}, 5)},\n}\n\nfunc (l *ProviderLimiter) Acquire() {\n    l.semaphore \u003C- struct{}{} \u002F\u002F 占用一个槽位，如果满了就阻塞等待\n}\n\nfunc (l *ProviderLimiter) Release() {\n    \u003C-l.semaphore \u002F\u002F 释放槽位\n}\n\n\u002F\u002F 使用\nlimiter := limiters[npc.Provider]\nlimiter.Acquire()\ndefer limiter.Release()\nresult, err := callLLM(ctx, prompt)\n```\n\n这个方案让每家厂商的请求并发数在任何时刻都不超过预设上限，把\"依赖运气的随机抖动\"升级为\"确定性的并发控制\"。\n\n两种方案并不互斥，我们现在同时使用了随机抖动（第一道防线，预防洪峰）和信号量限流（第二道防线，控制并发上限）。\n\n---\n\n## 五、意外收获：频控倒逼出了更好的架构\n\n处理频控问题的过程，意外地带来了一个副产品——我们开始认真思考**请求的优先级**。\n\n在一个多 Agent 系统里，不是所有的 LLM 调用都同等重要。如果系统同时有 10 个请求在等待，哪些应该先发出去？\n\n比如：\n- 某个 Agent 刚刚触碰到了一个关键的剧情节点，他的决策会影响周围多个 Agent\n- 另外几个 Agent 只是在空旷区域漫无目的地闲逛\n\n前者的优先级显然更高。但在最初的设计里，所有请求是完全平等的，没有优先级的概念。\n\n频控问题让我们不得不引入优先级队列，这反过来让整个多智能体系统的行为变得更有\"重点\"，减少了在不重要的决策上浪费 Token 的情况。\n\n这是一个典型的**被约束逼出来的好设计**。\n\n---\n\n## 六、关于异构实验的一点反思\n\n回到最开始的同分异构体实验。\n\n在工程层面，让多个不同厂商的模型同时跑有明显的好处：\n\n1. **规避单一厂商风险**：某家厂商的 API 突然抖动或宕机，只影响部分 Agent，系统整体仍然可用\n2. **成本优化**：可以把高频但简单的决策交给便宜的模型，把关键复杂的决策交给高质量的模型\n3. **实验性**：当你不确定哪个模型更适合某类任务时，可以让它们在同一环境里 A\u002FB，用运行数据来回答问题\n\n但这种架构的代价是**运维复杂度的提升**：你需要同时维护多个厂商的 API Key 和配置，每家的错误码和频控规则都不一样，需要分别适配。\n\n这种复杂度是否值得，取决于你的系统规模和具体需求。对我们来说，涌现出来的有趣行为差异，让这一切都是值得的。\n\n---\n\n## 结语\n\n这篇文章的两个主题——异构多模型实验和 API 频控处理——在我们项目里其实是同一个问题的两面。\n\n**因为想要做异构实验，所以需要并发；因为并发，所以遭遇了频控；因为要解决频控，所以被逼着把请求优先级和并发控制都想清楚了。**\n\n工程问题就是这样，往往是一个问题套着另一个问题。但解决它们的过程，也是对系统理解不断加深的过程。\n\n如果你也在做类似的 LLM 应用，希望这篇文章里的踩坑经验对你有用。有问题欢迎评论区交流。\n","> **Article Background**: In a multi-agent system project, we had different agents connect to large models from various vendors and run simultaneously. The concurrency orchestration layer was driven by my open-source Go multi-agent library, **[GopherGraph](https:\u002F\u002Fgithub.com\u002Funclesam-ly\u002FGopherGraph)**. This article discusses two topics: first, the interesting phenomena that emerged during this “heterogeneous experiment”; second, the problems related to rate limits when concurrently calling APIs from multiple providers, and how we overcame these challenges step by step.\n\n---\n\n## I. The starting point of the experiment: The same problem, presented to different “brains”\n\nThis idea actually comes from a concept in chemistry: **isomers**.\n\nIsomers refer to compounds that have the same molecular formula (for example, both are C₄H₁₀), but their atoms are arranged differently in space. As a result, their chemical properties are completely different. Butane and isobutane may look “the same,” but their boiling points and reactivity differ significantly.\n\nOur experiment was similar: we fed the same **Prompt template** to different Agents (the same “molecular formula”), but used different large models behind the scenes (different “structural frameworks”): DeepSeek used one, Qwen used another, and Gemini used yet another.\n\nObserve what different behaviors emerge when they’re faced with the same situation.\n\nThe significance of this experiment isn’t merely “comparing which model is smarter.” A more valuable question is: **When multiple heterogeneous agents are placed in the same environment, what kind of collective behavior will emerge from them?**\n\n---\n\n## II. Experimental Design: The Same Rules, Different “Souls”\n\n### Principles of Prompt Design\n\nTo ensure the validity of the experimental results, we imposed strict constraints on the design of the Prompt:\n\n**All agents share the same set of Prompt templates.** The templates include:\n1. **Identity Settings**: The Agent’s name and personality traits (e.g., “cautious and introverted”, “enthusiastic and extroverted”)\n2. **Environmental Observation**: Information that the Agent can perceive at the current moment (who’s nearby, how far away they are, what the other party is doing)\n3. **Historical Memory**: Relevant historical snippets retrieved from the vector database (e.g., who was spoken to and what was said last time)\n4. **Output Format Requirements**: A strict JSON structure must be returned, including action instructions, target location, and spoken content.\n\nTemplates themselves are neutral and unbiased. In theory, if all large models have identical capabilities, their outputs should be consistent.\n\nBut reality is nothing like that at all.\n\n### The differences we observed\n\nAfter dozens of runs, Agents driven by different models exhibited distinct “personalities”:\n\n**Social-oriented vs. Introverted**\n\nSome models prefer that the Agent take the initiative to initiate conversations. As long as there are other Agents within its field of vision, it will make its character go over and talk to them. The messages are usually long and enthusiastic, filled with introductory greetings.\n\nOther models, on the other hand, behave completely differently. Even when there are agents nearby, they continue to move toward their destination. They only occasionally say a few words or remain silent altogether.\n\n**How to Handle “No-Man’s Land” Areas**\n\nWhen there’s nothing within the Agent’s field of vision, the reactions of different models vary greatly:\n\n- Some models cause the Agent to wait in place, displaying statuses like “Observing surroundings”.\n- Some models cause the Agent to actively explore certain coordinates, essentially giving itself something to do.\n- In some cases, certain models may occasionally produce unstable output formats, with additional “self-talk” elements mixed into the structured JSON.\n\n**Interpretations of the Same Event**\n\nThis is the most interesting part. When two Agents are both within sight at the same time, and one of them is speaking, different models will react completely differently: one might cause the Agent to respond to the speech, another might ignore it completely and keep walking, and yet another might make the Agent stop and “watch” without saying anything.\n\nThese differences aren’t inherently good or bad, but they make the entire environment **surprisingly dynamic**. In one word: emergence. No one dictates that “things should behave like real society,” yet that’s exactly how it evolves.\n\n---\n\n## III. The Cost of Concurrency: API Rate Limit Exceeded Warnings\n\nThe experiments described above sound promising. However, in practical engineering applications, we quickly encountered a very difficult problem.\n\n### Where did the problem come from?\n\nTo enable all agents to make decisions simultaneously (rather than waiting in line), we’ve adopted a concurrent architecture, with the underlying logic handled by the open-source library **[GopherGraph](https:\u002F\u002Fgithub.com\u002Funclesam-ly\u002FGopherGraph)**. This is a multi-agent workflow engine I’ve written in Go. Its core functionality is to allow all agents to process tasks concurrently, collect their results, and then process them together. This is achieved through a declarative directed graph API, eliminating the need to manually manage goroutines and WaitGroups.\n\nSpecifically, with each world tick, GopherGraph expands the decision nodes of all agents into concurrent branches and initiates LLM calls simultaneously. After all calls are completed, the results are merged back into the main state using a Merger function. To the caller, the entire concurrent process remains transparent.\n\n10 agents sending requests simultaneously doesn’t seem like a problem at all.\n\nBut among these 10 agents right now: 3 use DeepSeek, 3 use Qwen, 2 use Gemini, and 2 use OpenAI. Each company’s API has its own **rate limit**.\n\nTaking free quotas or low-cost packages as examples, the common restrictions are roughly as follows (varies greatly depending on the account and package; for reference only):\n\n| Limitation Dimension | Meaning | Common Value Ranges |\n|----------|------|-------------|\n| RPM (Requests Per Minute) | Maximum number of requests per minute | Ranging from dozens to hundreds |\n| TPM (Tokens per Minute) | Maximum number of tokens processed per minute | Ranging from tens of thousands to hundreds of thousands |\n| RPD (Requests per Day) | Daily Maximum Requests | Several hundred to several thousand requests |\n\nWhen we send 10 requests concurrently in the same millisecond, from the server’s perspective, this represents a massive traffic spike in a very short period of time. Even if the total volume doesn’t exceed the limit, the peak value alone can trigger rate limiting.\n\nAs a result, some agents received a “429 Too Many Requests” response from the server. This meant that the decision-making data for that tick was incomplete, causing the corresponding agents to become “stuck”.\n\n### The Essence of the Problem\n\nThe essence of frequency control issues is: **the conflict between your concurrency model (requests sent simultaneously) and the API provider’s expectations (requests received smoothly).**\n\nService providers want your requests to come in steadily, like a gentle stream. What we provide, however, is a sudden, massive influx of requests at once.\n\nTo solve this problem, there are several approaches. We’ve gone through the following stages of development.\n\n---\n\n## IV. Evolution of Solutions\n\n### Phase 1: The simplest approach – simply retry\n\nThe first reaction is: Just retry when encountering a 429 error.\n\ngo\n\u002F\u002F Pseudocode illustration\nfunc callLLMWithRetry(ctx context.Context, prompt string) (string, error) {}\nmaxRetries := 3\nfor i := 0; i \u003C maxRetries; i++ {\nresult, err := callLLM(ctx, prompt)\nif err == nil {\nreturn result, nil\n}\nif isRateLimitError(err) {\ntime.Sleep(time.Second * 2) \u002F\u002F Wait 2 seconds before trying again\nContinue\n}\nreturn \"\", err \u002F\u002F Return any other errors directly\n}\nreturn \"\", fmt.Errorf(\"Maximum number of retries exceeded\")\n}\n```\n\nThis solution is effective, but there are two problems:\n1. **Retry timing is fixed**: All failed requests wait for 2 seconds, then are retried together. This can lead to another surge in requests, resulting in further 429 errors and creating a cycle.\n2. **Delay Accumulation**: If multiple Agents trigger retries within a single Tick, the waiting time will significantly prolong the overall execution time of that Tick.\n\n### Phase Two: Exponential Backoff – Smarter Retries\n\nExponential Backoff is a classic strategy for handling server-side rate limiting: wait 1 second after the first failure, 2 seconds after the second failure, 4 seconds after the third failure, and so on. The waiting time doubles with each attempt, preventing a retry storm.\n\nAt the same time, random jitter is added, so that the waiting times aren’t fixed at 1, 2, or 4 seconds. Instead, they range from 1±0.5, 2±1, to 4±2 seconds. This ensures that even when multiple agents attempt to retry at the same time, their retry times will be staggered naturally.\n\ngo\n\u002F\u002F Pseudocode illustration\nfunc exponentialBackoff(attempt int) time.Duration {\nbase := time.Duration(1\u003C\u003Cattempt) * time.Second \u002F\u002F 1s, 2s, 4s, 8s…\n\u002F\u002F Add random jitter: Random values within ±50% of the base time\njitter := time.Duration(rand.Int63n(int64(base)))\nreturn base + jitter\n}\n```\n\nThis solution works very well when the total amount doesn’t exceed the limit. The success rate of retries is significantly improved.\n\nBut we quickly realized that retrying is essentially a form of **passive defense**—the problem has already occurred before we can address it. Can’t we prevent problems from happening in the first place?\n\n### Phase 3: Proactive Peak-Shifting – Request Timing Control\n\nWe’ve noticed that the main reason for frequency-controlled triggering is **the issuance of a large number of requests at the same time**.\n\nIn that case, can we proactively spread out concurrent requests along the timeline before they’re actually sent?\n\nWe’ve introduced a simple yet effective mechanism: **each Agent waits for a random amount of time before initiating a request**.\n\ngo\n\u002F\u002F Each Agent “sleeps randomly” for a moment before thinking.\nsleepDuration := time.Duration(rand.Intn(3000)) * time.Millisecond\ntime.Sleep(sleepDuration)\n\n\u002F\u002F Then send another request\ndecision, err := callLLM(ctx, prompt)\n```\n\nThis 3000-millisecond (0 to 3 seconds) random window causes the 10 requests that were originally concentrated within one single millisecond to be distributed evenly over the 3-second time period.\n\nFrom the perspective of API providers, what they receive is a steady stream of requests, not sudden surges.\n\n**This change involves only one line of code, but the effect is immediate: the frequency control trigger rate drops to near zero.**\n\nOf course, the downside of this approach is that it increases the tick time by 0–3 seconds in the best-case scenario. However, since LLM calls themselves take several seconds, this delay is barely noticeable. From the user’s perspective, there’s almost no impact at all.\n\n### Phase Four: Throttling by Manufacturer Group – More Precise Control\n\nThe random jitter mentioned above is a global policy that applies equally to all Agents. In reality, however, different vendors have varying frequency control strategies: Gemini’s free quota is extremely limited, while DeepSeek’s is relatively more generous.\n\nA more refined approach is to **maintain a separate token bucket or semaphore for each LLM vendor, thereby limiting the number of concurrent requests sent to that vendor at any given time.**\n\ngo\n\u002F\u002F Pseudocode illustration\ntype ProviderLimiter struct {}\nsemaphore chan struct{} \u002F\u002F Semaphore for controlling concurrency\n}\n\n\u002F\u002F Initialization: Gemini allows up to 2 concurrent requests at a time, while DeepSeek allows up to 5.\nlimiters := map[string]*ProviderLimiter{}\n\"gemini\": {semaphore: make(chan struct{}, 2)},\n\"deepseek\": {semaphore: make(chan struct{}, 5)}\n}\n\nfunc (l *ProviderLimiter) Acquire() {}\nl.semaphore \u003C- struct{}{} \u002F\u002F Occupies one slot; if full, blocks and waits\n}\n\nfunc (l *ProviderLimiter) Release() {}\n\u003C-l.semaphore \u002F\u002F Release the slot\n}\n\n\u002F\u002F Use it\nlimiter := limiters[npc.Provider]\nlimiter.Acquire()\ndefer limiter.Release()\nresult, err := callLLM(ctx, prompt)\n```\n\nThis solution ensures that no manufacturer’s request concurrency ever exceeds the preset limit, transforming “random fluctuations dependent on luck” into “deterministic concurrency control”.\n\nThe two approaches aren’t mutually exclusive. We’re using both random jittering (the first line of defense, to prevent traffic spikes) and semaphore throttling (the second line of defense, to control the maximum concurrency level).\n\n---\n\n## V. Unexpected Benefits: Frequency Control Led to a Better Architecture\n\nIn dealing with frequency control issues, an unexpected side effect occurred: we began to seriously consider **request priorities**.\n\nIn a multi-agent system, not all LLM calls are equally important. If there are 10 requests waiting in the system at the same time, which ones should be sent first?\n\nFor example:\n- A certain Agent has just reached a crucial plot point. His decisions will affect multiple surrounding Agents.\n- The other agents were just wandering around aimlessly in open areas.\n\nThe former clearly has a higher priority. But in the initial design, all requests were completely equal, with no concept of priority at all.\n\nThe frequency control issue forced us to introduce priority queues. This, in turn, made the behavior of the entire multi-agent system more “focused,” reducing the waste of tokens on less important decisions.\n\nThis is a typical example of a **good design that results from being forced by constraints**.\n\n---\n\n## VI. Some Reflections on Heterogeneous Experiments\n\nBack to the isomerism experiment at the beginning.\n\nAt the engineering level, running models from multiple different vendors simultaneously has clear advantages:\n\n1. **Avoiding single-vendor risks**: If a vendor’s API experiences sudden fluctuations or downtime, it will only affect certain agents, while the overall system remains operational.\n2. **Cost Optimization**: Simple, high-frequency decisions can be handled by cheaper models, while complex, critical decisions should be entrusted to higher-quality models.\n3. **Experimental approach**: When you’re unsure which model is better for a particular task, you can compare them in the same environment using A\u002FB testing, and use the resulting data to determine the best option.\n\nHowever, the downside of this architecture is **increased operational complexity**: You need to manage API Keys and configurations from multiple vendors. Each vendor has different error codes and rate-limiting rules, requiring separate customization for each one.\n\nWhether this level of complexity is worthwhile depends on the scale of your system and your specific requirements. For us, the interesting behavioral differences that arise make it all worthwhile.\n\n---\n\n## Conclusion\n\nThe two themes of this article—heterogeneous multi-model experiments and API rate limiting—are essentially two sides of the same issue in our project.\n\n**Since heterogeneous experiments require concurrency, frequency control becomes an issue. To address this, it’s necessary to properly manage request priorities and concurrency control.**\n\nThat’s how engineering problems work: one problem often leads to another. But solving them also helps us gain a deeper understanding of the entire system.\n\nIf you’re also working on similar LLM applications, I hope the lessons learned from this article will be useful to you. Feel free to ask questions in the comments section.\n","AI",20,"https:\u002F\u002Fblog4-1316398321.cos.ap-nanjing.myqcloud.com\u002Fblog5\u002F20250823013157__暗黑光影.png",[15,11,16],"GO","Gemini",false,{"id":19,"title":20,"title_en":21},48,"我是如何用 Go 语言造出一个\"虚拟世界\"的——从死锁地狱到事件驱动架构","How I Built a \"Virtual World\" in Go: From Deadlock Hell to Event-Driven Architecture",{"id":23,"title":24,"title_en":25},50,"用 Go + Gemini Embedding + pgvector 给 AI Agent 造一个\"会遗忘的大脑","Building a Scalable AI Agent Memory System with Golang, Gemini Embeddings, and pgvector","2026-06-26T21:00:20.601725+08:00"]