[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-55":3},{"id":4,"title":5,"title_en":6,"abstract":7,"abstract_en":8,"content":9,"content_en":10,"category":11,"banner_id":12,"banner_path":13,"tags":14,"is_recommend":17,"prev_article":18,"next_article":22,"created_at":26},55," 从零实现一个 Log Agent：单文件部署的 Go 程序","Building a Log Agent from Scratch: A Single-File Go Program for Zero-Dependency Deployment\n","很多人刚开始搭服务器的时候，排查问题的方式是这样的：\n\n1. SSH 登上服务器\n2. `tail ","When starting out with servers, this is how most of us troubleshoot issues:","很多人刚开始搭服务器的时候，排查问题的方式是这样的：\n\n1. SSH 登上服务器\n2. `tail -f \u002Fvar\u002Flog\u002Fapp.log`\n3. 盯着屏幕等报错\n4. 出问题了，赶紧复制粘贴，发给同事\n\n要是服务器有三台呢？要是凌晨三点你在被窝里突然告警呢？\n\n其实这个问题在业界早就有成熟的解法——**日志采集 Agent**。它是一个跑在你服务器上的轻量小程序，负责实时读取日志文件，然后把日志推送到一个中央平台，你只需要打开浏览器就能看到所有服务器的日志。\n\n这篇文章就来教你从零开始，用 Go 语言实现一个这样的 Agent。**最终产物是一个单文件二进制程序，编译后约 6MB，扔到任何 Linux 服务器上就能跑，不需要安装任何依赖。**\n\n---\n\n## 一、先把问题想清楚，再动手写代码\n\n很多人喜欢上来就开始写代码，结果写到一半发现设计有问题，推倒重来。所以我们先花几分钟把整个 Agent 需要解决的问题梳理一遍。\n\n### Agent 需要做哪些事？\n\n**① 实时监听日志文件，感知新内容**\n\n日志文件是不断追加内容的，Agent 需要一直盯着文件，有新的一行就立刻读出来。听起来简单，但有个细节：如果每次都从文件头开始读，那历史日志会被重复发送。所以 Agent 启动的时候，应该**直接跳到文件末尾**，只采集\"我启动之后新产生的日志\"。\n\n**② 批量上报，而不是来一条发一条**\n\n假设你的服务器在高峰期每秒产生 1000 行日志，如果每行都发一次 HTTP 请求，那就是每秒 1000 次网络请求。别说平台扛不住，你自己服务器的网络都会被打爆。\n\n正确的做法是把日志**攒一批**再一起发。但是攒多少？攒多久？这里有个小技巧，后面详细讲。\n\n**③ 网络断了，日志不能丢**\n\n这是很多人没想到的场景。如果 Agent 往平台发日志的时候，刚好网络抖动了一下，这批日志就会发送失败。直接丢掉显然不行，我们需要**失败重试**机制。\n\n好，三个核心问题清楚了。我们把整个数据流画出来：\n\n```\n日志文件 (app.log)\n       │\n       │  一直盯着文件，有新行就读出来\n       ▼\n  ┌─────────────┐\n  │   Tailer    │  负责\"监听\"\n  │  (采集层)   │\n  └──────┬──────┘\n         │  把每一行写入一个内存队列（channel）\n         ▼\n  ┌─────────────┐\n  │   Batcher   │  负责\"攒批\"\n  │  (缓冲层)   │  积攒 100 条 OR 超过 500ms\n  └──────┬──────┘\n         │  打包成一个 JSON 请求体\n         ▼\n  ┌─────────────┐\n  │   Sender    │  负责\"发送\"\n  │  (上报层)   │  HTTP POST + 失败重试\n  └──────┬──────┘\n         │\n         ▼\n   平台的接收接口\n   POST \u002Fapi\u002Fv1\u002Fingest\n```\n\n三层结构，**职责完全分离**。Tailer 不关心怎么发送，Sender 不关心怎么读文件，Batcher 在中间负责协调节奏。这样的设计，每一层都可以独立修改和优化，后续扩展也很方便。\n\n---\n\n## 二、初始化项目\n\nAgent 是一个**独立的 Go 程序**，和你的后端平台完全分开，有自己的 `main` 函数和目录结构。\n\n```bash\n# 创建目录结构\nmkdir -p agent\u002Ftailer\nmkdir -p agent\u002Fsender\n\n# 初始化 go module（agent 是独立编译的，有自己的 module）\ncd agent\ngo mod init logstream-agent\n```\n\n最终的目录结构长这样：\n\n```\nagent\u002F\n├── go.mod\n├── go.sum\n├── main.go          # 程序入口：解析参数、串联三层\n├── tailer\u002F\n│   └── file.go      # Tailer：文件监听\n└── sender\u002F\n    └── http.go      # Sender：HTTP 上报\n```\n\n---\n\n## 三、定义数据结构\n\n在开始写三层逻辑之前，先定义一个贯穿全局的数据结构——`LogEntry`，它代表从文件里采集到的**一条日志**。\n\n```go\n\u002F\u002F agent\u002Fmain.go\npackage main\n\nimport \"time\"\n\n\u002F\u002F LogEntry 代表一条采集到的日志\ntype LogEntry struct {\n    Message   string    `json:\"message\"`   \u002F\u002F 日志的原文内容\n    Source    string    `json:\"source\"`    \u002F\u002F 来自哪个文件，例如 \u002Fvar\u002Flog\u002Fapp.log\n    Timestamp time.Time `json:\"timestamp\"` \u002F\u002F 采集时间\n}\n```\n\n这里有一个值得解释的设计决策：**时间戳用服务端（Agent 本机）的 `time.Now()`，而不是日志文件里的时间**。\n\n为什么？\n\n日志文件里的时间格式五花八门：有的写 `2026-07-09 00:26:59`，有的写 `Jul 9 00:26:59`，有的甚至没有时间，只有一段文字。解析这些格式需要写大量的适配代码，而且并不可靠。\n\n用 Agent 本机的 `time.Now()` 虽然不是日志\"产生\"的精确时间，但它是日志被\"采集\"的时间，在大多数业务场景下已经足够精准（误差在几百毫秒之内）。\n\n---\n\n## 四、Tailer：实现文件实时监听\n\n### 原理：`tail -f` 是怎么工作的？\n\n你一定用过 `tail -f app.log` 这个命令。它的原理其实非常简单：\n\n1. 打开文件，把读取指针移到文件末尾\n2. 不断尝试读取新内容\n3. 如果没有新内容，等一小会儿（比如 100ms），再继续尝试\n\n就是个死循环里的轮询，没有什么神奇之处。我们用 Go 来复现这个行为：\n\n```go\n\u002F\u002F agent\u002Ftailer\u002Ffile.go\npackage tailer\n\nimport (\n    \"bufio\"\n    \"fmt\"\n    \"os\"\n    \"time\"\n)\n\n\u002F\u002F Tail 持续监听 filePath 文件，将每一行新日志写入 out channel\n\u002F\u002F 这个函数会一直阻塞运行，直到发生不可恢复的错误\nfunc Tail(filePath string, out chan\u003C- string) error {\n    f, err := os.Open(filePath)\n    if err != nil {\n        return fmt.Errorf(\"打开文件失败: %w\", err)\n    }\n    defer f.Close()\n\n    \u002F\u002F 重点：把文件的读取指针移到末尾\n    \u002F\u002F os.SEEK_END 表示\"从文件末尾开始\"，偏移量 0 表示\"就在末尾\"\n    \u002F\u002F 这样 Agent 启动后，只会读取新产生的日志，不会把历史日志全部重发\n    if _, err := f.Seek(0, os.SEEK_END); err != nil {\n        return fmt.Errorf(\"定位文件末尾失败: %w\", err)\n    }\n\n    scanner := bufio.NewScanner(f)\n\n    for {\n        if scanner.Scan() {\n            \u002F\u002F 读到了新的一行\n            line := scanner.Text()\n\n            \u002F\u002F 跳过空行（日志文件里偶尔会有空行）\n            if line != \"\" {\n                out \u003C- line \u002F\u002F 把这行发送到 channel，等待 Batcher 处理\n            }\n        } else {\n            \u002F\u002F scanner.Scan() 返回 false，说明当前没有新内容\n            \u002F\u002F 休息 100ms 后继续轮询，避免 CPU 空转\n            time.Sleep(100 * time.Millisecond)\n        }\n    }\n}\n```\n\n几个细节说明：\n\n**为什么用 `bufio.Scanner` 而不是 `ioutil.ReadAll`？**\n\n`ioutil.ReadAll` 会把整个文件内容一次性读入内存，对于一直在增长的日志文件来说，这会占用大量内存，而且没法做到\"读一行发一行\"。`bufio.Scanner` 是按行读取的，内存占用极低，非常适合这个场景。\n\n**`chan\u003C- string` 是什么意思？**\n\n这是 Go 的语法，表示这是一个**只写的 channel**。在 `Tail` 函数里，我们只往 channel 里发数据（`out \u003C- line`），不从里面读。用 `chan\u003C- string` 而不是 `chan string`，是为了让编译器帮你检查：防止你不小心在这个函数里反向读取 channel，出现逻辑错误。\n\n### 处理日志轮转（Log Rotation）\n\n上面的代码在大多数情况下都能正常工作，但有一个生产环境里很常见的问题：**日志轮转（Log Rotation）**。\n\n很多服务器会配置 logrotate 工具，它会在每天凌晨把 `app.log` 重命名为 `app.log.2026-07-09`，然后创建一个新的空 `app.log` 文件，供程序继续写入。\n\n如果 Agent 还在用旧的文件描述符读取已经被重命名的那个文件，它会发现\"永远没有新内容了\"，但实际上新的日志已经在写入新的 `app.log` 了。\n\n解决方案是**检测文件的 inode 是否发生变化**。在 Linux 文件系统中，每个文件都有一个唯一的编号叫 inode。当 `app.log` 被重命名为 `app.log.old`，然后创建新的 `app.log` 时，新文件的 inode 会和旧文件不同。\n\n我们可以定期检测这个变化：\n\n```go\n\u002F\u002F agent\u002Ftailer\u002Ffile.go（完整版，支持日志轮转检测）\npackage tailer\n\nimport (\n    \"bufio\"\n    \"fmt\"\n    \"os\"\n    \"syscall\"\n    \"time\"\n)\n\n\u002F\u002F getInode 获取文件的 inode 编号（Linux 文件系统的唯一标识）\nfunc getInode(f *os.File) uint64 {\n    info, err := f.Stat()\n    if err != nil {\n        return 0\n    }\n    \u002F\u002F 将 os.FileInfo 转换为底层的 syscall.Stat_t 结构，里面有 inode 信息\n    stat, ok := info.Sys().(*syscall.Stat_t)\n    if !ok {\n        return 0\n    }\n    return stat.Ino\n}\n\nfunc Tail(filePath string, out chan\u003C- string) error {\n    f, err := os.Open(filePath)\n    if err != nil {\n        return fmt.Errorf(\"打开文件失败: %w\", err)\n    }\n    defer f.Close()\n\n    if _, err := f.Seek(0, os.SEEK_END); err != nil {\n        return fmt.Errorf(\"定位文件末尾失败: %w\", err)\n    }\n\n    currentInode := getInode(f) \u002F\u002F 记录当前文件的 inode\n    scanner := bufio.NewScanner(f)\n    checkTicker := time.NewTicker(3 * time.Second) \u002F\u002F 每 3 秒检查一次是否发生了日志轮转\n    defer checkTicker.Stop()\n\n    for {\n        if scanner.Scan() {\n            line := scanner.Text()\n            if line != \"\" {\n                out \u003C- line\n            }\n        } else {\n            \u002F\u002F 没有新内容，趁这个空隙检查一下文件是否被轮转了\n            select {\n            case \u003C-checkTicker.C:\n                \u002F\u002F 尝试打开同名文件，看看它的 inode 是否和当前的一样\n                newF, err := os.Open(filePath)\n                if err != nil {\n                    \u002F\u002F 文件暂时不存在（轮转间隙），等一下再试\n                    break\n                }\n                newInode := getInode(newF)\n                if newInode != currentInode {\n                    \u002F\u002F inode 不同，说明文件被轮转了！\n                    fmt.Printf(\"[Tailer] 检测到日志轮转，切换到新文件: %s\\n\", filePath)\n                    f.Close()\n                    f = newF\n                    currentInode = newInode\n                    scanner = bufio.NewScanner(f)\n                    \u002F\u002F 新文件从头开始读（轮转后的新文件是空的，没有历史数据）\n                } else {\n                    newF.Close()\n                }\n            default:\n                \u002F\u002F 不到检查时间，先休眠\n                time.Sleep(100 * time.Millisecond)\n            }\n        }\n    }\n}\n```\n\n---\n\n## 五、Batcher：实现智能批量缓冲\n\nBatcher 是整个 Agent 里**最有意思**的一层，它需要解决一个很经典的工程问题：\n\n> **怎样在\"实时性\"和\"吞吐量\"之间找到平衡？**\n\n- 如果每来一行日志就立刻发送：实时性最好，但网络请求太频繁，高并发下会出问题。\n- 如果积攒 10000 行再发送：吞吐量很好，但日志可能要等几分钟才能在平台上看到，失去了\"实时\"的意义。\n\n业界通用的解法是**双触发机制**：\n\n> 只要满足以下**任意一个条件**，立刻发送当前积攒的所有日志：\n> - 条件 A：积攒的日志条数达到了上限（比如 100 条）\n> - 条件 B：距离上次发送已经超过了一段时间（比如 500ms）\n\n条件 A 保证高峰期的吞吐量，条件 B 保证低谷期的实时性。这个设计在 Kafka、Logstash 等成熟的日志系统中都有类似的应用。\n\n用 Go 的 `select` + `time.Ticker` 来实现这个逻辑，非常优雅：\n\n```go\n\u002F\u002F agent\u002Fmain.go 中的 batchAndSend 函数\nfunc batchAndSend(server, token, source string, lines \u003C-chan string) {\n    \u002F\u002F 用来存储当前积攒的日志\n    var batch []LogEntry\n\n    \u002F\u002F 定时器：每 500ms 触发一次\n    ticker := time.NewTicker(500 * time.Millisecond)\n    defer ticker.Stop()\n\n    \u002F\u002F flush 函数：把当前 batch 里的所有日志发送出去，然后清空 batch\n    flush := func() {\n        if len(batch) == 0 {\n            return \u002F\u002F 如果 batch 是空的，什么都不做\n        }\n\n        \u002F\u002F 发送这批日志（这里调用 Sender 层，后面会实现）\n        sender.SendWithRetry(server, token, batch, 3)\n\n        \u002F\u002F 清空 batch，注意这里不是 batch = nil，而是 batch = batch[:0]\n        \u002F\u002F 后者可以复用底层数组的内存，避免 GC 频繁分配和回收内存\n        batch = batch[:0]\n    }\n\n    \u002F\u002F 主循环：通过 select 同时监听两个事件\n    for {\n        select {\n\n        case line := \u003C-lines:\n            \u002F\u002F 事件一：Tailer 发来了一行新日志\n            batch = append(batch, LogEntry{\n                Message:   line,\n                Source:    source,\n                Timestamp: time.Now(),\n            })\n\n            \u002F\u002F 检查是否触发\"数量上限\"条件\n            if len(batch) >= 100 {\n                flush()\n            }\n\n        case \u003C-ticker.C:\n            \u002F\u002F 事件二：定时器到了 500ms\n            \u002F\u002F 不管 batch 里有没有数据，尝试 flush 一次\n            flush()\n        }\n    }\n}\n```\n\n这里有一个 Go 并发编程的小技巧值得说一下：**`select` 是 Go 里处理多个 channel 事件最常用的模式**。它会同时监听 `case` 里的所有 channel，哪个先有数据就执行哪个，类似于操作系统里的 `epoll`。\n\n---\n\n## 六、Sender：HTTP 上报与失败重试\n\n### 6.1 基础发送逻辑\n\nSender 的核心任务很简单：把一批 `LogEntry` 打包成 JSON，通过 HTTP POST 发送到平台的接收接口。\n\n```go\n\u002F\u002F agent\u002Fsender\u002Fhttp.go\npackage sender\n\nimport (\n    \"bytes\"\n    \"encoding\u002Fjson\"\n    \"fmt\"\n    \"net\u002Fhttp\"\n    \"time\"\n)\n\n\u002F\u002F LogEntry 和 main.go 里的一样（实际开发中可以提取到公共包）\ntype LogEntry struct {\n    Message   string    `json:\"message\"`\n    Source    string    `json:\"source\"`\n    Timestamp time.Time `json:\"timestamp\"`\n}\n\n\u002F\u002F 全局共用一个 http.Client，并设置超时时间\n\u002F\u002F 注意：http.Client 是线程安全的，可以复用\nvar httpClient = &http.Client{\n    Timeout: 10 * time.Second,\n}\n\n\u002F\u002F Send 将一批日志发送到平台\nfunc Send(serverAddr, token string, logs []LogEntry) error {\n    \u002F\u002F 第一步：把日志列表序列化成 JSON\n    \u002F\u002F 接口接受的格式是 {\"logs\": [...]}\n    body, err := json.Marshal(map[string]any{\n        \"logs\": logs,\n    })\n    if err != nil {\n        \u002F\u002F json.Marshal 极少失败，除非数据里有不可序列化的类型\n        return fmt.Errorf(\"JSON 序列化失败: %w\", err)\n    }\n\n    \u002F\u002F 第二步：构建 HTTP POST 请求\n    req, err := http.NewRequest(\n        \"POST\",\n        serverAddr+\"\u002Fapi\u002Fv1\u002Fingest\",\n        bytes.NewReader(body), \u002F\u002F 把 JSON 字节数组作为请求体\n    )\n    if err != nil {\n        return fmt.Errorf(\"构建 HTTP 请求失败: %w\", err)\n    }\n\n    \u002F\u002F 第三步：设置请求头\n    \u002F\u002F X-Agent-Token 是身份验证凭证，平台通过它识别是哪个 Agent 在上报\n    req.Header.Set(\"X-Agent-Token\", token)\n    req.Header.Set(\"Content-Type\", \"application\u002Fjson\")\n\n    \u002F\u002F 第四步：发送请求\n    resp, err := httpClient.Do(req)\n    if err != nil {\n        \u002F\u002F 网络故障（超时、连接拒绝等）\n        return fmt.Errorf(\"发送请求失败: %w\", err)\n    }\n    defer resp.Body.Close()\n\n    \u002F\u002F 第五步：检查响应状态码\n    if resp.StatusCode != http.StatusOK {\n        return fmt.Errorf(\"平台返回异常状态码: %d\", resp.StatusCode)\n    }\n\n    return nil\n}\n```\n\n### 6.2 失败重试：为什么需要\"退避\"？\n\n天真的重试策略是：失败了就立刻再试，还失败就再试……\n\n这个策略有个严重的问题：如果服务器宕机了，大量 Agent 同时以极高的频率发起重试，会在服务器恢复的瞬间造成\"**雪崩**\"——服务器刚喘口气，就被一波请求又打倒了。\n\n正确的做法是**指数退避（Exponential Backoff）**：每次重试之间的等待时间**指数级增长**。\n\n```\n第 1 次失败 → 等待 1 秒后重试\n第 2 次失败 → 等待 2 秒后重试\n第 3 次失败 → 等待 4 秒后重试\n第 4 次失败 → 等待 8 秒后重试\n...\n```\n\n这样，既保证了短暂的网络抖动能快速恢复，又避免了长时间故障时的\"重试风暴\"。\n\n```go\n\u002F\u002F agent\u002Fsender\u002Fhttp.go（续）\n\n\u002F\u002F SendWithRetry 带指数退避的发送，最多重试 maxRetries 次\nfunc SendWithRetry(serverAddr, token string, logs []LogEntry, maxRetries int) {\n    for attempt := 0; attempt \u003C maxRetries; attempt++ {\n        err := Send(serverAddr, token, logs)\n        if err == nil {\n            \u002F\u002F 发送成功，直接返回\n            return\n        }\n\n        \u002F\u002F 计算下次重试的等待时间：1s, 2s, 4s, 8s...\n        \u002F\u002F 用位移运算：1 \u003C\u003C 0 = 1, 1 \u003C\u003C 1 = 2, 1 \u003C\u003C 2 = 4\n        waitTime := time.Duration(1\u003C\u003Cuint(attempt)) * time.Second\n\n        fmt.Printf(\"[Sender] 第 %d 次发送失败，将在 %v 后重试。错误: %v\\n\",\n            attempt+1, waitTime, err)\n\n        time.Sleep(waitTime)\n    }\n\n    \u002F\u002F 超过最大重试次数，这批日志只能丢弃了\n    \u002F\u002F TODO: 生产环境中，这里可以把日志写入本地临时文件，网络恢复后补发\n    fmt.Printf(\"[Sender] 已重试 %d 次，放弃发送 %d 条日志\\n\", maxRetries, len(logs))\n}\n```\n\n---\n\n## 七、主程序：把三层串联起来\n\n有了 Tailer、Batcher 和 Sender，主程序的工作就是**解析命令行参数**，然后把三层串起来。\n\n```go\n\u002F\u002F agent\u002Fmain.go（完整版）\npackage main\n\nimport (\n    \"flag\"\n    \"fmt\"\n    \"logstream-agent\u002Fsender\"\n    \"logstream-agent\u002Ftailer\"\n    \"os\"\n    \"time\"\n)\n\ntype LogEntry struct {\n    Message   string    `json:\"message\"`\n    Source    string    `json:\"source\"`\n    Timestamp time.Time `json:\"timestamp\"`\n}\n\nfunc main() {\n    \u002F\u002F 定义命令行参数\n    serverAddr := flag.String(\"server\", \"\", \"平台地址，例如: http:\u002F\u002F127.0.0.1:8000\")\n    token      := flag.String(\"token\", \"\", \"Agent Token，在平台创建 Agent 时获取\")\n    filePath   := flag.String(\"file\", \"\", \"监听的日志文件路径，例如: \u002Fvar\u002Flog\u002Fapp.log\")\n    flag.Parse()\n\n    \u002F\u002F 校验必填参数\n    if *serverAddr == \"\" || *token == \"\" || *filePath == \"\" {\n        fmt.Println(\"缺少必要参数！\")\n        fmt.Println()\n        fmt.Println(\"使用方式:\")\n        fmt.Println(\"  logstream-agent -server \u003C平台地址> -token \u003CAgent Token> -file \u003C日志文件路径>\")\n        fmt.Println()\n        fmt.Println(\"示例:\")\n        fmt.Println(\"  logstream-agent -server http:\u002F\u002Flogs.example.com:8000 -token abc123 -file \u002Fvar\u002Flog\u002Fapp.log\")\n        os.Exit(1)\n    }\n\n    fmt.Printf(\"[Agent] 启动成功\\n\")\n    fmt.Printf(\"[Agent] 监听文件: %s\\n\", *filePath)\n    fmt.Printf(\"[Agent] 上报目标: %s\\n\", *serverAddr)\n\n    \u002F\u002F 创建一个带缓冲的 channel，作为 Tailer 和 Batcher 之间的\"传送带\"\n    \u002F\u002F 缓冲大小设为 1000：当 Batcher 来不及处理时，Tailer 最多可以先积压 1000 行\n    \u002F\u002F 超过 1000 行后，Tailer 会阻塞等待，而不是无限制地占用内存\n    lines := make(chan string, 1000)\n\n    \u002F\u002F 在独立的 goroutine 中启动 Tailer\n    \u002F\u002F Tailer 会一直阻塞运行，所以必须放在 goroutine 里\n    go func() {\n        if err := tailer.Tail(*filePath, lines); err != nil {\n            fmt.Printf(\"[Agent] Tailer 异常退出: %v\\n\", err)\n            os.Exit(1)\n        }\n    }()\n\n    \u002F\u002F 主 goroutine 负责 Batcher（阻塞运行，程序不会退出）\n    batchAndSend(*serverAddr, *token, *filePath, lines)\n}\n\nfunc batchAndSend(server, token, source string, lines \u003C-chan string) {\n    var batch []LogEntry\n    ticker := time.NewTicker(500 * time.Millisecond)\n    defer ticker.Stop()\n\n    flush := func() {\n        if len(batch) == 0 {\n            return\n        }\n        \u002F\u002F 注意：这里需要复制一份 batch 的内容，因为 SendWithRetry 是同步的\n        \u002F\u002F 如果直接传 batch，在 SendWithRetry 运行期间，batch 可能被清空\n        toSend := make([]sender.LogEntry, len(batch))\n        for i, e := range batch {\n            toSend[i] = sender.LogEntry{\n                Message:   e.Message,\n                Source:    e.Source,\n                Timestamp: e.Timestamp,\n            }\n        }\n        sender.SendWithRetry(server, token, toSend, 3)\n        batch = batch[:0]\n    }\n\n    for {\n        select {\n        case line := \u003C-lines:\n            batch = append(batch, LogEntry{\n                Message:   line,\n                Source:    source,\n                Timestamp: time.Now(),\n            })\n            if len(batch) >= 100 {\n                flush()\n            }\n        case \u003C-ticker.C:\n            flush()\n        }\n    }\n}\n```\n\n---\n\n## 八、编译与部署\n\n### 编译：交叉编译到 Linux\n\nGo 最强大的特性之一就是**交叉编译**——你可以在 Mac 或 Windows 上，直接编译出能在 Linux 服务器上运行的二进制文件。\n\n```bash\n# 在你的开发机上执行，编译出 Linux 64位 的可执行文件\nGOOS=linux GOARCH=amd64 go build -o logstream-agent .\n\n# 查看文件大小（Go 编译出的二进制包含所有依赖，无需额外安装）\nls -lh logstream-agent\n# -rwxr-xr-x  1 user staff  6.1M logstream-agent\n```\n\n两个环境变量的含义：\n- `GOOS=linux`：目标操作系统是 Linux\n- `GOARCH=amd64`：目标 CPU 架构是 x86_64（绝大多数服务器的架构）\n\n如果你的服务器是 ARM 架构（比如树莓派、部分云服务器），把 `amd64` 改成 `arm64` 即可。\n\n### 部署：传到服务器，直接跑\n\n```bash\n# 把编译好的文件传到服务器\nscp logstream-agent root@your-server-ip:\u002Fusr\u002Flocal\u002Fbin\u002F\n\n# SSH 到服务器\nssh root@your-server-ip\n\n# 给文件赋予执行权限\nchmod +x \u002Fusr\u002Flocal\u002Fbin\u002Flogstream-agent\n\n# 测试运行（前台跑，先看看有没有问题）\nlogstream-agent \\\n  -server http:\u002F\u002Fyour-platform:8000 \\\n  -token 你的AgentToken \\\n  -file \u002Fvar\u002Flog\u002Fnginx\u002Faccess.log\n```\n\n### 生产部署：配置为 systemd 服务\n\n直接在命令行跑有个问题——你一关 SSH 会话，Agent 就停了。生产环境里，我们应该把它配置成系统服务，**开机自启、崩溃自动重启**。\n\n```bash\n# 创建 systemd 服务文件\ncat > \u002Fetc\u002Fsystemd\u002Fsystem\u002Flogstream-agent.service \u003C\u003C 'EOF'\n[Unit]\nDescription=LogStream Log Agent\n# 等网络就绪后再启动\nAfter=network.target\n\n[Service]\nType=simple\n# 启动命令\nExecStart=\u002Fusr\u002Flocal\u002Fbin\u002Flogstream-agent \\\n  -server http:\u002F\u002Fyour-platform:8000 \\\n  -token 你的AgentToken \\\n  -file \u002Fvar\u002Flog\u002Fapp.log\n# 异常退出后，等 5 秒自动重启\nRestart=on-failure\nRestartSec=5s\n# 以 root 用户运行（如果日志文件权限需要的话）\nUser=root\n\n[Install]\nWantedBy=multi-user.target\nEOF\n\n# 重新加载 systemd 配置\nsystemctl daemon-reload\n\n# 设置开机自启\nsystemctl enable logstream-agent\n\n# 立刻启动\nsystemctl start logstream-agent\n\n# 查看运行状态\nsystemctl status logstream-agent\n```\n\n如果一切正常，你会看到类似这样的输出：\n\n```\n● logstream-agent.service - LogStream Log Agent\n     Loaded: loaded (\u002Fetc\u002Fsystemd\u002Fsystem\u002Flogstream-agent.service; enabled)\n     Active: active (running) since Mon 2026-07-09 00:26:59 CST; 5s ago\n   Main PID: 12345 (logstream-agent)\n```\n\n---\n\n## 九、测试验证\n\n全部部署好之后，我们来验证一下整个链路是否通畅。\n\n**在服务器上模拟写入日志：**\n\n```bash\n# 每秒写一行日志到文件\nwhile true; do\n    echo \"$(date '+%Y-%m-%d %H:%M:%S') [INFO] 用户登录成功 user_id=12345\" >> \u002Fvar\u002Flog\u002Fapp.log\n    echo \"$(date '+%Y-%m-%d %H:%M:%S') [ERROR] 数据库连接超时 db=mysql\" >> \u002Fvar\u002Flog\u002Fapp.log\n    sleep 1\ndone\n```\n\n打开你的日志平台 Dashboard，应该能看到这些日志以大约 **500ms 的延迟**实时出现在界面上。\n\n---\n\n## 十、总结与后续扩展\n\n我们用大约 200 行 Go 代码，从零实现了一个具备以下能力的 Log Agent：\n\n| 功能 | 实现方式 |\n|------|----------|\n| 实时文件监听 | `bufio.Scanner` + 100ms 轮询 |\n| 跳过历史日志 | 启动时 `Seek` 到文件末尾 |\n| 日志轮转感知 | inode 变更检测 |\n| 批量上报 | `select` + `time.Ticker` 双触发 |\n| 失败重试 | 指数退避，最多 N 次 |\n| 身份验证 | HTTP Header `X-Agent-Token` |\n| 单文件部署 | Go 静态编译，零依赖 |\n\n这只是一个 MVP 版本，后续还有很多可以扩展的方向：\n\n- 📂 **多文件监听**：同时监听多个日志文件，例如 access.log 和 error.log\n- 🐳 **Docker 日志支持**：通过 Docker API 采集容器的标准输出\n- 💾 **本地磁盘缓冲**：网络长时间断开时，把日志暂存到本地文件，网络恢复后补发，实现真正的零丢失\n- 🔍 **日志级别识别**：通过正则自动解析日志里的 `[ERROR]`、`[WARN]`、`[INFO]` 标签\n- 📊 **上报指标**：统计并定期打印\"已采集行数 \u002F 已发送批次 \u002F 发送失败次数\"等运行指标\n\n希望这篇文章对你有帮助！如果有任何问题，欢迎在评论区讨论。\n","When starting out with servers, this is how most of us troubleshoot issues:\n\n1. SSH into the server.\n2. Run `tail -f \u002Fvar\u002Flog\u002Fapp.log`.\n3. Stare at the terminal waiting for an error.\n4. When something breaks, copy-paste the stack trace and DM it to a teammate.\n\nBut what if you have three servers? What if an alert fires at 3 AM while you are in bed?\n\nThere is a mature solution for this in the industry: **Log Collection Agents**. It is a lightweight program running on your server that watches your log files in real-time and pushes new entries to a central dashboard. All you need to do is open your browser.\n\nIn this guide, we will build a log agent from scratch in Go. **The final output is a single binary executable (about 6MB) that you can drop onto any Linux server with zero external dependencies.**\n\n---\n\n## 1. Plan the Architecture Before Coding\n\nIt's tempting to jump straight into writing code, only to realize halfway through that the design is flawed and needs a complete rewrite. Let's spend a few minutes planning the core problems our Agent needs to solve.\n\n### What does the Agent need to do?\n\n**① Watch the log file in real-time and capture new lines**\n\nLog files are append-only. The Agent needs to monitor the file and read new lines as they appear. This sounds simple, but here is a catch: if the Agent reads from the beginning of the file every time it starts up, it will re-send historical logs. Therefore, when the Agent starts, it should **seek directly to the end of the file** and only capture logs generated after its launch.\n\n**② Send logs in batches, not line-by-line**\n\nSuppose your server generates 1,000 log lines per second during peak hours. If the Agent sends an HTTP request for every single line, that is 1,000 network requests per second. Not only will the receiving platform struggle to handle the load, but you will also saturate your own server's network.\n\nThe correct approach is to **buffer logs and send them in batches**. But how many should we buffer? How long should we wait? There is a clever technique for this that we will cover later.\n\n**③ Don't lose logs when the network goes down**\n\nNetworks are unreliable. If the network drops while the Agent is uploading logs, the request will fail. We cannot simply discard the logs. We need a robust **retry mechanism**.\n\nWith these three core challenges defined, let's visualize the data flow:\n\n```\nLog File (app.log)\n       │\n       │  Watch file and read new lines\n       ▼\n  ┌─────────────┐\n  │   Tailer    │  Handles \"Watching\"\n  │ (Collector) │\n  └──────┬──────┘\n         │  Write each line to an in-memory queue (channel)\n         ▼\n  ┌─────────────┐\n  │   Batcher   │  Handles \"Batching\"\n  │  (Buffer)   │  Trigger: 100 lines OR 500ms elapsed\n  └──────┬──────┘\n         │  Pack into a single JSON payload\n         ▼\n  ┌─────────────┐\n  │   Sender    │  Handles \"Sending\"\n  │ (Uploader)  │  HTTP POST + Retry with Exponential Backoff\n  └──────┬──────┘\n         │\n         ▼\n  Central Log Platform\n  POST \u002Fapi\u002Fv1\u002Fingest\n```\n\nThis three-layer architecture ensures **strict separation of concerns**. The Tailer doesn't care how logs are sent, and the Sender doesn't care how the file is read. The Batcher sits in the middle coordinating the rhythm. This decoupled design makes it easy to modify, optimize, and extend each layer independently.\n\n---\n\n## 2. Initialize the Project\n\nThe Agent is a **stand-alone Go program** with its own `main` function and directory structure, completely decoupled from the central platform.\n\n```bash\n# Create the directory structure\nmkdir -p agent\u002Ftailer\nmkdir -p agent\u002Fsender\n\n# Initialize the go module\ncd agent\ngo mod init logstream-agent\n```\n\nYour directory structure will look like this:\n\n```\nagent\u002F\n├── go.mod\n├── go.sum\n├── main.go          # Entrypoint: parses arguments, coordinates the three layers\n├── tailer\u002F\n│   └── file.go      # Tailer: file watcher\n└── sender\u002F\n    └── http.go      # Sender: HTTP uploader\n```\n\n---\n\n## 3. Define the Data Structure\n\nBefore writing the logic for the three layers, let's define the core data structure—`LogEntry`—which represents a **single log line** captured from the file.\n\n```go\n\u002F\u002F agent\u002Fmain.go\npackage main\n\nimport \"time\"\n\n\u002F\u002F LogEntry represents a single captured log line\ntype LogEntry struct {\n    Message   string    `json:\"message\"`   \u002F\u002F The actual log content\n    Source    string    `json:\"source\"`    \u002F\u002F The source file path, e.g., \u002Fvar\u002Flog\u002Fapp.log\n    Timestamp time.Time `json:\"timestamp\"` \u002F\u002F Collection timestamp\n}\n```\n\nHere is an important design choice: **use the local collection time (`time.Now()`) from the Agent, not the timestamp written inside the log file**.\n\nWhy?\n\nLog timestamps come in all shapes and sizes: some write `2026-07-09 00:26:59`, others write `Jul 9 00:26:59`, and some don't have timestamps at all. Parsing these formats requires a massive amount of fragile adapter code.\n\nUsing `time.Now()` is simple, reliable, and more than precise enough for most use cases (with an offset of just a few hundred milliseconds).\n\n---\n\n## 4. Tailer: Real-Time File Monitoring\n\n### How does `tail -f` work under the hood?\n\nIf you have used the `tail -f app.log` command, its underlying mechanics are straightforward:\n\n1. Open the file and move the read pointer to the end.\n2. Continuously attempt to read new content.\n3. If there is no new content, sleep for a short duration (e.g., 100ms) and try again.\n\nIt is essentially a polling loop. Let's replicate this behavior in Go:\n\n```go\n\u002F\u002F agent\u002Ftailer\u002Ffile.go\npackage tailer\n\nimport (\n    \"bufio\"\n    \"fmt\"\n    \"os\"\n    \"time\"\n)\n\n\u002F\u002F Tail watches the filePath and writes new log lines to the out channel.\n\u002F\u002F This function blocks indefinitely until an unrecoverable error occurs.\nfunc Tail(filePath string, out chan\u003C- string) error {\n    f, err := os.Open(filePath)\n    if err != nil {\n        return fmt.Errorf(\"failed to open file: %w\", err)\n    }\n    defer f.Close()\n\n    \u002F\u002F Key step: Seek to the end of the file\n    \u002F\u002F os.SEEK_END tells it to seek relative to the end, offset 0 means stay at the end.\n    \u002F\u002F This ensures we only capture new logs, skipping historical lines.\n    if _, err := f.Seek(0, os.SEEK_END); err != nil {\n        return fmt.Errorf(\"failed to seek to end of file: %w\", err)\n    }\n\n    scanner := bufio.NewScanner(f)\n\n    for {\n        if scanner.Scan() {\n            \u002F\u002F Read a new line\n            line := scanner.Text()\n\n            \u002F\u002F Skip empty lines (occasionally found in log files)\n            if line != \"\" {\n                out \u003C- line \u002F\u002F Send the line to the channel for the Batcher\n            }\n        } else {\n            \u002F\u002F scanner.Scan() returns false when there is no new content.\n            \u002F\u002F Sleep for 100ms to prevent high CPU utilization.\n            time.Sleep(100 * time.Millisecond)\n        }\n    }\n}\n```\n\nA few important details:\n\n**Why use `bufio.Scanner` instead of `ioutil.ReadAll`?**\n\n`ioutil.ReadAll` reads the entire file into memory at once. For an actively growing log file, this would consume massive amounts of memory and prevent real-time line-by-line streaming. `bufio.Scanner` reads line-by-line with a tiny, constant memory footprint.\n\n**What does `chan\u003C- string` mean?**\n\nThis is Go's syntax for a **write-only channel**. Inside the `Tail` function, we only send data into the channel (`out \u003C- line`) and do not read from it. Declaring it this way allows the compiler to prevent accidental read operations, catching bugs early.\n\n### Handling Log Rotation\n\nThe code above works well until **log rotation** occurs—a common practice in production where tools like `logrotate` rename `app.log` to `app.log.2026-07-09` and create a fresh `app.log` file.\n\nIf the Agent keeps reading from the old file descriptor, it will capture nothing new because the application is now writing to the newly created file.\n\nThe solution is to **monitor the file's inode**. In Linux filesystems, every file has a unique identifier called an inode. When a file is rotated, the new `app.log` will have a different inode than the old one.\n\nWe can poll for this change periodically:\n\n```go\n\u002F\u002F agent\u002Ftailer\u002Ffile.go (Complete version with Log Rotation support)\npackage tailer\n\nimport (\n    \"bufio\"\n    \"fmt\"\n    \"os\"\n    \"syscall\"\n    \"time\"\n)\n\n\u002F\u002F getInode returns the inode number of the file (unique ID in Linux filesystems)\nfunc getInode(f *os.File) uint64 {\n    info, err := f.Stat()\n    if err != nil {\n        return 0\n    }\n    \u002F\u002F Cast os.FileInfo to the underlying syscall.Stat_t structure which holds the inode\n    stat, ok := info.Sys().(*syscall.Stat_t)\n    if !ok {\n        return 0\n    }\n    return stat.Ino\n}\n\nfunc Tail(filePath string, out chan\u003C- string) error {\n    f, err := os.Open(filePath)\n    if err != nil {\n        return fmt.Errorf(\"failed to open file: %w\", err)\n    }\n    defer f.Close()\n\n    if _, err := f.Seek(0, os.SEEK_END); err != nil {\n        return fmt.Errorf(\"failed to seek to end: %w\", err)\n    }\n\n    currentInode := getInode(f)\n    scanner := bufio.NewScanner(f)\n    checkTicker := time.NewTicker(3 * time.Second) \u002F\u002F Check for rotation every 3 seconds\n    defer checkTicker.Stop()\n\n    for {\n        if scanner.Scan() {\n            line := scanner.Text()\n            if line != \"\" {\n                out \u003C- line\n            }\n        } else {\n            \u002F\u002F If there's no new content, check if the file was rotated\n            select {\n            case \u003C-checkTicker.C:\n                newF, err := os.Open(filePath)\n                if err != nil {\n                    \u002F\u002F File might be temporarily missing during rotation, try again next time\n                    break\n                }\n                newInode := getInode(newF)\n                if newInode != currentInode {\n                    fmt.Printf(\"[Tailer] Log rotation detected, switching to new file: %s\\n\", filePath)\n                    f.Close()\n                    f = newF\n                    currentInode = newInode\n                    scanner = bufio.NewScanner(f)\n                } else {\n                    newF.Close()\n                }\n            default:\n                time.Sleep(100 * time.Millisecond)\n            }\n        }\n    }\n}\n```\n\n---\n\n## 5. Batcher: Smart Buffer Management\n\nThe Batcher solves a classic engineering trade-off:\n\n> **How do we balance \"latency\" and \"throughput\"?**\n\n- Sending logs line-by-line offers the lowest latency, but generates too many network requests.\n- Waiting for 10,000 lines to build a batch offers high throughput, but users will experience significant delays before logs show up on the dashboard.\n\nThe industry-standard solution is a **double-trigger mechanism**:\n\n> Flush the buffer and send logs immediately if **either** of these conditions is met:\n> - **Condition A**: The batch size reaches its limit (e.g., 100 lines).\n> - **Condition B**: The time elapsed since the last flush exceeds a threshold (e.g., 500ms).\n\nCondition A handles high-throughput periods, while Condition B maintains low latency during low-traffic periods.\n\nUsing Go's `select` statement combined with a `time.Ticker` makes this extremely elegant to implement:\n\n```go\n\u002F\u002F agent\u002Fmain.go - batchAndSend function\nfunc batchAndSend(server, token, source string, lines \u003C-chan string) {\n    var batch []LogEntry\n\n    \u002F\u002F Ticker triggers a flush every 500ms\n    ticker := time.NewTicker(500 * time.Millisecond)\n    defer ticker.Stop()\n\n    flush := func() {\n        if len(batch) == 0 {\n            return\n        }\n\n        \u002F\u002F Send the batch (handled by the Sender layer, detailed below)\n        sender.SendWithRetry(server, token, batch, 3)\n\n        \u002F\u002F Reset the slice capacity without deallocating memory.\n        \u002F\u002F batch = batch[:0] keeps the underlying array allocated, avoiding garbage collection overhead.\n        batch = batch[:0]\n    }\n\n    for {\n        select {\n        case line := \u003C-lines:\n            \u002F\u002F Event 1: New log line received from the Tailer\n            batch = append(batch, LogEntry{\n                Message:   line,\n                Source:    source,\n                Timestamp: time.Now(),\n            })\n\n            \u002F\u002F Trigger A: Batch size limit reached\n            if len(batch) >= 100 {\n                flush()\n            }\n\n        case \u003C-ticker.C:\n            \u002F\u002F Event 2: Ticker ticked (500ms passed)\n            \u002F\u002F Attempt to flush whatever is currently in the buffer\n            flush()\n        }\n    }\n}\n```\n\n*Note: The `select` statement multiplexes channel operations. It blocks until one of its cases is ready, acting similarly to `epoll` at the language level.*\n\n---\n\n## 6. Sender: HTTP Upload and Retry with Exponential Backoff\n\n### 6.1 Basic Upload Logic\n\nThe Sender's task is simple: serialize the `LogEntry` batch into a JSON payload and HTTP POST it to the platform.\n\n```go\n\u002F\u002F agent\u002Fsender\u002Fhttp.go\npackage sender\n\nimport (\n    \"bytes\"\n    \"encoding\u002Fjson\"\n    \"fmt\"\n    \"net\u002Fhttp\"\n    \"time\"\n)\n\ntype LogEntry struct {\n    Message   string    `json:\"message\"`\n    Source    string    `json:\"source\"`\n    Timestamp time.Time `json:\"timestamp\"`\n}\n\n\u002F\u002F Share a single, thread-safe client with a custom timeout\nvar httpClient = &http.Client{\n    Timeout: 10 * time.Second,\n}\n\n\u002F\u002F Send uploads a batch of logs to the server\nfunc Send(serverAddr, token string, logs []LogEntry) error {\n    \u002F\u002F 1. Serialize logs into JSON {\"logs\": [...]}\n    body, err := json.Marshal(map[string]any{\n        \"logs\": logs,\n    })\n    if err != nil {\n        return fmt.Errorf(\"JSON marshaling failed: %w\", err)\n    }\n\n    \u002F\u002F 2. Build the HTTP POST request\n    req, err := http.NewRequest(\n        \"POST\",\n        serverAddr+\"\u002Fapi\u002Fv1\u002Fingest\",\n        bytes.NewReader(body),\n    )\n    if err != nil {\n        return fmt.Errorf(\"failed to build request: %w\", err)\n    }\n\n    \u002F\u002F 3. Set headers (X-Agent-Token identifies which Agent is uploading)\n    req.Header.Set(\"X-Agent-Token\", token)\n    req.Header.Set(\"Content-Type\", \"application\u002Fjson\")\n\n    \u002F\u002F 4. Execute request\n    resp, err := httpClient.Do(req)\n    if err != nil {\n        return fmt.Errorf(\"request execution failed: %w\", err)\n    }\n    defer resp.Body.Close()\n\n    \u002F\u002F 5. Verify response code\n    if resp.StatusCode != http.StatusOK {\n        return fmt.Errorf(\"unexpected HTTP status code: %d\", resp.StatusCode)\n    }\n\n    return nil\n}\n```\n\n### 6.2 Retry Strategy: Why Exponential Backoff?\n\nA naive retry strategy retries immediately after a failure.\n\nIf your central server goes down, hundreds of Agents retrying at maximum frequency will create a **Thundering Herd** problem. The moment your server tries to boot back up, it will immediately get overwhelmed by the wall of retry requests and crash again.\n\n**Exponential Backoff** resolves this by scaling up the wait time after each subsequent failure:\n\n```\nFailure 1 → Wait 1s before retrying\nFailure 2 → Wait 2s before retrying\nFailure 3 → Wait 4s before retrying\nFailure 4 → Wait 8s before retrying\n...\n```\n\nThis gives your server breathing room to recover while still handling transient network drops gracefully.\n\n```go\n\u002F\u002F agent\u002Fsender\u002Fhttp.go (Continued)\n\n\u002F\u002F SendWithRetry sends logs with exponential backoff, up to maxRetries times\nfunc SendWithRetry(serverAddr, token string, logs []LogEntry, maxRetries int) {\n    for attempt := 0; attempt \u003C maxRetries; attempt++ {\n        err := Send(serverAddr, token, logs)\n        if err == nil {\n            return \u002F\u002F Success\n        }\n\n        \u002F\u002F Calculate backoff: 1s, 2s, 4s, 8s... using bit shift\n        waitTime := time.Duration(1\u003C\u003Cuint(attempt)) * time.Second\n\n        fmt.Printf(\"[Sender] Attempt %d failed. Retrying in %v. Error: %v\\n\",\n            attempt+1, waitTime, err)\n\n        time.Sleep(waitTime)\n    }\n\n    \u002F\u002F Maximum retries exceeded, logs are discarded\n    \u002F\u002F Production note: Here you could write logs to a local file\u002Fsqlite to retry later\n    fmt.Printf(\"[Sender] Discarding %d logs after %d failed attempts\\n\", len(logs), maxRetries)\n}\n```\n\n---\n\n## 7. Connecting the Layers in Main\n\nWith the Tailer, Batcher, and Sender ready, our `main` function simply needs to parse arguments and connect the pipes.\n\n```go\n\u002F\u002F agent\u002Fmain.go (Complete version)\npackage main\n\nimport (\n    \"flag\"\n    \"fmt\"\n    \"logstream-agent\u002Fsender\"\n    \"logstream-agent\u002Ftailer\"\n    \"os\"\n    \"time\"\n)\n\ntype LogEntry struct {\n    Message   string    `json:\"message\"`\n    Source    string    `json:\"source\"`\n    Timestamp time.Time `json:\"timestamp\"`\n}\n\nfunc main() {\n    serverAddr := flag.String(\"server\", \"\", \"Server address, e.g., http:\u002F\u002F127.0.0.1:8000\")\n    token      := flag.String(\"token\", \"\", \"Agent Token obtained from the server dashboard\")\n    filePath   := flag.String(\"file\", \"\", \"Path to the log file, e.g., \u002Fvar\u002Flog\u002Fapp.log\")\n    flag.Parse()\n\n    if *serverAddr == \"\" || *token == \"\" || *filePath == \"\" {\n        fmt.Println(\"Error: Missing required parameters!\")\n        fmt.Println()\n        fmt.Println(\"Usage:\")\n        fmt.Println(\"  logstream-agent -server \u003Caddress> -token \u003Ctoken> -file \u003Cpath>\")\n        fmt.Println()\n        fmt.Println(\"Example:\")\n        fmt.Println(\"  logstream-agent -server http:\u002F\u002Flogs.example.com:8000 -token abc123 -file \u002Fvar\u002Flog\u002Fapp.log\")\n        os.Exit(1)\n    }\n\n    fmt.Printf(\"[Agent] Running successfully\\n\")\n    fmt.Printf(\"[Agent] Monitoring: %s\\n\", *filePath)\n    fmt.Printf(\"[Agent] Reporting to: %s\\n\", *serverAddr)\n\n    \u002F\u002F A buffered channel acts as the conveyor belt between Tailer and Batcher.\n    \u002F\u002F Setting capacity to 1000 allows the Tailer to keep reading even if the Batcher falls behind momentarily.\n    lines := make(chan string, 1000)\n\n    \u002F\u002F Run the Tailer in a separate, non-blocking goroutine\n    go func() {\n        if err := tailer.Tail(*filePath, lines); err != nil {\n            fmt.Printf(\"[Agent] Tailer exited unexpectedly: %v\\n\", err)\n            os.Exit(1)\n        }\n    }()\n\n    \u002F\u002F Run the Batcher on the main goroutine (blocks execution)\n    batchAndSend(*serverAddr, *token, *filePath, lines)\n}\n\nfunc batchAndSend(server, token, source string, lines \u003C-chan string) {\n    var batch []LogEntry\n    ticker := time.NewTicker(500 * time.Millisecond)\n    defer ticker.Stop()\n\n    flush := func() {\n        if len(batch) == 0 {\n            return\n        }\n        \u002F\u002F Deep copy batch data since SendWithRetry runs synchronously\n        \u002F\u002F and we will reset the slice capacity immediately after\n        toSend := make([]sender.LogEntry, len(batch))\n        for i, e := range batch {\n            toSend[i] = sender.LogEntry{\n                Message:   e.Message,\n                Source:    e.Source,\n                Timestamp: e.Timestamp,\n            }\n        }\n        sender.SendWithRetry(server, token, toSend, 3)\n        batch = batch[:0]\n    }\n\n    for {\n        select {\n        case line := \u003C-lines:\n            batch = append(batch, LogEntry{\n                Message:   line,\n                Source:    source,\n                Timestamp: time.Now(),\n            })\n            if len(batch) >= 100 {\n                flush()\n            }\n        case \u003C-ticker.C:\n            flush()\n        }\n    }\n}\n```\n\n---\n\n## 8. Compilation and Deployment\n\n### Cross-Compilation to Linux\n\nOne of Go's best features is its out-of-the-box support for **cross-compilation**—you can compile a binary for a Linux server right from your Mac or Windows development machine.\n\n```bash\n# Run this on your development machine to target 64-bit Linux servers\nGOOS=linux GOARCH=amd64 go build -o logstream-agent .\n\n# Inspect the binary size (Go packages everything into a single zero-dependency file!)\nls -lh logstream-agent\n# -rwxr-xr-x  1 user staff  6.1M logstream-agent\n```\n\nWhat the environment variables mean:\n- `GOOS=linux`: Target OS is Linux.\n- `GOARCH=amd64`: Target CPU architecture is x86_64 (most cloud servers).\n\n*Note: If your server uses an ARM CPU (e.g., Raspberry Pi, Graviton instances), set `GOARCH=arm64`.*\n\n### Deployment: Copy and Run\n\n```bash\n# 1. Copy the binary to your server\nscp logstream-agent root@your-server-ip:\u002Fusr\u002Flocal\u002Fbin\u002F\n\n# 2. SSH into your server\nssh root@your-server-ip\n\n# 3. Grant execution permissions\nchmod +x \u002Fusr\u002Flocal\u002Fbin\u002Flogstream-agent\n\n# 4. Dry run to verify it works (in the foreground)\nlogstream-agent \\\n  -server http:\u002F\u002Fyour-platform:8000 \\\n  -token your_agent_token \\\n  -file \u002Fvar\u002Flog\u002Fnginx\u002Faccess.log\n```\n\n### Production Setup: Configure as a systemd Service\n\nRunning directly from the terminal is fine for testing, but closing the SSH session will terminate the Agent. For production, register it as a systemd service to handle **auto-restart on crash** and **boot persistence**.\n\n```bash\n# Create the systemd service file\ncat > \u002Fetc\u002Fsystemd\u002Fsystem\u002Flogstream-agent.service \u003C\u003C 'EOF'\n[Unit]\nDescription=LogStream Log Agent\nAfter=network.target\n\n[Service]\nType=simple\nExecStart=\u002Fusr\u002Flocal\u002Fbin\u002Flogstream-agent \\\n  -server http:\u002F\u002Fyour-platform:8000 \\\n  -token your_agent_token \\\n  -file \u002Fvar\u002Flog\u002Fapp.log\nRestart=on-failure\nRestartSec=5s\nUser=root\n\n[Install]\nWantedBy=multi-user.target\nEOF\n\n# Reload systemd configs\nsystemctl daemon-reload\n\n# Enable boot autostart\nsystemctl enable logstream-agent\n\n# Start the service now\nsystemctl start logstream-agent\n\n# Verify it is running\nsystemctl status logstream-agent\n```\n\nIf configured correctly, you should see an active status:\n\n```\n● logstream-agent.service - LogStream Log Agent\n     Loaded: loaded (\u002Fetc\u002Fsystemd\u002Fsystem\u002Flogstream-agent.service; enabled)\n     Active: active (running) since Mon 2026-07-09 00:26:59 CST; 5s ago\n   Main PID: 12345 (logstream-agent)\n```\n\n---\n\n## 9. Verification\n\nWith everything deployed, let's verify that the pipeline works.\n\n**Simulate active logging on your server:**\n\n```bash\n# Write test logs to the file once per second\nwhile true; do\n    echo \"$(date '+%Y-%m-%d %H:%M:%S') [INFO] User login successful user_id=12345\" >> \u002Fvar\u002Flog\u002Fapp.log\n    echo \"$(date '+%Y-%m-%d %H:%M:%S') [ERROR] Database connection timed out db=mysql\" >> \u002Fvar\u002Flog\u002Fapp.log\n    sleep 1\ndone\n```\n\nOpen your log platform dashboard. You should see these logs showing up in real-time with an approximate **500ms delay**.\n\n---\n\n## 10. Conclusion and Next Steps\n\nIn about 200 lines of Go, we built a fully functional Log Agent:\n\n| Feature | Implementation |\n|------|----------|\n| Real-time monitoring | `bufio.Scanner` + 100ms polling |\n| Skip historical logs | `Seek` to end of file on startup |\n| Log rotation support | Inode change detection |\n| Batching buffer | `select` + `time.Ticker` double-trigger |\n| Resiliency | Exponential backoff retry |\n| Authentication | HTTP Header `X-Agent-Token` |\n| Zero dependency | Go static compilation |\n\nFor a production-ready system, you can extend this agent further:\n- 📂 **Multi-file monitoring**: Track multiple log paths simultaneously (e.g., access.log and error.log).\n- 🐳 **Docker container logs**: Capture stdout\u002Fstderr of running containers by tapping into the Docker daemon.\n- 💾 **Local disk buffer**: Write logs to local storage (like SQLite or a ring buffer file) when the server goes down, flushing them once connection is restored.\n- 🔍 **Log level parser**: Parse log contents using regex to automatically assign tags (`[INFO]`, `[ERROR]`, etc.).\n- 📊 **Monitoring metrics**: Collect stats on \"lines read \u002F batches sent \u002F failures\" to monitor the health of the Agent itself.\n\nEnjoy building your logging pipeline!\n","Go",37,"https:\u002F\u002Fblog4-1316398321.cos.ap-nanjing.myqcloud.com\u002Fblog5\u002F20260623050851__古建筑.png",[15,16],"GO","Agent",false,{"id":19,"title":20,"title_en":21},54,"# 构建高可维护的 Go Web 脚手架：从反射型 ORM 到图引擎 Ent 的架构演进","#Building a highly maintainable Go Web scaffold: The architectural evolution from reflective ORM to graph engine Ent",{"id":23,"title":24,"title_en":25},56,"用 Go 从零搭建 WebTransport 实时推送服务","Building a WebTransport Real-Time Push Service from Scratch in Go","2026-07-09T00:35:47.852523+08:00"]