Building a Log Agent from Scratch: A Single-File Go Program for Zero-Dependency Deployment

When starting out with servers, this is how most of us troubleshoot issues:

SSH into the server.
Run tail -f /var/log/app.log.
Stare at the terminal waiting for an error.
When something breaks, copy-paste the stack trace and DM it to a teammate.

But what if you have three servers? What if an alert fires at 3 AM while you are in bed?

There is a mature solution for this in the industry: Log Collection Agents. It is a lightweight program running on your server that watches your log files in real-time and pushes new entries to a central dashboard. All you need to do is open your browser.

In this guide, we will build a log agent from scratch in Go. The final output is a single binary executable (about 6MB) that you can drop onto any Linux server with zero external dependencies.

1. Plan the Architecture Before Coding

It's tempting to jump straight into writing code, only to realize halfway through that the design is flawed and needs a complete rewrite. Let's spend a few minutes planning the core problems our Agent needs to solve.

What does the Agent need to do?

① Watch the log file in real-time and capture new lines

Log files are append-only. The Agent needs to monitor the file and read new lines as they appear. This sounds simple, but here is a catch: if the Agent reads from the beginning of the file every time it starts up, it will re-send historical logs. Therefore, when the Agent starts, it should seek directly to the end of the file and only capture logs generated after its launch.

② Send logs in batches, not line-by-line

Suppose your server generates 1,000 log lines per second during peak hours. If the Agent sends an HTTP request for every single line, that is 1,000 network requests per second. Not only will the receiving platform struggle to handle the load, but you will also saturate your own server's network.

The correct approach is to buffer logs and send them in batches. But how many should we buffer? How long should we wait? There is a clever technique for this that we will cover later.

③ Don't lose logs when the network goes down

Networks are unreliable. If the network drops while the Agent is uploading logs, the request will fail. We cannot simply discard the logs. We need a robust retry mechanism.

With these three core challenges defined, let's visualize the data flow:

Copy

Log File (app.log)
       │
       │  Watch file and read new lines
       ▼
  ┌─────────────┐
  │   Tailer    │  Handles "Watching"
  │ (Collector) │
  └──────┬──────┘
         │  Write each line to an in-memory queue (channel)
         ▼
  ┌─────────────┐
  │   Batcher   │  Handles "Batching"
  │  (Buffer)   │  Trigger: 100 lines OR 500ms elapsed
  └──────┬──────┘
         │  Pack into a single JSON payload
         ▼
  ┌─────────────┐
  │   Sender    │  Handles "Sending"
  │ (Uploader)  │  HTTP POST + Retry with Exponential Backoff
  └──────┬──────┘
         │
         ▼
  Central Log Platform
  POST /api/v1/ingest

This three-layer architecture ensures strict separation of concerns. The Tailer doesn't care how logs are sent, and the Sender doesn't care how the file is read. The Batcher sits in the middle coordinating the rhythm. This decoupled design makes it easy to modify, optimize, and extend each layer independently.

2. Initialize the Project

The Agent is a stand-alone Go program with its own main function and directory structure, completely decoupled from the central platform.

bash Copy

# Create the directory structure
mkdir -p agent/tailer
mkdir -p agent/sender

# Initialize the go module
cd agent
go mod init logstream-agent

Your directory structure will look like this:

Copy

agent/
├── go.mod
├── go.sum
├── main.go          # Entrypoint: parses arguments, coordinates the three layers
├── tailer/
│   └── file.go      # Tailer: file watcher
└── sender/
    └── http.go      # Sender: HTTP uploader

3. Define the Data Structure

Before writing the logic for the three layers, let's define the core data structure—LogEntry—which represents a single log line captured from the file.

go Copy

// agent/main.go
package main

import "time"

// LogEntry represents a single captured log line
type LogEntry struct {
    Message   string    `json:"message"`   // The actual log content
    Source    string    `json:"source"`    // The source file path, e.g., /var/log/app.log
    Timestamp time.Time `json:"timestamp"` // Collection timestamp
}

Here is an important design choice: use the local collection time (time.Now()) from the Agent, not the timestamp written inside the log file.

Why?

Log timestamps come in all shapes and sizes: some write 2026-07-09 00:26:59, others write Jul 9 00:26:59, and some don't have timestamps at all. Parsing these formats requires a massive amount of fragile adapter code.

Using time.Now() is simple, reliable, and more than precise enough for most use cases (with an offset of just a few hundred milliseconds).

4. Tailer: Real-Time File Monitoring

How does `tail -f` work under the hood?

If you have used the tail -f app.log command, its underlying mechanics are straightforward:

Open the file and move the read pointer to the end.
Continuously attempt to read new content.
If there is no new content, sleep for a short duration (e.g., 100ms) and try again.

It is essentially a polling loop. Let's replicate this behavior in Go:

go Copy

// agent/tailer/file.go
package tailer

import (
    "bufio"
    "fmt"
    "os"
    "time"
)

// Tail watches the filePath and writes new log lines to the out channel.
// This function blocks indefinitely until an unrecoverable error occurs.
func Tail(filePath string, out chan<- string) error {
    f, err := os.Open(filePath)
    if err != nil {
        return fmt.Errorf("failed to open file: %w", err)
    }
    defer f.Close()

    // Key step: Seek to the end of the file
    // os.SEEK_END tells it to seek relative to the end, offset 0 means stay at the end.
    // This ensures we only capture new logs, skipping historical lines.
    if _, err := f.Seek(0, os.SEEK_END); err != nil {
        return fmt.Errorf("failed to seek to end of file: %w", err)
    }

    scanner := bufio.NewScanner(f)

    for {
        if scanner.Scan() {
            // Read a new line
            line := scanner.Text()

            // Skip empty lines (occasionally found in log files)
            if line != "" {
                out <- line // Send the line to the channel for the Batcher
            }
        } else {
            // scanner.Scan() returns false when there is no new content.
            // Sleep for 100ms to prevent high CPU utilization.
            time.Sleep(100 * time.Millisecond)
        }
    }
}

A few important details:

Why use bufio.Scanner instead of ioutil.ReadAll?

ioutil.ReadAll reads the entire file into memory at once. For an actively growing log file, this would consume massive amounts of memory and prevent real-time line-by-line streaming. bufio.Scanner reads line-by-line with a tiny, constant memory footprint.

What does chan<- string mean?

This is Go's syntax for a write-only channel. Inside the Tail function, we only send data into the channel (out <- line) and do not read from it. Declaring it this way allows the compiler to prevent accidental read operations, catching bugs early.

Handling Log Rotation

The code above works well until log rotation occurs—a common practice in production where tools like logrotate rename app.log to app.log.2026-07-09 and create a fresh app.log file.

If the Agent keeps reading from the old file descriptor, it will capture nothing new because the application is now writing to the newly created file.

The solution is to monitor the file's inode. In Linux filesystems, every file has a unique identifier called an inode. When a file is rotated, the new app.log will have a different inode than the old one.

We can poll for this change periodically:

go Copy

// agent/tailer/file.go (Complete version with Log Rotation support)
package tailer

import (
    "bufio"
    "fmt"
    "os"
    "syscall"
    "time"
)

// getInode returns the inode number of the file (unique ID in Linux filesystems)
func getInode(f *os.File) uint64 {
    info, err := f.Stat()
    if err != nil {
        return 0
    }
    // Cast os.FileInfo to the underlying syscall.Stat_t structure which holds the inode
    stat, ok := info.Sys().(*syscall.Stat_t)
    if !ok {
        return 0
    }
    return stat.Ino
}

func Tail(filePath string, out chan<- string) error {
    f, err := os.Open(filePath)
    if err != nil {
        return fmt.Errorf("failed to open file: %w", err)
    }
    defer f.Close()

    if _, err := f.Seek(0, os.SEEK_END); err != nil {
        return fmt.Errorf("failed to seek to end: %w", err)
    }

    currentInode := getInode(f)
    scanner := bufio.NewScanner(f)
    checkTicker := time.NewTicker(3 * time.Second) // Check for rotation every 3 seconds
    defer checkTicker.Stop()

    for {
        if scanner.Scan() {
            line := scanner.Text()
            if line != "" {
                out <- line
            }
        } else {
            // If there's no new content, check if the file was rotated
            select {
            case <-checkTicker.C:
                newF, err := os.Open(filePath)
                if err != nil {
                    // File might be temporarily missing during rotation, try again next time
                    break
                }
                newInode := getInode(newF)
                if newInode != currentInode {
                    fmt.Printf("[Tailer] Log rotation detected, switching to new file: %s\n", filePath)
                    f.Close()
                    f = newF
                    currentInode = newInode
                    scanner = bufio.NewScanner(f)
                } else {
                    newF.Close()
                }
            default:
                time.Sleep(100 * time.Millisecond)
            }
        }
    }
}

5. Batcher: Smart Buffer Management

The Batcher solves a classic engineering trade-off:

How do we balance "latency" and "throughput"?

Sending logs line-by-line offers the lowest latency, but generates too many network requests.
Waiting for 10,000 lines to build a batch offers high throughput, but users will experience significant delays before logs show up on the dashboard.

The industry-standard solution is a double-trigger mechanism:

Flush the buffer and send logs immediately if either of these conditions is met:

Condition A: The batch size reaches its limit (e.g., 100 lines).

Condition B: The time elapsed since the last flush exceeds a threshold (e.g., 500ms).

Condition A handles high-throughput periods, while Condition B maintains low latency during low-traffic periods.

Using Go's select statement combined with a time.Ticker makes this extremely elegant to implement:

go Copy

// agent/main.go - batchAndSend function
func batchAndSend(server, token, source string, lines <-chan string) {
    var batch []LogEntry

    // Ticker triggers a flush every 500ms
    ticker := time.NewTicker(500 * time.Millisecond)
    defer ticker.Stop()

    flush := func() {
        if len(batch) == 0 {
            return
        }

        // Send the batch (handled by the Sender layer, detailed below)
        sender.SendWithRetry(server, token, batch, 3)

        // Reset the slice capacity without deallocating memory.
        // batch = batch[:0] keeps the underlying array allocated, avoiding garbage collection overhead.
        batch = batch[:0]
    }

    for {
        select {
        case line := <-lines:
            // Event 1: New log line received from the Tailer
            batch = append(batch, LogEntry{
                Message:   line,
                Source:    source,
                Timestamp: time.Now(),
            })

            // Trigger A: Batch size limit reached
            if len(batch) >= 100 {
                flush()
            }

        case <-ticker.C:
            // Event 2: Ticker ticked (500ms passed)
            // Attempt to flush whatever is currently in the buffer
            flush()
        }
    }
}

Note: The select statement multiplexes channel operations. It blocks until one of its cases is ready, acting similarly to epoll at the language level.

6. Sender: HTTP Upload and Retry with Exponential Backoff

6.1 Basic Upload Logic

The Sender's task is simple: serialize the LogEntry batch into a JSON payload and HTTP POST it to the platform.

go Copy

// agent/sender/http.go
package sender

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "time"
)

type LogEntry struct {
    Message   string    `json:"message"`
    Source    string    `json:"source"`
    Timestamp time.Time `json:"timestamp"`
}

// Share a single, thread-safe client with a custom timeout
var httpClient = &http.Client{
    Timeout: 10 * time.Second,
}

// Send uploads a batch of logs to the server
func Send(serverAddr, token string, logs []LogEntry) error {
    // 1. Serialize logs into JSON {"logs": [...]}
    body, err := json.Marshal(map[string]any{
        "logs": logs,
    })
    if err != nil {
        return fmt.Errorf("JSON marshaling failed: %w", err)
    }

    // 2. Build the HTTP POST request
    req, err := http.NewRequest(
        "POST",
        serverAddr+"/api/v1/ingest",
        bytes.NewReader(body),
    )
    if err != nil {
        return fmt.Errorf("failed to build request: %w", err)
    }

    // 3. Set headers (X-Agent-Token identifies which Agent is uploading)
    req.Header.Set("X-Agent-Token", token)
    req.Header.Set("Content-Type", "application/json")

    // 4. Execute request
    resp, err := httpClient.Do(req)
    if err != nil {
        return fmt.Errorf("request execution failed: %w", err)
    }
    defer resp.Body.Close()

    // 5. Verify response code
    if resp.StatusCode != http.StatusOK {
        return fmt.Errorf("unexpected HTTP status code: %d", resp.StatusCode)
    }

    return nil
}

6.2 Retry Strategy: Why Exponential Backoff?

A naive retry strategy retries immediately after a failure.

If your central server goes down, hundreds of Agents retrying at maximum frequency will create a Thundering Herd problem. The moment your server tries to boot back up, it will immediately get overwhelmed by the wall of retry requests and crash again.

Exponential Backoff resolves this by scaling up the wait time after each subsequent failure:

Copy

Failure 1 → Wait 1s before retrying
Failure 2 → Wait 2s before retrying
Failure 3 → Wait 4s before retrying
Failure 4 → Wait 8s before retrying
...

This gives your server breathing room to recover while still handling transient network drops gracefully.

go Copy

// agent/sender/http.go (Continued)

// SendWithRetry sends logs with exponential backoff, up to maxRetries times
func SendWithRetry(serverAddr, token string, logs []LogEntry, maxRetries int) {
    for attempt := 0; attempt < maxRetries; attempt++ {
        err := Send(serverAddr, token, logs)
        if err == nil {
            return // Success
        }

        // Calculate backoff: 1s, 2s, 4s, 8s... using bit shift
        waitTime := time.Duration(1<<uint(attempt)) * time.Second

        fmt.Printf("[Sender] Attempt %d failed. Retrying in %v. Error: %v\n",
            attempt+1, waitTime, err)

        time.Sleep(waitTime)
    }

    // Maximum retries exceeded, logs are discarded
    // Production note: Here you could write logs to a local file/sqlite to retry later
    fmt.Printf("[Sender] Discarding %d logs after %d failed attempts\n", len(logs), maxRetries)
}

7. Connecting the Layers in Main

With the Tailer, Batcher, and Sender ready, our main function simply needs to parse arguments and connect the pipes.

go Copy

// agent/main.go (Complete version)
package main

import (
    "flag"
    "fmt"
    "logstream-agent/sender"
    "logstream-agent/tailer"
    "os"
    "time"
)

type LogEntry struct {
    Message   string    `json:"message"`
    Source    string    `json:"source"`
    Timestamp time.Time `json:"timestamp"`
}

func main() {
    serverAddr := flag.String("server", "", "Server address, e.g., http://127.0.0.1:8000")
    token      := flag.String("token", "", "Agent Token obtained from the server dashboard")
    filePath   := flag.String("file", "", "Path to the log file, e.g., /var/log/app.log")
    flag.Parse()

    if *serverAddr == "" || *token == "" || *filePath == "" {
        fmt.Println("Error: Missing required parameters!")
        fmt.Println()
        fmt.Println("Usage:")
        fmt.Println("  logstream-agent -server <address> -token <token> -file <path>")
        fmt.Println()
        fmt.Println("Example:")
        fmt.Println("  logstream-agent -server http://logs.example.com:8000 -token abc123 -file /var/log/app.log")
        os.Exit(1)
    }

    fmt.Printf("[Agent] Running successfully\n")
    fmt.Printf("[Agent] Monitoring: %s\n", *filePath)
    fmt.Printf("[Agent] Reporting to: %s\n", *serverAddr)

    // A buffered channel acts as the conveyor belt between Tailer and Batcher.
    // Setting capacity to 1000 allows the Tailer to keep reading even if the Batcher falls behind momentarily.
    lines := make(chan string, 1000)

    // Run the Tailer in a separate, non-blocking goroutine
    go func() {
        if err := tailer.Tail(*filePath, lines); err != nil {
            fmt.Printf("[Agent] Tailer exited unexpectedly: %v\n", err)
            os.Exit(1)
        }
    }()

    // Run the Batcher on the main goroutine (blocks execution)
    batchAndSend(*serverAddr, *token, *filePath, lines)
}

func batchAndSend(server, token, source string, lines <-chan string) {
    var batch []LogEntry
    ticker := time.NewTicker(500 * time.Millisecond)
    defer ticker.Stop()

    flush := func() {
        if len(batch) == 0 {
            return
        }
        // Deep copy batch data since SendWithRetry runs synchronously
        // and we will reset the slice capacity immediately after
        toSend := make([]sender.LogEntry, len(batch))
        for i, e := range batch {
            toSend[i] = sender.LogEntry{
                Message:   e.Message,
                Source:    e.Source,
                Timestamp: e.Timestamp,
            }
        }
        sender.SendWithRetry(server, token, toSend, 3)
        batch = batch[:0]
    }

    for {
        select {
        case line := <-lines:
            batch = append(batch, LogEntry{
                Message:   line,
                Source:    source,
                Timestamp: time.Now(),
            })
            if len(batch) >= 100 {
                flush()
            }
        case <-ticker.C:
            flush()
        }
    }
}

8. Compilation and Deployment

Cross-Compilation to Linux

One of Go's best features is its out-of-the-box support for cross-compilation—you can compile a binary for a Linux server right from your Mac or Windows development machine.

bash Copy

# Run this on your development machine to target 64-bit Linux servers
GOOS=linux GOARCH=amd64 go build -o logstream-agent .

# Inspect the binary size (Go packages everything into a single zero-dependency file!)
ls -lh logstream-agent
# -rwxr-xr-x  1 user staff  6.1M logstream-agent

What the environment variables mean:

GOOS=linux: Target OS is Linux.
GOARCH=amd64: Target CPU architecture is x86_64 (most cloud servers).

Note: If your server uses an ARM CPU (e.g., Raspberry Pi, Graviton instances), set GOARCH=arm64.

Deployment: Copy and Run

bash Copy

# 1. Copy the binary to your server
scp logstream-agent root@your-server-ip:/usr/local/bin/

# 2. SSH into your server
ssh root@your-server-ip

# 3. Grant execution permissions
chmod +x /usr/local/bin/logstream-agent

# 4. Dry run to verify it works (in the foreground)
logstream-agent \
  -server http://your-platform:8000 \
  -token your_agent_token \
  -file /var/log/nginx/access.log

Production Setup: Configure as a systemd Service

Running directly from the terminal is fine for testing, but closing the SSH session will terminate the Agent. For production, register it as a systemd service to handle auto-restart on crash and boot persistence.

bash Copy

# Create the systemd service file
cat > /etc/systemd/system/logstream-agent.service << 'EOF'
[Unit]
Description=LogStream Log Agent
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/logstream-agent \
  -server http://your-platform:8000 \
  -token your_agent_token \
  -file /var/log/app.log
Restart=on-failure
RestartSec=5s
User=root

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd configs
systemctl daemon-reload

# Enable boot autostart
systemctl enable logstream-agent

# Start the service now
systemctl start logstream-agent

# Verify it is running
systemctl status logstream-agent

If configured correctly, you should see an active status:

Copy

● logstream-agent.service - LogStream Log Agent
     Loaded: loaded (/etc/systemd/system/logstream-agent.service; enabled)
     Active: active (running) since Mon 2026-07-09 00:26:59 CST; 5s ago
   Main PID: 12345 (logstream-agent)

9. Verification

With everything deployed, let's verify that the pipeline works.

Simulate active logging on your server:

bash Copy

# Write test logs to the file once per second
while true; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') [INFO] User login successful user_id=12345" >> /var/log/app.log
    echo "$(date '+%Y-%m-%d %H:%M:%S') [ERROR] Database connection timed out db=mysql" >> /var/log/app.log
    sleep 1
done

Open your log platform dashboard. You should see these logs showing up in real-time with an approximate 500ms delay.

10. Conclusion and Next Steps

In about 200 lines of Go, we built a fully functional Log Agent:

Feature	Implementation
Real-time monitoring	`bufio.Scanner` + 100ms polling
Skip historical logs	`Seek` to end of file on startup
Log rotation support	Inode change detection
Batching buffer	`select` + `time.Ticker` double-trigger
Resiliency	Exponential backoff retry
Authentication	HTTP Header `X-Agent-Token`
Zero dependency	Go static compilation

For a production-ready system, you can extend this agent further:

📂 Multi-file monitoring: Track multiple log paths simultaneously (e.g., access.log and error.log).
🐳 Docker container logs: Capture stdout/stderr of running containers by tapping into the Docker daemon.
💾 Local disk buffer: Write logs to local storage (like SQLite or a ring buffer file) when the server goes down, flushing them once connection is restored.
🔍 Log level parser: Parse log contents using regex to automatically assign tags ([INFO], [ERROR], etc.).
📊 Monitoring metrics: Collect stats on "lines read / batches sent / failures" to monitor the health of the Agent itself.

Enjoy building your logging pipeline!

1. Plan the Architecture Before Coding

What does the Agent need to do?

2. Initialize the Project

3. Define the Data Structure

4. Tailer: Real-Time File Monitoring

How does tail -f work under the hood?

Handling Log Rotation

5. Batcher: Smart Buffer Management

6. Sender: HTTP Upload and Retry with Exponential Backoff

6.1 Basic Upload Logic

6.2 Retry Strategy: Why Exponential Backoff?

7. Connecting the Layers in Main

8. Compilation and Deployment

Cross-Compilation to Linux

Deployment: Copy and Run

Production Setup: Configure as a systemd Service

9. Verification

10. Conclusion and Next Steps

How does `tail -f` work under the hood?