Taming LLM Outputs: How to Reliably Parse JSON in Golang

• Target Keywords: Golang LLM JSON parsing, OpenAI JSON mode Go, json.Decoder vs json.Unmarshal, robust LLM integration Go, parse LLM response Golang.
• Meta Description: Learn how to reliably parse JSON outputs from Large Language Models (LLMs) in Golang. Discover why json.Decoder beats json.Unmarshal, how to strip Markdown, and strategies for handling hallucinations and broken JSON structures.

When building applications that require structured output from Large Language Models (LLMs), you will inevitably encounter a fundamental truth:

LLMs are not databases; their output is probabilistic, not deterministic.

Even if your prompt explicitly states "Please return JSON format," the model might give you clean JSON, or it might wrap it in a Markdown code block, prepend conversational filler like "Here is the JSON you requested:", append a polite "Hope this helps!", or just return completely malformed JSON.

If your code lacks defensive mechanisms and relies solely on json.Unmarshal, it will crash at the first sign of dirty output, breaking your entire business workflow.

This article systematically breaks down a complete, production-ready solution for reliably parsing LLM JSON outputs in Golang—starting from basic prompt engineering to resilient parsing techniques.

1. Understanding the Enemy: How "Dirty" Can LLM Outputs Get?

Before designing a defense, we must understand the types of hallucinations and formatting errors we are dealing with.

Common Types of Dirty JSON Outputs

1. Markdown Code Block Wrappers

This is the most frequent issue, especially if your prompt includes code examples. The LLM will "politely" wrap its output:

Here is the JSON you requested:
```json
{"action": "move", "target_x": 10, "target_y": 5}

Let me know if you need anything else!

**2. Conversational Prefixes**

The model adds introductory text before the JSON:

```text
Based on the current scenario, I recommend the following action:
{"action": "talk", "speak": "Nice weather today!"}

3. Conversational Suffixes

Text appended after the JSON:

{"action": "stay", "target_x": 0, "target_y": 0}

Note: The coordinates indicate remaining stationary. Provide a target to move.

4. Multiple JSON Objects

The model outputs several JSON blocks, but only the first one is relevant:

Initial thought: {"action": "move", "target_x": 3}
Alternative plan: {"action": "stay", "target_x": 0}

5. Malformed JSON Structure

The hardest to handle: the structure itself is broken (e.g., trailing commas, unclosed quotes, or hallucinated property names).

{"action": "talk", "speak": "Hello there,}

Knowing these patterns allows us to build targeted parsing strategies.

2. The First Line of Defense: Constraining the Model

While robust parsing is essential, reducing the probability of dirty outputs at the source makes everything easier.

2.1 Enforce JSON Mode (Structured Outputs)

Modern LLM APIs often provide a "JSON Mode." In OpenAI-compatible APIs (which many providers use), this is controlled via response_format:

req := openai.ChatCompletionRequest{
    Model: model,
    ResponseFormat: &openai.ChatCompletionResponseFormat{
        Type: openai.ChatCompletionResponseFormatTypeJSONObject,
    },
    Messages: []openai.ChatCompletionMessage{
        {Role: openai.ChatMessageRoleUser, Content: prompt},
    },
}

Enabling JSON Mode forces the model to return a valid JSON object in most cases, eliminating conversational filler.

However, this is not a silver bullet.
First, not all providers support it. Second, even when supported, some models might still wrap the output in Markdown backticks (which is technically valid "JSON content" for the model, but invalid for standard JSON parsers). Finally, adherence varies by model version. JSON Mode is a critical first step, but not the final safeguard.

2.2 Prompt Engineering for Strict Constraints

Beyond API parameters, your prompt design is crucial. Use these techniques to minimize hallucinations:

Explicit Instructions:

You must output strictly in the following JSON format. Do not include any conversational text, explanations, or Markdown formatting.

{
  "action": "move | stay | talk",
  "speak": "Text to speak if action is 'talk', else empty string",
  "target_x": Target X coordinate (integer),
  "target_y": Target Y coordinate (integer)
}

Few-Shot Prompting (Provide Examples):
Providing a standard example JSON heavily influences the model to mimic the exact structure.

Example output (use for format reference only):
{"action": "move", "speak": "", "target_x": 5, "target_y": 3}

Lower the Temperature:
Temperature controls randomness. A high value makes the output "creative" but less compliant. For structural tasks, setting a low Temperature (e.g., 0.1 to 0.3) forces the model to strictly adhere to the requested format.

3. The Second Line of Defense: Resilient Parsing in Golang

Even with constraints, dirty outputs can slip through. This is where your Go code must be resilient.

3.1 The Naive Approach: Direct Unmarshal

Many developers default to this:

var result MyStruct
err := json.Unmarshal([]byte(respText), &result)
if err != nil {
    return nil, err
}

The fatal flaw here is that json.Unmarshal expects the entire byte slice to be a single, valid JSON value. Even a single leading space or a conversational prefix will cause it to panic or return an error.

3.2 The Pro Approach: Find { and Use json.Decoder

This is the battle-tested pattern we use in production. It is elegant and highly effective.

func parseJSONResponse(respText string, target any) error {
    // Step 1: Locate the first "{"
    // This strips away all conversational prefixes and markdown formatting
    start := strings.Index(respText, "{")
    if start == -1 {
        return fmt.Errorf("LLM response contains no JSON object: %s", respText)
    }

    // Step 2: Use json.Decoder instead of json.Unmarshal
    // Decoder treats the input as a stream and stops exactly after parsing the first valid JSON object.
    // Subsequent garbage (suffixes, extra JSON blocks) is simply ignored.
    decoder := json.NewDecoder(strings.NewReader(respText[start:]))
    if err := decoder.Decode(target); err != nil {
        return fmt.Errorf("JSON decode failed: %w, raw response: %s", err, respText)
    }

    return nil
}

Key Technique 1: strings.Index(respText, "{")
Finding the first { ignores conversational prefixes, Markdown tags like ````json`, and leading whitespace. With one line of code, you bypass 90% of prefix pollution.

Key Technique 2: json.Decoder vs json.Unmarshal
This is the secret sauce.
json.Unmarshal is a strict, full-payload parser.
json.Decoder is a forgiving, streaming parser. It reads the input stream, extracts the first valid JSON object, and stops.

• Conversational suffix? Ignored.
• Multiple JSON objects? Only the first is parsed.
• Trailing Markdown backticks? Safely ignored.

In short: Use json.Decoder for LLM outputs.

4. Handling JSON Arrays

If you expect the LLM to return an array ([...]), simply change the search target to [:

func parseJSONArrayResponse(respText string, target any) error {
    start := strings.Index(respText, "[")
    if start == -1 {
        return fmt.Errorf("LLM response contains no JSON array: %s", respText)
    }
    decoder := json.NewDecoder(strings.NewReader(respText[start:]))
    return decoder.Decode(target)
}

If the return type (Object vs. Array) is unpredictable, you can search for both and use the one that appears first.

5. Stripping Markdown Blocks (An Alternative Approach)

For models that stubbornly return Markdown code blocks, explicitly stripping them before parsing provides an extra layer of safety:

func stripMarkdownCodeBlock(s string) string {
    s = strings.TrimSpace(s)
    if strings.HasPrefix(s, "```") {
        // Find the end of the first line (e.g., after ```json)
        firstNewline := strings.Index(s, "\n")
        if firstNewline != -1 {
            s = s[firstNewline+1:]
        }
    }
    if strings.HasSuffix(s, "```") {
        s = s[:len(s)-3]
    }
    return strings.TrimSpace(s)
}

You can chain this with the previous method for dual protection:

cleanText := stripMarkdownCodeBlock(respText)
err := parseJSONResponse(cleanText, &result)

6. What If the JSON is Structurally Broken?

If the JSON itself is malformed (missing brackets, unescaped quotes), the parser will fail. You have three main strategies:

Strategy A: The Retry Loop
The most practical solution. If parsing fails, retry the LLM call. Models often correct themselves on the second try.

var result MyStruct
var lastErr error
for attempt := 0; attempt < 3; attempt++ {
    respText, err := callLLM(ctx, prompt)
    if err != nil {
        lastErr = err
        continue
    }
    if err = parseJSONResponse(respText, &result); err == nil {
        return &result, nil
    }
    lastErr = err
    time.Sleep(time.Duration(1<<attempt) * time.Second) // Exponential backoff
}
return nil, fmt.Errorf("failed after 3 attempts: %w", lastErr)

Strategy B: Feedback Loop to the LLM
Send the error back to the model and ask it to fix it:

Your previous output could not be parsed as JSON. Error: [Error Message]
Your output was: [Original Output]
Please output strictly valid JSON without conversational text.

Strategy C: Forgiving Parsers
You can use third-party Go libraries like json5 that tolerate trailing commas or unquoted keys. However, evaluate if the extra dependency is worth it for your use case.

7. The Production-Ready Workflow

Combining these techniques, a robust production workflow looks like this:

Call LLM API 
    ↓
Is JSON Mode enabled? (Yes, if supported)
    ↓
Strip Markdown wrappers (Optional)
    ↓
Find the first '{' or '['
    ↓
Parse using json.Decoder
    ↓
    ├── Success → Return struct
    └── Failure → Retry (with exponential backoff or LLM feedback)
                ↓
                Still Failing? → Log error, return default fallback state (Graceful Degradation)

Crucial Note on Graceful Degradation: If all retries fail, do not crash the application. Return a safe default value (e.g., an "idle" action for an agent) and log the raw output for debugging. Resilience > Perfection.

Summary

There is no silver bullet when dealing with probabilistic systems like LLMs. However, the combination of finding the first { and parsing with json.Decoder resolves over 90% of parsing errors with minimal code complexity.

Taming LLM outputs requires a defensive mindset. By combining strict prompt engineering with forgiving parsing logic, you can build Go applications that integrate LLMs reliably at scale.