Strange Bug caused by Go cross-compilation caching: The development environment is normal, the production environment is abnormal

Problem Phenomenon

Recently, I encountered a very weird problem while developing a Go multi-language feature:

• ** Development environment **: The article list API normally returns English fields (title_en,abstract_en)
• ** Production environment **: The same API does not return English fields, but the fields exist and have data in the database

This kind of "Schrödinger's Bug" is crazy. The code is exactly the same. Why does it behave differently?

Troubleshooting Process

First round: Regular screening

The first suspicion is a common problem:

1. **JSON label problem **: Check the model definition, there is no omitent label
2. ** Database query **: Confirm that the production environment database fields exist and have data
3. ** Response structure **: Confirm that English fields are included
4. ** Cache problem **: Redis caching is suspected, but the problem still exists after clearing

Round 2: In-depth analysis

Check the relevant code logic:

//Asynchronous translation when article creation
go func(article models.ArticleModel) {
    if err := article_ser.ArticleWithTranslate(article.ID, article.Title, article.Abstract, article.Content); err != nil {
        global.Log.Error("Article translation failed", zap.Error(err))
    }
}(article)

// Translation service calls Tencent Cloud API
func ArticleWithTranslate(id uint, title, abstract, content string) error {
    // Call Tencent Cloud Translation API
    titleEn, err := translate.TranslateTexts(ctx, title, "auto", "en", 0)
    // ...
    // Update database
    return global.DB.Model(&models.ArticleModel{}).Where("id = ? ", id).Updates(map[string]any{
        "title_en":    titleEn,
        "abstract_en": abstractEn,
        "content_en":  contentEn,
    }).Error
}

It is suspected that it is a translation service problem, but there is no error message indicating translation failure in the production environment log.

Round 3: Configuration and Environment

Checked:

• Configuration file differences
• network access
• Tencent Cloud API configuration
• database connection

No problems were found.

The truth comes out

Finally, with the mentality of giving it a try, I implemented:

# Clear Go cache and recompile
go clean -cache -modcache
GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -trimpath -o deploy/xxxxx

** The problem was magically solved!**

Reason analysis

Problems caused by Go compilation caching, possible causes:

1. Inconsistent dependent versions

//version range in go.mod
require github.com/tencentcloud/tencentcloud-sdk-go v1.0.x

• A new version that fixes the bug was downloaded during development
• There is a problematic older version in the cross-compilation cache
• Tencent Cloud SDK may have translation API bugs in a certain version

2. cross-compilation cache pollution

# Possible execution orders
GOOS=linux GOARCH= amd64 go build #The problematic version is cached
# Modify code locally or update dependencies
go build                          #Local normal
GOOS=linux GOARCH= amd64 go build #Still using the old cross-compile cache

3. Module checksum issues

Checksums in the 'go.sum' file may not match, resulting in the use of the wrong version of the dependency.

Experience summary

Troubleshooting ideas

When encountering "the development environment is normal, the production environment is abnormal" and no obvious reason can be found:

1. Check for general issues (code, configuration, network)
2. Check application layer cache (Redis, memory cache)
3. ** Don't ignore the possibility of compilation **

Preventive measures

1. ** Lock dependent versions **

go mod tidy

2. ** Use defined build scripts **

#!/ bin/bash
go clean -cache -modcache
go mod download
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-s -w" -trimpath -o deploy/xxxx

3. ** Clean up cache regularly **

# Add cache cleanup steps to CI/CD
go clean -cache -modcache

Debugging skills

1. ** Add detailed log **

//Add logs to critical paths
global.Log.Info("Start asynchronous translation", zap.Uint("articleID", article.ID))
global.Log.Info("Translation Configuration", zap.String("region", global.Config.Translate.Region))

2. ** Version information record **

//Record dependent version information at startup
go list -m all

Conclusion

The weird thing about this bug is that it completely violates our common sense: how can the same code behave differently?

But reality tells us that in complex software systems, compilation caching, dependency management, cross-compilation and other aspects may become hiding places for bugs.

** Remember: When all regular troubleshooting fails, try clearing the compilation cache. Sometimes, the simplest and most crude method is the most effective. **

• Have you encountered similar problems? Welcome to share your experience of poth-breaking in the comment area!*