Go + Elasticsearch enables intelligent search

🎯 Project background

In modern content platforms, the search function is the core entrance for users to obtain information. While traditional database LIKE queries degrade dramatically when faced with massive amounts of data, Elasticsearch, a full-text search engine, provides millisecond search responses. This article will detail how to use Go + Elasticsearch to build a fully functional intelligent search system.

✨ System Characteristics

🚀 High-performance search

Full-text search: Supports multi-field search for title, content, and abstract
Weight Sorting: Title weight 3x, Content weight 2x, Summary weight 1x
Highlighting: Automatically highlight keywords for search keywords
Millisecond Response: Average response time < 50ms

🎯 Smart Recommendation

Tag Similarity: Calculates article similarity based on the Jaccard coefficient
User behavior: Likes and favorites affect the weight of recommendations
Time Decay: New articles receive higher recommendation weight
Cache Optimization: Redis caches recommended results for improved performance

🔄 Data synchronization

Real-time synchronization: Article CRUD operations are automatically synced to ES
Batch Operations: Supports batch deletion and updates
Error Retry: Automatic retry mechanism for network exceptions
Scheduled Full Volume: Scheduled tasks ensure data consistency

🏗️ System Architecture

Core Components

graph TD A[Web API] --> B[Service] B --> C[Elasticsearch] A --> D[MySQL] B --> E[Redis] C --> F [Timed Mission]

Data Flow

Write Flow: API → MySQL → ES synchronization
Search Flow: API → ES query → result processing
Recommendation Process: API → Similarity Calculation → Redis Cache

💡 Core implementation

1. Intelligent search query building

Go Copy

func BuildSearchQuery(keyword, sortField, sortOrder, from, size string, tags []string, category string) *bytes. Buffer {
    var queryBody map[string]interface{}
    
    if keyword == "" {
        Full query
        queryBody = map[string]interface{}{
            "query": map[string]interface{}{
                "match_all": map[string]interface{}{},
            },
        }
    } else {
        Multi-field weight search
        queryBody = map[string]interface{}{
            "query": map[string]interface{}{
                "multi_match": map[string]interface{}{
                    "query":  keyword,
                    "fields": []string{"title^3", "content^2", "abstract"},
                    "type":   "best_fields",
                },
            },
        }
    }

    Build a full query
    query := map[string]interface{}{
        "query": queryBody["query"],
        "highlight": map[string]interface{}{
            "pre_tags":  []string{"<em class='highlight'>"},
            "post_tags": []string{"</em>"},
            "fields": map[string]interface{}{
                "title":   map[string]interface{}{},
                "content": map[string]interface{}{},
            },
        },
        "from": from,
        "size": size,
    }

    Add filters
    if category != "" || len(tags) > 0 {
        filters := []map[string]interface{}{}
        
        if category != "" {
            filters = append(filters, map[string]interface{}{
                "term": map[string]interface{}{"category": category},
            })
        }
        
        if len(tags) > 0 {
            filters = append(filters, map[string]interface{}{
                "terms": map[string]interface{}{"tags": tags},
            })
        }

        query["query"] = map[string]interface{}{
            "bool": map[string]interface{}{
                "must":   queryBody["query"],
                "filter": filters,
            },
        }
    }

    var buf bytes. Buffer
    json. NewEncoder(&buf). Encode(query)
    return &buf
}

2. Data synchronization

Go Copy

func SyncToES(article models. ArticleModel) error {
    if !global. Config.Elasticsearch.Enable {
        return nil
    }

    Structure the document
    doc := map[string]any{
        "title":          article. Title,
        "content":        article. Content,
        "abstract":       article. Abstract,
        "tags":           article. Tags,
        "category":       article. Category,
        "created_at":     article. CreatedAt.Format("2006-01-02 15:04:05"),
        "digg_count":     article. DiggCount,
        "look_count":     article. LookCount,
        "collects_count": article. CollectsCount,
        "comment_count":  article. CommentCount,
    }

    docData, err := json. Marshal(doc)
    if err != nil {
        return err
    }

    req := esapi. IndexRequest{
        Index:      "articles",
        DocumentID: fmt. Sprintf("%d", article.ID),
        Body:       strings. NewReader(string(docData)),
        Refresh: "wait_for", // Immediately searchable
    }

    Execution with retries
    It is better to add a circuit breaker mechanism here, if the server is down and overloaded, retrying will make the server worse
    return retry. DoWithRetry(3, 1*time. Second, func() error {
        res, err := req. Do(context. Background(), global. Elasticsearch)
        if err != nil {
            return err
        }
        defer res. Body.Close()

        if res. IsError() {
            body, _ := io. ReadAll(res. Body)
            return fmt. Errorf("ES operation failed: %s | %s", res. Status(), string(body))
        }
        return nil
    })
}

3. Intelligent recommendation algorithm

Go Copy

func GetRecommendations(articleID uint, userID uint, limit int) ([]ArticleSimilarity, error) {
    Cache check
    cacheKey := fmt. Sprintf("article_recommend:%d:%d:%d", articleID, userID, limit)
    if val, err := global. Redis.Get(context. Background(), cacheKey). Result(); err == nil {
        var cachedData []ArticleSimilarity
        if err := json. Unmarshal([]byte(val), &cachedData); err == nil {
            return cachedData, nil
        }
    }

    var currentArticle models. ArticleModel
    if err := global. DB. First(&currentArticle, articleID). Error; err != nil {
        return nil, err
    }

    var allArticles []models. ArticleModel
    global. DB. Where("id != ?", articleID). Find(&allArticles)

    recommendations := make([]ArticleSimilarity, 0, len(allArticles))

    for _, article := range allArticles {
        Base similarity (Jaccard coefficient)
        score := calculateTagSimilarity(currentArticle.Tags, article. Tags)

        User behavior weighting
        if userID > 0 {
            The act of likes is weighted
            if global. Redis.SIsMember(context. Background(), 
                fmt. Sprintf("article_digg_users:%d", article.ID), userID). Val() {
                score *= 1.3
            }

            Collection behavior is weighted
            var collectExists int64
            global. DB. Model(&models. UserCollectModel{}).
                Where("user_id = ? AND article_id = ?", userID, article.ID).
                Count(&collectExists)
            if collectExists > 0 {
                score *= 1.5
            }
        }

        Time decay factor
        daysOld := time. Since(article. CreatedAt). Hours() / 24
        timeFactor := math. Exp(-0.1 * daysOld)
        score *= 1 + 0.5*timeFactor

        recommendations = append(recommendations, ArticleSimilarity{
            ArticleID: article.ID,
            Score:     score,
        })
    }

    Sort by score
    sort. Slice(recommendations, func(i, j int) bool {
        return recommendations[i]. Score > recommendations[j]. Score
    })

    if limit > len(recommendations) {
        limit = len(recommendations)
    }

    result := recommendations[:limit]

    Cache results
    if data, err := json. Marshal(result); err == nil {
        global. Redis.Set(context. Background(), cacheKey, data, 30*time. Minute)
    }

    return result, nil
}

Jaccard similarity calculation
func calculateTagSimilarity(currentTags, targetTags []string) float64 {
    tagSet := make(map[string]bool)
    for _, tag := range currentTags {
        tagSet[tag] = true
    }

    commons := 0
    for _, tag := range targetTags {
        if tagSet[tag] {
            commons++
        }
    }

    Jaccard coefficient = intersection / union
    return float64(commons) / float64(len(currentTags)+len(targetTags)-commons)
}

4. Analysis of search results

Go Copy

type SearchResult struct {
    Total int
    Hits  []SearchHit
}

type SearchHit struct {
    ID        string                 `json:"id"`
    Source    map[string]interface{} `json:"_source"`
    Highlight map[string][]string    `json:"highlight"`
}

func ParseSearchResponse(body io. Reader) (*SearchResult, error) {
    var response struct {
        Hits struct {
            Total struct {
                Value int `json:"value"`
            } `json:"total"`
            Hits []struct {
                ID        string                 `json:"_id"`
                Source    map[string]interface{} `json:"_source"`
                Highlight map[string][]string    `json:"highlight"`
            } `json:"hits"`
        } `json:"hits"`
    }

    if err := json. NewDecoder(body). Decode(&response); err != nil {
        return nil, err
    }

    result := &SearchResult{
        Total: response. Hits.Total.Value,
        Hits:  make([]SearchHit, len(response. Hits.Hits)),
    }

    for i, hit := range response. Hits.Hits {
        result. Hits[i] = SearchHit{
            ID:        hit.ID,
            Source:    hit. Source,
            Highlight: hit. Highlight,
        }
    }

    return result, nil
}

📊 Performance Optimization

1. Search performance

Index optimization: Set the number of shards and replicas reasonably
Query caching: ES has built-in query result caching
Pagination Optimization: Use from/size for efficient pagination

2. Recommended performance

Redis Cache: Recommended results cache for 30 minutes
Asynchronous Calculation: The background updates the recommendation data regularly
Batch Processing: Batch calculates similarity to reduce database queries

3. Synchronized performance

Batch Operations: Supports batch indexing and deletion
Asynchronous Synchronization: Asynchronously synchronizes ES after writing to MySQL
Error Retry: Automatic retry mechanism for network exceptions

🛡️ Reliability guarantee

1. Data consistency

Go Copy

Synchronize tasks in full at regular internship
func FullSyncToES() {
    var articles []models. ArticleModel
    global. DB. Find(&articles)

    for _, article := range articles {
        if err := SyncToES(article); err != nil {
            global. Log.Error("Full sync failed", 
                zap. Uint("id", article.ID), zap. Error(err))
        }
    }
}

2. Error handling

Go Copy

Retry mechanism
func retry. DoWithRetry(maxRetries int, delay time. Duration, fn func() error) error {
    for i := 0; i < maxRetries; i++ {
        if err := fn(); err == nil {
            return nil
        }
        if i < maxRetries-1 {
            time. Sleep(delay)
        }
    }
    return fmt. Errorf("Still failing after %d retry", maxRetries)
}

3. Monitoring alarms

Search Latency Monitoring: Record search response times
Synchronization failure alarm: An automatic ES synchronization failure alarm
Index Health Checks: Regularly check the status of your ES cluster

🚀 Usage Examples

Basic search

Go Copy

Keyword search
query := BuildSearchQuery("Go", "created_at", "desc", "0", "10", nil, "")
Perform a search...

Advanced search

Go Copy

Search with tags and categories
tags := []string{"backend", "microservices"}
query := BuildSearchQuery("distributed", "digg_count", "desc", "0", "20", tags, "technical")

Smart Recommendation

Go Copy

Get related article recommendations
recommendations, err := GetRecommendations(articleID, userID, 5)

Tech stack: Go + Elasticsearch + Redis + MySQL
Applicable scenarios: content platform, e-commerce search, knowledge base
Core algorithms: TF-IDF, Jaccard similarity, time decay