database-elasticsearch-search

Full-text search and relevance optimization with Elasticsearch

Elasticsearch Search

Scope: Full-text search, Query DSL, aggregations, relevance tuning, performance optimization Lines: ~450 Last Updated: 2025-10-27 Format Version: 1.0 (Atomic)

When to Use This Skill

Activate this skill when:

Core Concepts

Inverted Index

Elasticsearch uses inverted indexes for fast full-text search.

How it works:

Documents:
  Doc 1: "Elasticsearch is fast"
  Doc 2: "Elasticsearch is scalable"

Inverted Index:
  Term           → Document IDs
  elasticsearch  → [1, 2]
  fast           → [1]
  scalable       → [2]

Search Process:

  1. Query analyzed → terms extracted
  2. Look up terms in inverted index → get document IDs
  3. Score and rank documents by relevance
  4. Return top results

Mapping and Analyzers

Mapping defines field types and how they're indexed.

Text vs Keyword:

{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "fields": {
          "keyword": {"type": "keyword"}
        }
      }
    }
  }
}

Analyzer processes text during indexing and searching:

Input: "The QUICK Brown Fox"
  ↓ Standard Tokenizer
["The", "QUICK", "Brown", "Fox"]
  ↓ Lowercase Filter
["the", "quick", "brown", "fox"]
  ↓ Stop Words Filter
["quick", "brown", "fox"]

Query Types

Match Query (full-text):

{
  "query": {
    "match": {
      "description": {
        "query": "gaming laptop",
        "operator": "and"
      }
    }
  }
}

Term Query (exact match):

{
  "query": {
    "term": {
      "status.keyword": "published"
    }
  }
}

Bool Query (combine queries):

{
  "query": {
    "bool": {
      "must": [{"match": {"title": "laptop"}}],
      "filter": [
        {"term": {"category": "electronics"}},
        {"range": {"price": {"gte": 500}}}
      ],
      "should": [{"term": {"brand": "Apple"}}],
      "must_not": [{"term": {"status": "discontinued"}}]
    }
  }
}

Patterns

Pattern 1: Multi-Field Search with Boosting

When to use:

{
  "query": {
    "multi_match": {
      "query": "gaming laptop",
      "fields": ["title^3", "description^1.5", "tags"],
      "type": "best_fields",
      "fuzziness": "AUTO"
    }
  }
}

Benefits:

Pattern 2: Filtered Search (Performance)

Use case: Exact filters + full-text search

// ❌ Bad: Everything in must (slower, scoring overhead)
{
  "query": {
    "bool": {
      "must": [
        {"match": {"description": "laptop"}},
        {"term": {"status": "published"}},
        {"range": {"price": {"gte": 500}}}
      ]
    }
  }
}

// ✅ Good: Use filter for exact matches (faster, cached)
{
  "query": {
    "bool": {
      "must": [{"match": {"description": "laptop"}}],
      "filter": [
        {"term": {"status": "published"}},
        {"range": {"price": {"gte": 500}}}
      ]
    }
  }
}

Benefits:

Pattern 3: Autocomplete with Completion Suggester

Use case: Fast typeahead suggestions

Mapping:

{
  "mappings": {
    "properties": {
      "suggest": {
        "type": "completion",
        "contexts": [
          {"name": "category", "type": "category"}
        ]
      }
    }
  }
}

Index:

{
  "name": "Gaming Laptop",
  "suggest": {
    "input": ["gaming laptop", "laptop gaming", "gaming"],
    "weight": 10,
    "contexts": {"category": "electronics"}
  }
}

Query:

{
  "suggest": {
    "product_suggest": {
      "prefix": "gam",
      "completion": {
        "field": "suggest",
        "size": 10,
        "contexts": {"category": "electronics"},
        "fuzzy": {"fuzziness": "AUTO"}
      }
    }
  }
}

Benefits:

Pattern 4: Faceted Search

Use case: E-commerce filters (category, price, brand)

{
  "query": {
    "match": {"title": "laptop"}
  },
  "aggs": {
    "categories": {
      "terms": {"field": "category.keyword", "size": 10}
    },
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          {"to": 500},
          {"from": 500, "to": 1500},
          {"from": 1500}
        ]
      }
    },
    "brands": {
      "terms": {"field": "brand.keyword", "size": 20}
    }
  }
}

Benefits:

Pattern 5: Pagination with search_after

Use case: Deep pagination (page 100+)

// ❌ Bad: from/size for deep pagination (expensive)
GET /products/_search?from=10000&size=10

// ✅ Good: search_after (efficient, stateless)
{
  "query": {"match_all": {}},
  "size": 10,
  "sort": [
    {"created_at": "desc"},
    {"_id": "desc"}
  ],
  "search_after": ["2025-10-27T00:00:00Z", "prod_123"]
}

Why search_after?

Trade-off: Can't jump to arbitrary pages (sequential only)

Pattern 6: Custom Scoring with function_score

Use case: Boost results by multiple factors

{
  "query": {
    "function_score": {
      "query": {"match": {"title": "laptop"}},
      "functions": [
        {
          "filter": {"term": {"is_premium": true}},
          "weight": 2
        },
        {
          "field_value_factor": {
            "field": "rating",
            "factor": 1.5,
            "modifier": "sqrt"
          }
        },
        {
          "gauss": {
            "created_at": {
              "origin": "now",
              "scale": "30d",
              "decay": 0.5
            }
          }
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

Factors:

Pattern 7: Aggregations with Nested Buckets

Use case: Multi-level analytics

{
  "size": 0,
  "aggs": {
    "categories": {
      "terms": {"field": "category.keyword"},
      "aggs": {
        "brands": {
          "terms": {"field": "brand.keyword"},
          "aggs": {
            "avg_price": {"avg": {"field": "price"}},
            "avg_rating": {"avg": {"field": "rating"}}
          }
        }
      }
    }
  }
}

Result Structure:

Electronics
├── Apple: avg_price: $1200, avg_rating: 4.5
├── Dell: avg_price: $800, avg_rating: 4.2
Books
├── Penguin: avg_price: $15, avg_rating: 4.7

Pattern 8: Highlighting Matches

Use case: Show matching text snippets

{
  "query": {"match": {"description": "gaming"}},
  "highlight": {
    "fields": {
      "description": {
        "fragment_size": 150,
        "number_of_fragments": 3
      }
    },
    "pre_tags": ["<mark>"],
    "post_tags": ["</mark>"]
  }
}

Result:

{
  "highlight": {
    "description": [
      "High performance <mark>gaming</mark> laptop...",
      "Ideal for <mark>gaming</mark> and content creation..."
    ]
  }
}

Quick Reference

Query Types

Type              | Use Case                    | Analyzed
------------------|-----------------------------|---------
match             | Full-text search            | Yes
term              | Exact match                 | No
range             | Numeric/date ranges         | No
bool              | Combine queries             | N/A
multi_match       | Multi-field search          | Yes
prefix            | Prefix matching             | No
wildcard          | Pattern matching            | No
fuzzy             | Typo tolerance              | Yes
match_phrase      | Exact phrase                | Yes

Aggregation Types

Type              | Purpose
------------------|----------------------------------
terms             | Group by field value
range             | Group by ranges
date_histogram    | Group by time intervals
avg/min/max/sum   | Calculate metrics
cardinality       | Unique count (approximate)
percentiles       | Distribution percentiles
top_hits          | Top documents per bucket

Mapping Types

Type              | Use Case
------------------|----------------------------------
text              | Full-text search
keyword           | Exact match, aggregations
integer/long      | Whole numbers
float/double      | Decimals
date              | Dates/timestamps
boolean           | true/false
nested            | Array of objects
geo_point         | Lat/lon coordinates
completion        | Autocomplete

Common Pitfalls

❌ Using term query on text fields

{"term": {"title": "Quick"}}  // Won't match (analyzed as "quick")

✅ Use match or query title.keyword

❌ Deep pagination with from/size

GET /products/_search?from=10000&size=10  // Expensive

✅ Use search_after for deep pagination

❌ Large terms aggregations

{"aggs": {"all_users": {"terms": {"size": 100000}}}}

✅ Use composite aggregation or limit size

❌ Not using filters for exact matches

{"bool": {"must": [{"term": {"status": "active"}}]}}

✅ Move to filter for caching and performance

❌ Leading wildcards

{"wildcard": {"name": "*smith"}}  // Extremely slow

✅ Use reverse field + prefix or full-text search

❌ Over-sharding

{"settings": {"number_of_shards": 100}}  // For 10GB index

✅ Target 10-50 GB per shard (2-3 shards for 10GB)

Level 3: Resources

This skill includes Level 3 Resources (executable tools, reference materials, examples):

Comprehensive Reference

resources/REFERENCE.md (3,500+ lines)

Deep dive covering:

Executable Scripts

resources/scripts/analyze_queries.py

Analyzes Elasticsearch Query DSL queries for performance issues:

# Analyze query file
./analyze_queries.py --query-file queries.json

# Analyze single query
./analyze_queries.py --query '{"query": {"match": {"field": "value"}}}'

# JSON output
./analyze_queries.py --query-file queries.json --json

# With Elasticsearch endpoint
./analyze_queries.py --query-file queries.json --endpoint http://localhost:9200

Features:

resources/scripts/optimize_indexes.py

Analyzes indices and provides optimization recommendations:

# Analyze single index
./optimize_indexes.py --index products --endpoint http://localhost:9200

# Analyze all indices
./optimize_indexes.py --all-indices

# JSON output
./optimize_indexes.py --index products --json

Features:

resources/scripts/benchmark_search.py

Benchmarks query performance with statistical analysis:

# Benchmark query file
./benchmark_search.py --query-file queries.json --index products

# Custom iterations and concurrency
./benchmark_search.py --query-file queries.json --iterations 500 --concurrent-requests 10

# JSON output
./benchmark_search.py --query-file queries.json --json

Features:

Examples

Mappings:

Queries:

Python:

Node.js:

Docker:

Quick Start

# Start Elasticsearch + Kibana
cd resources/examples/docker && docker-compose up -d

# Analyze query performance
./resources/scripts/analyze_queries.py --query-file my-queries.json

# Get index optimization recommendations
./resources/scripts/optimize_indexes.py --index products

# Benchmark queries
./resources/scripts/benchmark_search.py --query-file queries.json --index products

# Run Python examples
cd resources/examples/python
pip install elasticsearch
python elasticsearch-client.py

# Bulk index test data
python bulk-indexer.py --generate-test-data 10000 --index products

Related Skills

Last Updated: 2025-10-27 Format Version: 1.0 (Atomic)