Neural sparse ANN search

Introduced 3.3

Neural sparse approximate nearest neighbor (ANN) search improves query efficiency by balancing accuracy and latency. Unlike traditional neural sparse search, which performs exact search on rank_features fields, neural sparse ANN search uses the Spilled Clustering of Inverted Lists with Summaries for Maximum Inner Product Search (SEISMIC) algorithm on sparse_vector fields to provide optimized query performance using approximate search techniques.

Neural sparse ANN search provides the following advantages over traditional neural sparse search:

• Query performance improvement: Achieves significant query speed improvements compared to two-phase queries under ≥90% recall conditions with better than linear performance scaling as dataset size increases.
• Scalability: Maintains consistent query performance as datasets scale to 50 million vectors on a single node.
• Memory efficiency: Uses byte quantization to reduce index size. Depending on the engine, memory is managed by a JVM heap cache with a circuit breaker or by off-heap memory-mapped files, which prevents resource exhaustion.
• Hybrid approach: Automatically selects the optimal indexing strategy based on segment size, with minimal impact on indexing performance.
• Search flexibility: Provides tunable trade-offs between high recall and low latency using query parameters.
• Engine choice: Uses either the Lucene engine or the native engine, which builds and searches the index in off-heap memory. For more information, see Engines.

Consider neural sparse ANN search when you need the efficiency of sparse retrieval but require better performance than traditional neural sparse search methods can provide at scale:

• Large-scale applications: Datasets with millions to billions of documents in which query performance is critical.
• High-throughput scenarios: Applications requiring fast response times under heavy query loads.

How neural sparse ANN search works

Neural sparse ANN search implements several techniques to optimize both indexing and querying of neural sparse vectors.

Indexing

During the indexing phase, neural sparse ANN search implements several key optimizations:

1. Posting list clustering: For each term in the inverted index, the algorithm performs the following actions:
   • Sorts documents by their token weights in descending order.
   • Retains only the top n_postings documents with the highest weights.
   • Applies a clustering algorithm to group similar documents into one cluster. The native engine uses clustering_batch_size to split each posting list into batches and cluster each batch separately, which reduces the memory needed to build the index at the cost of a longer build time.
   • Generates summary sparse vectors for each cluster, keeping only the highest-weighted tokens.
2. Forward index maintenance: Neural sparse ANN search maintains both the clustered inverted index and a forward index that stores complete sparse vectors organized by document ID for efficient access during query processing.

Query processing

During query execution, neural sparse ANN search employs an efficient retrieval process:

1. Token-level pruning: For a given query, all tokens are sorted based on their weights. Only the top_n tokens with the highest weights are kept so that fewer posting lists are visited.

2. Cluster-level pruning: The algorithm first computes dot product scores between the query vector and cluster summary vectors. Only clusters with scores above a dynamic threshold are selected for detailed examination.

3. Document-level scoring: For selected clusters, neural sparse ANN search examines individual documents within those clusters, computing exact dot product scores between the query and document vectors retrieved from the forward index.

This approach dramatically reduces the number of documents that need to be scored, resulting in significant performance improvements while maintaining high accuracy.

Hybrid indexing

Neural sparse ANN search is a hybrid indexing approach that depends on the document count in each segment to balance indexing and query performance:

• Segments with fewer documents than approximate_threshold: Indexed without clustering, so queries against them score every matching document. The Lucene engine indexes these segments as plain neural sparse (rank_features) segments and queries them using the standard neural sparse query. The native engine indexes them as an equivalent inverted index built in its native format.

• Segments with more documents than approximate_threshold: Indexed as neural sparse ANN segments and queried using the sparse ANN query.

This hybrid approach balances indexing performance with query speed. Small segments avoid the overhead of clustering, while large segments benefit from approximate search optimizations. The system maintains backward compatibility by supporting both traditional neural sparse queries and neural sparse ANN queries within the same index.

For more information about the SEISMIC algorithm, see Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representations.

Engines

Introduced 3.9

An engine is the implementation that builds the neural sparse ANN index and runs queries against it. Both engines implement the SEISMIC algorithm and accept the same algorithm and query parameters; they differ in the location of the index and the method of managing its memory.

OpenSearch supports the following engines:

• Lucene: Builds the clustered posting lists and forward index in JVM heap and serves queries from a plugin-managed cache. This is the default.
• Native: Builds the index into a memory-mapped file on disk and serves queries from off-heap memory.

Select an engine using the engine parameter in the field’s method object:

PUT /my-sparse-ann-index
{
  "settings": {
    "index": {
      "sparse": true
    }
  },
  "mappings": {
    "properties": {
      "sparse_embedding": {
        "type": "sparse_vector",
        "method": {
          "name": "seismic",
          "engine": "native",
          "parameters": {
            "n_postings": 4000,
            "cluster_ratio": 0.1,
            "forward_index": "per_block",
            "clustering_batch_size": 1
          }
        }
      }
    }
  }
}

The method object cannot be updated after the field is created. To change the engine, create a new index with the intended mapping and reindex your data.

Comparing the engines

The following table compares the two engines.

For guidance on choosing between the engines, see Choosing an engine.

Lucene engine

The Lucene engine is the default and requires no additional configuration. It stores clustered posting lists and forward index data in JVM heap. Memory usage is bounded by a circuit breaker, and data is evicted from the cache when the limit is reached. For more information, see Memory and caching settings.

Because the index is held in heap, a node running the Lucene engine at scale needs a JVM heap large enough to hold the working set of every sparse segment it serves.

Native engine

The native engine writes the SEISMIC index to a file that OpenSearch memory maps at query time and reads in place. The on-disk layout is the runtime layout, so nothing is reconstructed when the index is loaded. This has the following consequences:

• The index does not consume JVM heap and adds no garbage collection pressure, so a node can serve large sparse indexes with a comparatively small heap.
• Index memory is reclaimable operating system page cache, so the operating system reclaims it under memory pressure. No circuit breaker or eviction policy is involved.
• The first query against a segment pays a one-time cost to establish the memory mapping. Later queries reuse it. This cost grows with segment size.

The native engine relies on the operating system page cache, and no setting bounds the amount of memory that its index uses. To size a node for the native engine, leave enough RAM for the page cache, the same as for any other memory-mapped Lucene data.

Enabling the native engine

The native engine is disabled by default. Both of the following cluster settings must be true before a field can use the native engine:

To enable the native engine, send the following request:

PUT _cluster/settings
{
  "persistent": {
    "plugins.neural_search.sparse.native_engine_enabled": true
  }
}

If either setting is false, OpenSearch rejects attempts to create a field with engine set to native, to index documents into an existing native engine field, and to query one. OpenSearch does not fall back to the Lucene engine, because the field’s mapping specifies the native engine.

While the native engine is disabled, OpenSearch skips building the native index during segment flush and merge. Raw vectors are still written to disk, so re-enabling the engine and then force merging the affected indexes rebuilds the native index.

Choosing a forward index layout

The forward_index algorithm parameter controls how the native engine stores the forward index. Specify it in the field’s method.parameters object. The following table describes the available layouts.

Both layouts are memory mapped and apply the same quantization. The forward_index parameter selects where forward index data is placed; the precision at which it is stored is the same for both layouts.

If you set engine to lucene and specify a forward_index value other than shared, the request is rejected.

Considerations for the native engine

Before choosing the native engine, note the following:

• The Warm Up and Clear Cache APIs operate on the Lucene engine’s cache and do not apply to the native engine.
• The sparse memory statistics returned by the Neural Search Stats API report Lucene engine cache usage only. They do not account for native engine index memory.
• The plugins.neural_search.circuit_breaker.limit setting has no effect on the native engine.
• The native engine requires an index whose data is stored on the local filesystem.

Step 1: Create an index

To use neural sparse ANN search, you must enable the sparse setting at the index level and use sparse_vector as the field type.

Index settings

Set index.sparse: true to enable neural sparse ANN search functionality:

PUT /my-sparse-ann-index
{
  "settings": {
    "index": {
      "sparse": true,
      "number_of_shards": 2,
      "number_of_replicas": 1
    }
  },
  "mappings": {
    "properties": {
      "sparse_embedding": {
        "type": "sparse_vector",
        "method": {
          "name": "seismic",
          "parameters": {
            "n_postings": 4000,
            "cluster_ratio": 0.1,
            "summary_prune_ratio": 0.4,
            "approximate_threshold": 1000000
          }
        }
      },
      "text": {
        "type": "text"
      }
    }
  }
}

This example omits the engine parameter, so the field uses the default Lucene engine. To use the native engine instead, see Engines.

For parameter information, see Sparse vector.

Step 2: Ingest data

Ingest documents with sparse embeddings where tokens are represented as integers with corresponding weights:

POST _bulk
{ "create": { "_index": "my-sparse-ann-index", "_id": "1" } }
{ "sparse_embedding": {"10": 0.85, "23": 1.92, "24": 0.67, "78": 2.54, "156": 0.73}, "text": "OpenSearch neural sparse search" }
{ "create": { "_index": "my-sparse-ann-index", "_id": "2" } }
{ "sparse_embedding": {"3": 1.22, "19": 0.11, "21": 0.35, "300": 1.74, "985": 0.96}, "text": "Machine learning algorithms" }

You can also use ingest pipelines to automatically format tokens as integers.

Step 3: Query the index

Query neural sparse ANN search using the neural_sparse query with method_parameters for performance tuning.

Do not combine neural sparse ANN search with two-phase pipelines.

Query using natural language

Query using natural language text, which requires a deployed sparse encoding model to convert the text into sparse vectors:

GET /my-sparse-ann-index/_search
{
  "query": {
    "neural_sparse": {
      "sparse_embedding": {
        "query_text": "machine learning algorithms",
        "model_id": "your_sparse_model_id",
        "method_parameters": {
          "k": 10,
          "top_n": 10,
          "heap_factor": 1.0
        }
      }
    }
  }
}

Query using raw vectors

Query using precomputed sparse vectors, where tokens are specified as integers with their corresponding weights:

GET /my-sparse-ann-index/_search
{
  "query": {
    "neural_sparse": {
      "sparse_embedding": {
        "query_tokens": {
          "1055": 1.7,
          "2931": 2.3
        },
        "method_parameters": {
          "k": 10,
          "top_n": 6,
          "heap_factor": 1.2
        }
      }
    }
  }
}

Query parameters

The query syntax and these parameters are the same for both engines. You select the engine in the field mapping. For more information, see Engines.

Filtering support

Neural sparse ANN search supports filtering. If the filter matches fewer documents than k, both engines run an exact search over the filtered documents. Otherwise, the two engines apply the filter at different points:

• The Lucene engine applies the filter after approximate retrieval (post-filtering). The results are the intersection of the top matches and the filter, so a selective filter can return fewer than k results.
• The native engine pushes the filter down as a candidate set before retrieval (pre-filtering). Retrieval runs within the filtered set, so the query can return the full k results.

For more information, see Filtering in neural sparse ANN search.

Cluster settings

Neural sparse ANN search supports the following cluster settings.

Thread pool configuration

Building a clustered inverted index structure requires intensive computation. By default, the algorithm uses a single-threaded thread pool to build clusters. You can increase the thread pool size to build clusters in parallel, using more CPU cores and reducing index build time. This setting applies to both engines.

To configure the thread pool size, update the plugins.neural_search.sparse.algo_param.index_thread_qty setting:

PUT /_cluster/settings
{
  "persistent": {
    "plugins.neural_search.sparse.algo_param.index_thread_qty": 4
  }
}

Memory and caching settings

The Lucene engine provides a circuit breaker that prevents the algorithm from using excessive memory and ensures that other OpenSearch operations remain unaffected. The default value of circuit_breaker.limit is 10%. You can adjust this setting to control the total memory allocated to the algorithm. When memory usage reaches the defined limit, a cache eviction occurs, removing the least recently used data.

A higher circuit breaker limit allows more memory usage and reduces the frequency of cache evictions but may impact other OpenSearch operations. A lower limit provides greater safety but can result in more frequent cache evictions.

To configure the circuit breaker limit, send the following request:

PUT _cluster/settings
{
  "persistent": {
    "plugins.neural_search.circuit_breaker.limit": "30%"
  }
}

The circuit breaker bounds the Lucene engine’s JVM heap cache. It has no effect on the native engine, which holds its index in a memory-mapped file managed by the operating system. For more information, see Native engine.

For more information, see Neural Search plugin settings.

Monitoring

Monitor memory usage and query statistics using the Neural Search Stats API.

The sparse memory statistics report Lucene engine cache usage only. Native engine index memory is held in the operating system page cache and is not reflected in these statistics.

Performance tuning

Neural sparse ANN search provides multiple parameters for balancing search accuracy and query speed. For comprehensive tuning guidance, see Neural sparse ANN search performance tuning.

Next steps

• For query syntax, see Neural sparse query.
• For field type information, see Sparse vector.
• For performance optimization, see Neural sparse ANN search performance tuning.
• For filtering options, see Filtering in neural sparse ANN search.