Neural sparse ANN search performance tuning

Neural sparse ANN search offers several parameters that allow you to balance the trade-off between query recall (accuracy) and query efficiency (latency). Query parameters, which you supply in the method_parameters object of a query, take effect immediately. Mapping parameters, which you supply in the method object of a sparse_vector field, are fixed when the field is created.To change engine or any value in method.parameters, create a new index with the intended mapping and reindex your data.

Choosing an engine

Introduced 3.9

Neural sparse ANN search runs the SEISMIC algorithm using one of two engines, which you select for each sparse_vector field using the method.engine mapping parameter. Valid values are lucene (default) and native. Both engines accept the same query parameters, and they accept the same method.parameters values except for clustering_batch_size and forward_index, which apply to the native engine only. The tuning guidance in the rest of this page therefore applies to both engines. For a full comparison of the engines and instructions for enabling the native engine, see Engines.

The engines differ in the location of the data structures that the algorithm reads, which affects node sizing:

• The Lucene engine holds clustered posting lists and the forward index in JVM heap caches bounded by plugins.neural_search.circuit_breaker.limit. Size the node’s heap to hold the working set of every sparse segment it serves.
• The native engine reads its index from a memory-mapped file on disk, so the index consumes no JVM heap and adds no garbage collection pressure. Size the node to leave enough RAM for the operating system page cache.

Consider the native engine when you want better query and index build performance, when your sparse indexes are large enough that holding them in JVM heap constrains the node, or when you want to serve a large sparse index from a comparatively small heap. Consider the Lucene engine when you want the default configuration, or when you rely on the Warm Up and Clear Cache APIs or the sparse memory statistics reported by the Neural Search Stats API.

Benchmark both engines against your own data and query mix before choosing one. Relative throughput, latency, and memory usage depend on your corpus, your parameter settings, and the resources available on the node.

Indexing performance tuning

These parameters control index construction and memory usage:

• n_postings: The maximum number of documents to retain in each posting list.

  A smaller n_postings value applies more aggressive pruning, meaning fewer document identifiers are kept in each posting list. Lower values speed up index building and query execution but reduce recall and memory consumption. If not specified, the algorithm calculates the value as \(0.0005 \times \text{document count}\) at the segment level.

• cluster_ratio: The fraction of documents in each posting list used to determine the cluster count.

  After pruning, each posting list contains cluster_ratio × posting_document_count. Increasing cluster_ratio results in more clusters, which improves recall but increases index build time, query latency, and memory usage.

• summary_prune_ratio: The fraction of tokens to keep in cluster summary vectors for approximate matching.

  This parameter controls how many tokens are retained in the summary of each cluster. The summary helps determine whether to examine a cluster during a query. If embeddings vary widely in token counts, adjust this parameter accordingly. Higher values retain more tokens in the summary.

• approximate_threshold: The minimum number of documents in a segment required to activate neural sparse ANN search.

  This parameter controls whether to activate the neural sparse ANN algorithm in a segment once the segment’s document count reaches the specified threshold. As the total number of documents increases, individual segments contain more documents. In this case, you can set approximate_threshold to a higher value in order to avoid rebuilding clusters repeatedly when segments with fewer documents are merged. This parameter is especially important if you do not use force merge operations to combine all segments into one, because segments with fewer documents than the threshold fall back to the rank_features (regular neural sparse search) mode. Note that if you set this value too high, neural sparse ANN search may never activate.

• clustering_batch_size: The number of batches that each inverted list is split into for clustering. Supported for the native engine only.

  By default, this parameter is 1 and clustering runs over the whole corpus. Setting it to a higher value, up to 10000, splits each inverted list into that many batches and clusters each batch separately, which reduces the memory required to build the index in exchange for a longer build time. Increase this value if index building is memory constrained.

• forward_index: How the forward index is stored on disk. Supported for the native engine only.

  The default shared layout stores one contiguous forward index for the field, which uses less disk space. The per_block layout stores each block’s vectors inline next to the block, so a query reads only the blocks that it selects, which reduces query latency but uses more disk space. For more information, see Choosing a forward index layout.

Query performance tuning

These parameters affect search performance and recall:

• top_n: The number of query tokens with the highest weights to retain for approximate sparse queries.

  In the neural sparse ANN search algorithm, only the top top_n tokens in a query are retained based on their weights. This parameter controls the balance between search efficiency (latency) and accuracy (recall). A higher value improves accuracy but increases latency, while a lower value reduces latency at the cost of accuracy.

• heap_factor: Controls the trade-off between recall and performance.

  During neural sparse ANN search, the algorithm decides whether to examine a cluster by comparing the cluster’s score with the top score in the result queue divided by heap_factor. A larger heap_factor lowers the threshold that clusters must meet in order to be examined, causing the algorithm to examine more clusters and improving accuracy at the cost of slower query speed. Conversely, a smaller heap_factor raises the threshold, making the algorithm more selective about which clusters to examine. This parameter provides finer control than top_n, allowing you to slightly adjust the trade-off between accuracy and latency.

Other optimization strategies

In addition to tuning the preceding parameters, you can employ the following optimization strategies.

Building clusters

Index building can benefit from using multiple threads. You can adjust the number of threads used for cluster building by specifying the plugins.neural_search.sparse.algo_param.index_thread_qty setting (by default, 1). For information about updating this setting, see Vector search settings. Using a higher plugins.neural_search.sparse.algo_param.index_thread_qty can reduce force merge time when neural sparse ANN search is enabled, though it also consumes more system resources. This setting applies to both the Lucene engine and the native engine.

Querying after a cold start

The Lucene engine’s cache is empty after rebooting OpenSearch, so the first several hundred queries may experience high latency. To address this “cold start” issue, you can use the Warm Up API. This API loads data from disk into cache, ensuring optimal performance for subsequent queries. You can also use the Clear Cache API to free up memory when needed.

The native engine does not use this cache, so the Warm Up and Clear Cache APIs do not apply to it. For the native engine, the first query against a segment pays a one-time cost to memory map the segment’s index file, and later queries reuse the existing mapping.

Force merging segments

Neural sparse ANN search automatically builds clustered posting lists once a segment’s document count exceeds approximate_threshold. However, you can often achieve lower query latency by merging all segments into a single segment:

POST /sparse-ann-documents/_forcemerge?max_num_segments=1

You can also set approximate_threshold to a high value so that individual segments do not trigger clustering but the merged segment does. This approach helps avoid repeated cluster building during indexing.

The native engine builds the sparse index faster, so force merges complete sooner. For more information, see Choosing an engine.

Best practices

• Start with default parameters and tune based on your specific dataset.
• For the Lucene engine, monitor memory usage and adjust cache settings accordingly.
• Consider the trade-off between indexing time and query performance.
• Choose an engine before creating the field. method is not updatable, so switching engines later requires reindexing into a new index.
• If you are memory constrained or your JVM heap is under pressure, we recommend the native engine, which keeps its index in a memory-mapped file on disk.
• When sizing a node for the native engine, leave enough RAM for the operating system page cache rather than increasing the JVM heap.
• Set forward_index to per_block when query latency matters more than disk usage, and keep the default shared layout when you want to minimize disk usage.
• For the native engine, if index building is memory constrained, increase clustering_batch_size to lower peak build memory in exchange for a longer build time.
• Do not combine neural sparse ANN search fields with a pipeline that includes a two-phase processor.

Next steps

• Neural sparse ANN search