Force Merge API

Introduced 1.0

The force merge API operation forces a merge on the shards of one or more indexes. For a data stream, the API forces a merge on the shards of the stream’s backing indexes.

Endpoints

POST /_forcemerge
POST /{index}/_forcemerge

The merge operation

In OpenSearch, a shard is a Lucene index, which consists of segments (or segment files). Segments store the indexed data and are immutable. Periodically, OpenSearch merges smaller segments into a new, larger segment and then deletes the original segments. Merging reduces the overall number of segments on each shard and frees up disk space.

OpenSearch performs background segment merges that produce segments no larger than index.merge.policy.max_merged_segment (the default is 5 GB).

Deleted documents

When a document is deleted from an OpenSearch index, it is not deleted from the Lucene segment but is rather only marked to be deleted. When the segment files are merged, deleted documents are removed (or expunged). Thus, merging also frees up space occupied by documents marked as deleted.

OpenSearch retains a shard’s history of operations, including deleted documents, for the period specified in the index.soft_deletes.retention_lease.period setting (default 12h). A force merge does not expunge deleted documents that are still retained, so the number of deleted documents might not decrease immediately after a force merge. For more information, see Index settings.

Force Merge API

In addition to periodic merging, you can force a segment merge using the Force Merge API. By default, the merge policy determines which segments to merge, so a force merge request might not change the number of segments. To merge all segments of each shard into one segment, set max_num_segments to 1.

Use the Force Merge API on an index only after all write requests sent to the index are completed. The force merge operation can produce very large segments. If write requests are still sent to the index, then the merge policy does not merge these segments until they primarily consist of deleted documents. This can increase disk space usage and lead to performance degradation.

By default, the Force Merge API call does not return until the merge completes. If the connection is lost during this time, the force merge operation continues in the background. To run the operation asynchronously, set wait_for_completion to false.

Each node runs force merge operations in its force_merge thread pool. Force merge requests that arrive while all threads in the pool are busy are queued until a thread becomes available.

Force merging multiple indexes

To force merge multiple indexes, you can call the Force Merge API on the following index combinations:

• Multiple indexes
• One or more data streams containing multiple backing indexes
• One or more index aliases pointing to multiple indexes
• All data streams and indexes in a cluster

When you force merge multiple indexes, each node merges its shards using the force_merge thread pool. The pool size is one-eighth of the node’s allocated processors, with a minimum of 1, so on most nodes shards are merged one at a time. While the force merge operation is in progress, the storage for the shard temporarily increases so that all segments can be rewritten into a new segment. When max_num_segments is set to 1, the storage for the shard can temporarily double.

Force merging data streams

It can be useful to force merge data streams in order to manage a data stream’s backing indexes, especially after a rollover operation. Time-based indexes receive indexing requests only during a specified time period. Once that time period has elapsed and the index receives no more write requests, you can force merge segments of all index shards into one segment. Searches on single-segment shards are more efficient because they use simpler data structures.

Path parameters

The following table lists the available path parameters. All path parameters are optional.

Query parameters

The following table lists the available query parameters. All query parameters are optional.

Example requests

The following examples show how to use the Force Merge API.

Force merge a specific index

The following example force merges a specific index:

POST /testindex1/_forcemerge

response = client.indices.forcemerge(
  index = "testindex1"
)

Force merge multiple indexes

The following example force merges multiple indexes:

POST /testindex1,testindex2/_forcemerge

response = client.indices.forcemerge(
  index = "testindex1,testindex2"
)

Force merge all indexes

The following example force merges all indexes:

POST /_forcemerge

response = client.indices.forcemerge()

Force merge a data stream’s backing indexes into one segment

The following example force merges a data stream’s backing indexes into one segment:

POST /testindex-logs/_forcemerge?max_num_segments=1

response = client.indices.forcemerge(
  index = "testindex-logs",
  params = { "max_num_segments": "1" }
)

Force merge primary shards

The following example force merges an index’s primary shards:

POST /testindex-logs/_forcemerge?primary_only=true

response = client.indices.forcemerge(
  index = "testindex-logs",
  params = { "primary_only": "true" }
)

Example response

The total count includes both primary and replica shards. In the following example response, the index has one primary shard and one replica shard. The replica shard is unassigned, so the operation succeeded only on the primary shard:

{
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  }
}

If wait_for_completion is false, the response contains the task ID, which you can use to monitor the task using the Tasks API:

{
  "task": "<node_id>:<task_number>"
}

Response body fields

The following table lists all response fields.

The _shards object contains the following fields.

Required permissions

If you use the Security plugin, make sure you have the appropriate permissions: indices:admin/forcemerge.