Bulk Replication API

Use the bulk replication operations to programmatically manage cross-cluster replication across multiple indexes using a single API call. You can specify an index pattern to select matching indexes and perform start, stop, pause, or resume operations in bulk.

All bulk replication operations are asynchronous. Each request returns a task_id that you can use to check the task progress or cancel the operation. Only one bulk replication task can run at a time per cluster; submitting a new bulk replication operation while one is in progress results in a 409 Conflict error. If a bulk replication operation fails for some indexes, the task continues processing the remaining indexes and reports a list of indexes for which the operation succeeded and failed. You can cancel a running bulk task at any time, but indexes already processed are not reverted. Indexes are processed in configurable batches (see Settings). Bulk replication operations are not idempotent: if you submit the same request after a previous task has finished running, a new task is created.

Pattern matching

The pattern field supports the same syntax as OpenSearch index patterns:

• * matches any number of characters.
• ? matches a single character.
• Patterns must not start with _.
• Patterns must not contain spaces, quotes ("), angle brackets (<, >), pipes (|), backslashes (\), hash marks (#), or commas (,).

Examples:

• my-index-* matches my-index-1, my-index-2, my-index-logs, and so on.
• * matches all indexes (excluding system indexes starting with .).
• logs-2024-0? matches logs-2024-01–logs-2024-09.

Bulk start replication

Introduced 3.7

Initiates replication for all indexes of the leader cluster that match the specified pattern. Send this request to the follower cluster.

Endpoints

POST /_plugins/_replication/_bulk_start

Request body fields

The following table lists the available request body fields.

Example request

POST /_plugins/_replication/_bulk_start
{
   "leader_alias": "<connection-alias-name>",
   "pattern": "<index-pattern>",
   "use_roles": {
      "leader_cluster_role": "<role-name>",
      "follower_cluster_role": "<role-name>"
   },
   "filters": {
      "exclude_index": ["<index-name-1>", "<index-name-2>"]
   }
}

Bulk stop replication

Introduced 3.7

Stops replication and converts all matching follower indexes to standard indexes that accept write operations. Send this request to the follower cluster.

Endpoints

POST /_plugins/_replication/_bulk_stop

Request body fields

The following table lists the available request body fields.

Example request

POST /_plugins/_replication/_bulk_stop
{
   "pattern": "<index-pattern>",
   "filters": {
      "exclude_index": ["<index-name-1>", "<index-name-2>"]
   }
}

Bulk pause replication

Introduced 3.7

Pauses replication for all matching follower indexes. Send this request to the follower cluster.

You can’t resume replication after it’s been paused for more than 12 hours. You must stop replication, delete the follower index, and restart replication of the leader.

Endpoints

POST /_plugins/_replication/_bulk_pause

Request body fields

The following table lists the available request body fields.

Example request

POST /_plugins/_replication/_bulk_pause
{
   "pattern": "<index-pattern>",
   "filters": {
      "exclude_index": ["<index-name-1>", "<index-name-2>"]
   }
}

Bulk resume replication

Introduced 3.7

Resumes replication for all paused follower indexes that match the specified pattern. The leader cluster must be reachable and retention leases must still be valid for each index. Send this request to the follower cluster.

Endpoints

POST /_plugins/_replication/_bulk_resume

Request body fields

The following table lists the available request body fields.

Example request

POST /_plugins/_replication/_bulk_resume
{
   "pattern": "<index-pattern>",
   "filters": {
      "exclude_index": ["<index-name-1>", "<index-name-2>"]
   }
}

Get bulk replication status

Introduced 3.7

Returns the current replication status of all indexes matching a pattern. Unlike the Get Bulk Task Status endpoint, this endpoint returns the live replication state rather than the progress of a bulk replication operation. Send this request to the follower cluster.

Endpoints

GET /_plugins/_replication/_bulk_status?pattern={index-pattern}

Query parameters

The following table lists the available query parameters.

Example response

{
   "indices": {
      "bulk-test-3": {
         "status": "SYNCING",
         "reason": "User initiated",
         "leader_alias": "leader-cluster",
         "leader_index": "bulk-test-3",
         "follower_index": "bulk-test-3",
         "syncing_details": {
            "leader_checkpoint": -1,
            "follower_checkpoint": -1,
            "seq_no": 0
         }
      },
      "bulk-test-6": {
         "status": "PAUSED",
         "reason": "bulk_pause",
         "leader_alias": "leader-cluster",
         "leader_index": "bulk-test-6",
         "follower_index": "bulk-test-6"
      },
      "bulk-test-9": {
         "status": "BOOTSTRAPPING",
         "reason": "User initiated",
         "leader_alias": "leader-cluster",
         "leader_index": "bulk-test-9",
         "follower_index": "bulk-test-9",
         "bootstrap_details": {
            "status": "IN_PROGRESS",
            "shard_restore_details": {
               "total_shards": 5,
               "successful_shards": 2,
               "failed_shards": 0,
               "in_progress_shards": 3
            }
         }
      }
   }
}

Response body fields

The following table lists the response body fields.

Get bulk task status

Introduced 3.7

Returns the progress of a bulk replication task. Use the task_id returned by any bulk replication operation to check the task status. Send this request to the follower cluster.

Endpoints

GET /_plugins/_replication/_task_status/{task_id}

Path parameters

The following table lists the available path parameters.

Example response

{
   "operation_type": "bulk_stop_replication",
   "pattern": "my-index-*",
   "start_time": 1717600000000,
   "num_success": 7,
   "num_failed": 1,
   "num_pending": 2,
   "num_cancelled": 0,
   "failed_indices": [
      {
         "index": "my-index-3",
         "failure_reason": "No replication in progress for index:my-index-3"
      }
   ]
}

Response body fields

The following table lists the response body fields.

Cancel bulk task

Introduced 3.7

Cancels a running bulk replication task. Indexes already processed remain in the state applied by the operation; remaining indexes are marked as canceled. Send this request to the follower cluster.

Endpoints

POST /_plugins/_replication/_task_cancel/{task_id}

Path parameters

The following table lists the available path parameters.

Settings

For bulk replication settings, see Bulk replication settings. To learn about updating dynamic settings, see Dynamic settings.

Error responses

The following examples show common error responses you may encounter when using the bulk replication API operations.

Bulk task already running

If you submit a bulk replication operation while another is already in progress, OpenSearch returns a 409 error:

{
   "error": {
      "root_cause": [
         {
            "type": "resource_already_exists_exception",
            "reason": "A bulk replication task is already running. Only one bulk task is allowed at a time."
         }
      ],
      "type": "resource_already_exists_exception",
      "reason": "A bulk replication task is already running. Only one bulk task is allowed at a time."
   },
   "status": 409
}

Wait for the current task to complete or cancel it before submitting a new one.

No indexes match pattern

If no indexes are found matching the specified pattern, OpenSearch returns a 404 error:

{
   "error": {
      "root_cause": [
         {
            "type": "resource_not_found_exception",
            "reason": "No indices found matching pattern: [non-existent-*] on leader cluster"
         }
      ],
      "type": "resource_not_found_exception",
      "reason": "No indices found matching pattern: [non-existent-*] on leader cluster"
   },
   "status": 404
}

Missing leader alias

If you submit a bulk start request without providing a leader_alias, OpenSearch returns a 400 error:

{
   "error": {
      "root_cause": [
         {
            "type": "action_request_validation_exception",
            "reason": "Validation Failed: 1: leader_alias is required for bulk start;"
         }
      ],
      "type": "action_request_validation_exception",
      "reason": "Validation Failed: 1: leader_alias is required for bulk start;"
   },
   "status": 400
}

All indexes failed validation

If all matched indexes are already in the state that the operation requests (for example, you send a pause replication operation to indexes for which replication is already paused), OpenSearch returns a 400 error:

{
   "error": {
      "root_cause": [
         {
            "type": "illegal_argument_exception",
            "reason": "Index bulk-test-1 is already paused; Index bulk-test-2 is already paused"
         }
      ],
      "type": "illegal_argument_exception",
      "reason": "Index bulk-test-1 is already paused; Index bulk-test-2 is already paused"
   },
   "status": 400
}

Task not found

If you attempt to cancel or check a task that does not exist or has already completed, OpenSearch returns a 404 error:

{
   "error": {
      "root_cause": [
         {
            "type": "resource_not_found_exception",
            "reason": "Task a3PiBHTWTTulezYixxCwkw:7709 not found or already completed"
         }
      ],
      "type": "resource_not_found_exception",
      "reason": "Task a3PiBHTWTTulezYixxCwkw:7709 not found or already completed"
   },
   "status": 404
}

Limitations

Note the following limitations:

• The Bulk Replication API does not automatically rerun operations for failed indexes. Rerun the bulk replication operation manually or use the Replication API for individual indexes.
• Bulk replication tasks are transient and do not persist across node restarts. Individual replication tasks initiated by a bulk start operation are persistent.