Update By Query API

Introduced 1.0

The Update by Query API updates all documents in an index that match a specified query. You can update documents without changing their source to pick up mapping changes, or use a script to modify field values based on custom logic.

Use this API in the following scenarios:

• Applying mapping changes to existing documents after adding new fields or changing field types.
• Updating field values across multiple documents based on calculated logic or conditions.
• Incrementing counters or performing bulk calculations on documents that match specific criteria.
• Conditionally deleting documents by setting ctx.op = "delete" in a script.
• Performing no-operation updates by setting ctx.op = "noop" when conditions aren’t met.

When you submit an update by query request, OpenSearch takes a snapshot of the index at the start of the operation and updates matching documents using internal versioning. If a document changes between when the snapshot is taken and when the update operation processes it, a version conflict occurs and the update fails for that document unless you set the conflicts parameter to proceed. When a version conflict doesn’t cause an abort, the document is updated and its version number is incremented. Successfully updated documents are not rolled back even if later operations in the batch fail.

All update and query failures cause the operation to abort and are returned in the failures array of the response. Successful updates persist even after an abort. While the first failure triggers the abort, all failures from the rejected bulk request appear in the failures element, so multiple failed entities may be reported.

OpenSearch retries rejected search or bulk requests up to 10 times with exponential backoff. If the maximum retry limit is reached, the operation halts and returns all failed requests in the response.

Note: OpenSearch cannot update documents with version 0 using this API. The internal versioning system requires that version numbers are greater than 0 in order to track and process update operations correctly.

Endpoints

POST /{index}/_update_by_query

Path parameters

The following table lists the available path parameters.

Query parameters

The following table lists the available query parameters. All query parameters are optional.

Important: When using _source, _source_includes, or _source_excludes in an update by query request, these settings affect not only the response but also the fields available to the update script. If a field is excluded from _source and not explicitly handled in the script, it may be removed from the document during the update operation. To preserve excluded fields, ensure that the script reads and reassigns them as needed.

Request body fields

The request body is optional but typically includes a query to specify which documents to update and a script to define the update logic.

Script operations

Within your update script, you can control what happens to each document by setting ctx.op:

Setting ctx.op to any other value causes an error. Modifying other fields in ctx besides ctx._source and ctx.op also causes an error.

Refreshing shards

Specifying the refresh parameter refreshes all shards involved in the update by query operation after the request completes. This behavior differs from the Update API’s refresh parameter, which only refreshes the shard that received the update request. The Update by Query API does not support the wait_for value for the refresh parameter.

Running update by query asynchronously

To run an update by query operation asynchronously, set the wait_for_completion query parameter to false. OpenSearch performs preflight checks, launches the request, and returns a task ID that you can use to monitor progress or cancel the operation. When running asynchronously, OpenSearch creates a record of the task as a document at .tasks/task/${taskId}. After the task completes, delete the task document to allow OpenSearch to reclaim the space.

Waiting for active shards

The wait_for_active_shards parameter controls how many shard copies must be active before processing the request. The timeout parameter controls how long each write request waits for unavailable shards to become available. These parameters work the same way as in the Bulk API. Because Update by Query uses scrolled searches, you can specify the scroll parameter to control how long the search context remains active. The default scroll time is 5 minutes.

Throttling update requests

To control the rate at which update by query issues batches of update operations, set requests_per_second to any positive decimal number. This pads each batch with a wait time to throttle the rate. Set requests_per_second to -1 to disable throttling.

Throttling uses wait time between batches so that internal scroll requests can be given a timeout that accounts for request padding. The padding time is the difference between the batch size divided by requests_per_second and the time spent writing. By default, the batch size is 1,000, so if requests_per_second is set to 500:

target_time = 1,000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - 0.5 seconds = 1.5 seconds

Because each batch is issued as a single bulk request, large batch sizes cause OpenSearch to create many requests and then wait before starting the next batch. This creates uneven processing patterns with periods of high activity followed by idle waiting.

Slicing for parallel processing

You can use slicing to run update operations in parallel across multiple threads. This approach divides the update operation into independent segments, improving performance for large-scale updates.

Setting slices to auto allows OpenSearch to choose a reasonable number for most indexes. When using automatic slicing or tuning it manually, consider these factors:

• Optimal query performance occurs when you match the slice count to your shard count. However, for indexes with many shards (500 or more), use fewer slices to avoid performance degradation from excessive parallelization overhead. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.
• Update performance scales linearly across available resources with the number of slices.
• Whether query or update performance dominates runtime depends on the documents being updated and available cluster resources.

Example: Updating all documents without changing source

The following example request updates all documents in the index without modifying their source. This is useful for picking up new mapping properties or other mapping changes:

POST /products/_update_by_query?conflicts=proceed

response = client.update_by_query(
  index = "products",
  params = { "conflicts": "proceed" },
  body = { "Insert body here" }
)

Example: Updating documents with a query filter

The following example request updates only electronics products by adding a 10% discount:

POST /products/_update_by_query
{
  "query": {
    "term": {
      "category": "electronics"
    }
  },
  "script": {
    "source": "ctx._source.discount = params.discountPercent",
    "lang": "painless",
    "params": {
      "discountPercent": 0.1
    }
  }
}

response = client.update_by_query(
  index = "products",
  body =   {
    "query": {
      "term": {
        "category": "electronics"
      }
    },
    "script": {
      "source": "ctx._source.discount = params.discountPercent",
      "lang": "painless",
      "params": {
        "discountPercent": 0.1
      }
    }
  }
)

Example: Incrementing a field value

The following example request increments the likes counter for all products from a specific user:

POST /products/_update_by_query
{
  "query": {
    "term": {
      "user_id": "user1"
    }
  },
  "script": {
    "source": "ctx._source.likes++",
    "lang": "painless"
  }
}

response = client.update_by_query(
  index = "products",
  body =   {
    "query": {
      "term": {
        "user_id": "user1"
      }
    },
    "script": {
      "source": "ctx._source.likes++",
      "lang": "painless"
    }
  }
)

Example: Conditionally deleting documents

The following example request deletes out-of-stock products with zero likes:

POST /products/_update_by_query
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "in_stock": false
          }
        },
        {
          "term": {
            "likes": 0
          }
        }
      ]
    }
  },
  "script": {
    "source": "ctx.op = 'delete'",
    "lang": "painless"
  }
}

response = client.update_by_query(
  index = "products",
  body =   {
    "query": {
      "bool": {
        "must": [
          {
            "term": {
              "in_stock": false
            }
          },
          {
            "term": {
              "likes": 0
            }
          }
        ]
      }
    },
    "script": {
      "source": "ctx.op = 'delete'",
      "lang": "painless"
    }
  }
)

Example: Using noop for conditional updates

The following example request increases discount only for products priced above $100, otherwise performs no operation:

POST /products/_update_by_query
{
  "script": {
    "source": "if (ctx._source.price > 100) { ctx._source.discount = 0.15 } else { ctx.op = 'noop' }",
    "lang": "painless"
  }
}

response = client.update_by_query(
  index = "products",
  body =   {
    "script": {
      "source": "if (ctx._source.price > 100) { ctx._source.discount = 0.15 } else { ctx.op = 'noop' }",
      "lang": "painless"
    }
  }
)

Example: Updating from multiple indexes

The following example request updates documents across multiple indexes:

POST /products,inventory/_update_by_query
{
  "query": {
    "match_all": {}
  }
}

response = client.update_by_query(
  index = "products,inventory",
  body =   {
    "query": {
      "match_all": {}
    }
  }
)

Example: Using routing for targeted updates

The following example request limits the update operation to shards with a specific routing value:

POST /products/_update_by_query?routing=user1
{
  "query": {
    "term": {
      "user_id": "user1"
    }
  },
  "script": {
    "source": "ctx._source.likes += 5",
    "lang": "painless"
  }
}

response = client.update_by_query(
  index = "products",
  params = { "routing": "user1" },
  body =   {
    "query": {
      "term": {
        "user_id": "user1"
      }
    },
    "script": {
      "source": "ctx._source.likes += 5",
      "lang": "painless"
    }
  }
)

Example: Using scroll_size to control batch size

The following example request uses a custom scroll batch size of 100 documents:

POST /products/_update_by_query?scroll_size=100
{
  "query": {
    "range": {
      "price": {
        "gte": 50
      }
    }
  },
  "script": {
    "source": "ctx._source.discount = 0.05",
    "lang": "painless"
  }
}

response = client.update_by_query(
  index = "products",
  params = { "scroll_size": "100" },
  body =   {
    "query": {
      "range": {
        "price": {
          "gte": 50
        }
      }
    },
    "script": {
      "source": "ctx._source.discount = 0.05",
      "lang": "painless"
    }
  }
)

Example: Manual slicing for parallel processing

The following example requests manually divide the update operation into two slices for parallel processing:

POST /products/_update_by_query
{
  "slice": {
    "id": 0,
    "max": 2
  },
  "script": {
    "source": "ctx._source.discount = 0.20",
    "lang": "painless"
  }
}

response = client.update_by_query(
  index = "products",
  body =   {
    "slice": {
      "id": 0,
      "max": 2
    },
    "script": {
      "source": "ctx._source.discount = 0.20",
      "lang": "painless"
    }
  }
)

In a separate request, process the second slice:

POST /products/_update_by_query
{
  "slice": {
    "id": 1,
    "max": 2
  },
  "script": {
    "source": "ctx._source.discount = 0.20",
    "lang": "painless"
  }
}

response = client.update_by_query(
  index = "products",
  body =   {
    "slice": {
      "id": 1,
      "max": 2
    },
    "script": {
      "source": "ctx._source.discount = 0.20",
      "lang": "painless"
    }
  }
)

Example: Automatic slicing

The following example request uses automatic slicing to parallelize the update operation across 5 slices:

POST /products/_update_by_query?slices=5&refresh=true
{
  "script": {
    "source": "ctx._source.discount = 0.25",
    "lang": "painless"
  }
}

response = client.update_by_query(
  index = "products",
  params = { "slices": "5", "refresh": "true" },
  body =   {
    "script": {
      "source": "ctx._source.discount = 0.25",
      "lang": "painless"
    }
  }
)

To allow OpenSearch to automatically determine the optimal number of slices, use slices=auto:

POST /products/_update_by_query?slices=auto
{
  "query": {
    "term": {
      "category": "furniture"
    }
  },
  "script": {
    "source": "ctx._source.discount = 0.30",
    "lang": "painless"
  }
}

response = client.update_by_query(
  index = "products",
  params = { "slices": "auto" },
  body =   {
    "query": {
      "term": {
        "category": "furniture"
      }
    },
    "script": {
      "source": "ctx._source.discount = 0.30",
      "lang": "painless"
    }
  }
)

Example response

The following example response shows a successful update by query operation that updated 8 documents:

{
  "took": 39,
  "timed_out": false,
  "total": 8,
  "updated": 8,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "failures": []
}

When using a script with conditional noop operations, the response includes a noops count showing how many documents were skipped:

{
  "took": 55,
  "timed_out": false,
  "total": 8,
  "updated": 4,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 4,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "failures": []
}

When using manual slicing, the response includes a slice_id field indicating which slice was processed:

{
  "took": 12,
  "timed_out": false,
  "slice_id": 0,
  "total": 4,
  "updated": 4,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "failures": []
}

Response body fields

The following table lists all response body fields.

Managing update by query tasks

When you run an update by query operation asynchronously by setting wait_for_completion=false, OpenSearch returns a task ID that you can use to monitor, modify, or cancel the operation.

Retrieving the status of an update by query operation

To retrieve the status of an update by query operation, use the Tasks API:

GET _tasks?detailed=true&actions=*/update/byquery

The response includes the status of all running update by query operations. To retrieve the status of a specific task, use the task ID:

GET _tasks/<task_id>

The response contains detailed information about the operation’s progress:

{
  "nodes": {
    "node_id": {
      "tasks": {
        "task_id": {
          "status": {
            "total": 1000,
            "updated": 450,
            "created": 0,
            "deleted": 0,
            "batches": 5,
            "version_conflicts": 0,
            "noops": 0,
            "retries": 0,
            "throttled_millis": 0
          }
        }
      }
    }
  }
}

The total field represents the total number of operations that the update by query operation expects to perform. You can estimate progress by adding the updated, deleted, and noops fields and comparing the sum to the total field. The operation is complete when their sum equals the total field.

Changing throttling for a running operation

To change the throttling of a running update by query operation, use the Rethrottle API with the task ID:

POST _update_by_query/<task_id>/_rethrottle?requests_per_second=100

Set requests_per_second to any positive decimal value or -1 to disable throttling. Rethrottling that speeds up the operation takes effect immediately. Rethrottling that slows down the operation takes effect after completing the current batch to prevent scroll timeouts.

Canceling an update by query operation

To cancel a running update by query operation, use the task cancel API:

POST _tasks/<task_id>/_cancel

Cancellation should happen quickly but might take a few seconds. The Tasks API continues to list the update by query task until it checks that it has been canceled and terminates itself. When you cancel an update by query operation with slices, OpenSearch cancels each sub-request.