You're viewing version 3.8 of the OpenSearch documentation. This version is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.

Append-only index

An append-only index is an immutable index that only allows document ingestion (appending) while blocking all updates or deletions after initial document creation. When you enable the append-only setting for an index, OpenSearch prevents any modifications to existing documents. You can only add new documents to the index.

When you configure an index as append-only, the following operations return an error:

• Document update call (Update API)
• Document delete call (Delete API)
• Update by query call
• Delete by query call
• Bulk API calls made with the update, delete, or upsert actions
• Bulk API calls containing an index action with a custom document ID

Because an append-only index performs no updates or deletions, it skips soft deletes and version tracking, which reduces storage use and the work done during segment merges. Use an append-only index for data that is not modified after it is ingested, such as logs, metrics, observability data, or security events.

Creating an append-only index

The following request creates a new index named my-append-only-index with all updates disabled:

PUT /my-append-only-index
{
  "settings": {
    "index.append_only.enabled": true
  }
}

After an index is set to append-only, it cannot be changed to another index type.

To append data from an existing index to a new append-only index, use the Reindex API. Because append-only indexes don’t support custom document IDs, you need to set the ctx._id of the source index to null. This allows documents to be added through reindexing.

The following example reindexes documents from a source index (my-source-index) into the new append-only index. Create the source index first:

POST /my-source-index/_doc?refresh=true
{
  "message": "login attempt"
}

Then reindex its documents into the append-only index:

POST /_reindex
{
  "source": {
    "index": "my-source-index"
  },
  "dest": {
    "index": "my-append-only-index"
  },
  "script": {
    "source": "ctx._id = null",
    "lang": "painless"
  }
}

Adaptive shard selection for bulk indexing

Introduced 3.5

For append-only indexes, OpenSearch automatically generates a random _id for write routing when you don’t explicitly specify one. In bulk writing, a single bulk entry may be split into dozens of sub-bulks and dispatched to different shards, which leads to significant long-tail latency and markedly degrades write performance.

Adaptive shard selection guarantees that all sub-bulks of a single bulk entry are routed to the same shard, thereby achieving a substantial boost in bulk write performance.

The index.bulk.adaptive_shard_selection.enabled setting is dynamic, so you can enable it on an existing append-only index:

PUT /my-append-only-index/_settings
{
  "index.bulk.adaptive_shard_selection.enabled": "true"
}

You can also set it when you create the index:

PUT /my-new-append-only-index
{
  "settings": {
    "index.append_only.enabled": "true",
    "index.bulk.adaptive_shard_selection.enabled": "true"
  }
}

For more information, see Dynamic settings.