ISM supported operations

ISM supports the following operations:

• Force merge
• Read only
• Read write
• Replica count
• Shrink
• Close
• Open
• Delete
• Rollover
• Notification
• Snapshot
• Convert index to remote
• Index priority
• Allocation
• Rollup
• Stop replication
• Search only

Force merge

Reduces the number of Lucene segments by merging the segments of individual shards. This operation attempts to set the index to a read-only state before starting the merging process.

The following example merges the segments of each shard into a single segment:

{
  "force_merge": {
    "max_num_segments": 1
  }
}

Read only

Sets a managed index to be read only.

The read_only operation takes no parameters:

{
  "read_only": {}
}

Set the index setting index.blocks.write to true for a managed index.

The index.blocks.write block does not prevent the index from refreshing.

Read write

Sets a managed index to be writeable.

The read_write operation takes no parameters:

{
  "read_write": {}
}

Replica count

Sets the number of replicas to assign to an index.

The following example assigns two replicas to the index:

{
  "replica_count": {
    "number_of_replicas": 2
  }
}

For information about setting replicas, see Primary and replica shards.

Shrink

Allows you to reduce the number of primary shards in your indexes. With this action, you can specify:

• The number of primary shards that the target index should contain.
• A max shard size for the primary shards in the target index.
• Specify a percentage to shrink the number of primary shards in the target index.

The following example shrinks the index to one primary shard, names the target index by appending _shrunken to the source index name, and adds the my-alias alias:

"shrink": {
    "num_new_shards": 1,
    "target_index_name_template": {
        "source": "{{ctx.index}}_shrunken"
    },
    "aliases": [
      {
        "my-alias": {}
      }
    ],
    "switch_aliases": true,
    "force_unsafe": false
}

If you want to add aliases to the action, the parameter must include an array of alias objects, as in the following example:

"aliases": [
  {
    "my-alias": {}
  },
  {
    "my-second-alias": {
      "is_write_index": false,
      "filter": {
        "multi_match": {
          "query": "QUEEN",
          "fields": ["speaker", "text_entry"]
        }
      },
      "index_routing" : "1",
      "search_routing" : "1"
    }
  }
]

Close

Closes the managed index.

The close operation takes no parameters:

{
  "close": {}
}

Closed indexes remain on disk, but consume no CPU or memory. You can’t read from, write to, or search closed indexes.

Closing an index is a good option if you need to retain data for longer than you need to actively search it and have sufficient disk space on your data nodes. If you need to search the data again, reopening a closed index is simpler than restoring an index from a snapshot.

Open

Opens a managed index.

The open operation takes no parameters:

{
  "open": {}
}

Delete

Deletes a managed index.

The delete operation takes no parameters:

{
  "delete": {}
}

Rollover

Rolls an alias over to a new index when the managed index meets one of the rollover conditions.

IMPORTANT

ISM checks the conditions for operations on every execution of the policy based on the set interval, not continuously. The rollover will be performed if the value has reached or has exceeded the configured limit when the check is performed. For example, with min_size configured to a value of 100 GiB, ISM might check the index at 99 GiB and not perform the rollover. However, if the index has grown past the limit by the next check (for example, to 105 GiB), the operation is performed.

If you need to skip the rollover action, you can set the index setting index.plugins.index_state_management.rollover_skip to true. For example, if you receive the error message “Missing alias or not the write index…”, you can set the index.plugins.index_state_management.rollover_skip parameter to true and retry to skip the rollover action.

The index format must match the pattern: ^.*-\d+$. For example, (logs-000001). Set index.plugins.index_state_management.rollover_alias as the alias to rollover.

The rollover operation has the following parameters, all of which are optional.

Conditions set directly on the rollover object are combined with a logical OR, so the rollover occurs as soon as one of them is met. The following rollover action rolls the index over when the index is at least 7 days old or at least 50 GiB in size:

{
  "rollover": {
    "min_index_age": "7d",
    "min_size": "50gb"
  }
}

To require that several conditions be met together, use any_of. This parameter takes a list of condition groups. The conditions within a group are combined with AND, and the groups are combined with OR, so the rollover occurs when every condition in at least one group is met.

The following rollover action rolls the index over when it is at least 7 days old and at least 50 GiB in size, or when it reaches 100,000,000 documents:

{
  "rollover": {
    "any_of": [
      {
        "min_index_age": "7d",
        "min_size": "50gb"
      },
      {
        "min_doc_count": 100000000
      }
    ]
  }
}

In a mixed-version cluster, every node must be running OpenSearch 3.7 or later to evaluate grouped conditions. Nodes running earlier versions do not process any_of.

Notification

Sends you a notification.

The destination system must return a response otherwise the notification operation throws an error.

Example 1: Chime notification

The following notification operation sends a message to an Amazon Chime webhook:

{
  "notification": {
    "destination": {
      "chime": {
        "url": "<url>"
      }
    },
    "message_template": {
      "source": "the index is {{ctx.index}}"
    }
  }
}

Example 2: Custom webhook notification

The following notification operation sends a message to a custom webhook:

{
  "notification": {
    "destination": {
      "custom_webhook": {
        "url": "https://<your_webhook>"
      }
    },
    "message_template": {
      "source": "the index is {{ctx.index}}"
    }
  }
}

Example 3: Slack notification

The following notification operation sends a message to a Slack webhook:

{
  "notification": {
    "destination": {
      "slack": {
        "url": "https://hooks.slack.com/services/xxx/xxxxxx"
      }
    },
    "message_template": {
      "source": "the index is {{ctx.index}}"
    }
  }
}

You can use ctx variables in your message to represent a number of policy parameters based on the past executions of your policy. For example, if your policy has a rollover action, you can use {{ctx.action.name}} in your message to represent the name of the rollover.

The following ctx variable options are available for every policy:

Guaranteed variables

Snapshot

Back up your cluster’s indexes and state. For more information about snapshots, see Take and restore snapshots.

The snapshot operation has the following parameters.

The following example takes a snapshot of the index in the my_backup repository and names the snapshot using the index UUID:

{
  "snapshot": {
    "repository": "my_backup",
    "snapshot": "{{ctx.indexUuid}}"
  }
}

Convert index to remote

Converts an existing index into a searchable snapshot by restoring it from a remote snapshot repository. This action reduces storage costs by moving infrequently accessed data to remote storage while keeping it searchable. Set delete_original_index to true to remove the original index once the restore request is accepted, so that only the remote snapshot-backed index remains.

The convert_index_to_remote operation has the following parameters.

Prerequisites

Before using the convert_index_to_remote action, ensure the following:

• A remote repository (S3, Azure, or GCS) is registered and accessible.
• A snapshot of the index exists in the specified repository, typically created using the snapshot action.
• The repository name matches the one used in the snapshot action.

Usage notes

Note the following to ensure a smooth and predictable conversion when restoring an index as a searchable snapshot:

• The original index is deleted after the remote snapshot restore is successfully accepted only if you set delete_original_index to true. By default, the original index remains alongside the searchable snapshot version.
• The repository name used in the convert_index_to_remote operation must match the repository name specified during the snapshot action.
• Each object in the actions array holds one action. Putting snapshot and convert_index_to_remote in the same object is accepted, but only one of them is stored, so the snapshot is never taken. List each one in its own object.
• You can reference the snapshot using Mustache variables like {{ctx.index}} or {{ctx.indexUuid}} for dynamic naming.
• Consider your cluster’s capacity when setting number_of_replicas. If there aren’t enough eligible nodes for replica restoration, the cluster may enter a yellow state.

Basic example

The following example shows a basic conversion using the minimum required parameters. The snapshot action creates the snapshot that convert_index_to_remote then restores, so each one is a separate object in the actions array:

"actions": [
  {
    "snapshot": {
      "repository": "my_backup",
      "snapshot": "{{ctx.index}}"
    }
  },
  {
    "convert_index_to_remote": {
      "repository": "my_backup",
      "snapshot": "{{ctx.index}}"
    }
  }
]

Advanced configuration example

The following example demonstrates using all available configuration options. This configuration includes aliases, ignores certain index settings during restore, and configures two replicas for the searchable snapshot:

{
   "convert_index_to_remote": {
      "repository": "my_backup",
      "snapshot": "daily-snapshot",
      "include_aliases": true,
      "ignore_index_settings": "index.refresh_interval,index.number_of_replicas",
      "number_of_replicas": 0,
      "rename_pattern": "remote_$1"
   }
}

Complete policy example

The following policy moves indexes older than 30 days to searchable snapshots with optimized settings for cost efficiency:

{
  "policy": {
    "description": "Convert old indexes to searchable snapshots",
    "default_state": "active",
    "states": [
      {
        "name": "active",
        "actions": [],
        "transitions": [
          {
            "state_name": "archive",
            "conditions": {
              "min_index_age": "30d"
            }
          }
        ]
      },
      {
        "name": "archive",
        "actions": [
          {
            "snapshot": {
              "repository": "remote-repo",
              "snapshot": "{{ctx.index}}"
            }
          },
          {
            "convert_index_to_remote": {
              "repository": "remote-repo",
              "snapshot": "{{ctx.index}}",
              "include_aliases": true,
              "ignore_index_settings": "index.refresh_interval,index.number_of_replicas",
              "number_of_replicas": 0
            }
          }
        ],
        "transitions": []
      }
    ]
  }
}

Index priority

Set the priority for the index in a specific state. Unallocated shards of indexes are recovered in the order of their priority, whenever possible. The indexes with higher priority values are recovered first followed by the indexes with lower priority values.

The index_priority operation has the following parameter.

The following example sets the index priority to 50:

"actions": [
  {
    "index_priority": {
      "priority": 50
    }
  }
]

Allocation

Allocate the index to a node with a specific attribute set like this. For example, setting require to warm moves your data only to “warm” nodes.

The allocation operation has the following parameters. At least one of require, include, or exclude must be specified.

The following example allocates the index to nodes whose temp attribute is set to warm:

"actions": [
  {
    "allocation": {
      "require": { "temp": "warm" }
    }
  }
]

Rollup

Index rollup lets you periodically reduce data granularity by rolling up old data into summarized indexes. Define the job in an ism_rollup object. For the fields it accepts, see Create or update an index rollup job.

Rollup jobs can be continuous or non-continuous. A rollup job created using an ISM policy can only be non-continuous.

The following policy rolls the opensearch_dashboards_sample_data_ecommerce fields up into hourly buckets in a target index:

PUT _plugins/_ism/policies/sample_rollup_policy
{
    "policy": {
        "description": "Sample rollup" ,
        "default_state": "rollup",
        "states": [
            {
                "name": "rollup",
                "actions": [
                    {
                        "rollup": {
                            "ism_rollup": {
                                "description": "Creating rollup through ISM",
                                "target_index": "target",
                                "target_index_settings":{
                                    "index.number_of_shards": 1,
                                    "index.number_of_replicas": 1,
                                    "index.codec": "best_compression"
                                 },
                                "page_size": 1000,
                                "dimensions": [
                                    {
                                        "date_histogram": {
                                            "fixed_interval": "60m",
                                            "source_field": "order_date",
                                            "target_field": "order_date",
                                            "timezone": "America/Los_Angeles"
                                        }
                                    },
                                    {
                                        "terms": {
                                            "source_field": "customer_gender",
                                            "target_field": "customer_gender"
                                        }
                                    },
                                    {
                                        "terms": {
                                            "source_field": "day_of_week",
                                            "target_field": "day_of_week"
                                        }
                                    }
                                ],
                                "metrics": [
                                    {
                                        "source_field": "taxless_total_price",
                                        "metrics": [
                                            {
                                                "sum": {}
                                            }
                                        ]
                                    },
                                    {
                                        "source_field": "total_quantity",
                                        "metrics": [
                                            {
                                                "avg": {}
                                            },
                                            {
                                                "max": {}
                                            }
                                        ]
                                    }
                                ]
                            }
                        }
                    }
                ],
                "transitions": []
            }
        ]
    }
}

To create a rollup job in OpenSearch Dashboards, see Creating a rollup job.

Stop replication

Stops replication and converts the follower index to a regular index.

The stop_replication operation takes no parameters:

{
  "stop_replication": {}
}

When cross-cluster replication is enabled, the follower index becomes read-only, preventing all write operations. To manage replicated indexes on a follower cluster, you can perform the stop_replication action before performing other write operations. For example, you can define a policy that first runs stop_replication and then deletes the index by running a delete action.

If security is enabled, in addition to stop replication permissions, you must have the indices:internal/plugins/replication/index/stop permission in order to use the stop_replication action.

Search only

When an index enters search_only mode, OpenSearch removes its primary and regular replica shards while retaining search replicas for query operations. All write operations to the index are blocked. This is useful for log lifecycle management where older indexes no longer need write capability but should remain searchable.

This action requires the following prerequisites:

• Remote store must be enabled on the cluster.
• Segment replication must be enabled on the index.
• Search replicas must be configured on the index.

For more information about search-only mode and reader/writer separation, see Separate index and search workloads.

Set an index to search-only mode using the following action:

{
  "search_only": {}
}

If the index is already in search-only mode, the action completes successfully without making any changes.

You can manually enable or disable search_only mode outside of ISM policies by calling the Scale API.

The following example policy transitions an index to search_only mode after 7 days:

PUT _plugins/_ism/policies/hot-warm-search-only
{
  "policy": {
    "description": "Move indexes to search-only mode after 7 days",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          {
            "state_name": "warm",
            "conditions": {
              "min_index_age": "7d"
            }
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "search_only": {}
          }
        ],
        "transitions": []
      }
    ]
  }
}