Link Search Menu Expand Document Documentation Menu
Documentation

Migrating to remote-backed storage

Introduced 2.15

Remote-backed storage offers a new way to protect against data loss by automatically creating backups of all index transactions and sending them to remote storage. To use this feature, segment replication must be enabled.

You can migrate a document-replication-based cluster to remote-backed storage through the rolling upgrade mechanism.

Rolling upgrades, sometimes referred to as node replacement upgrades, can be performed on running clusters. Nodes are individually stopped and migrated in place. Alternatively, nodes can be stopped and replaced, one at a time, by remote-backed hosts. During this process you can continue to index and query data in your cluster.

While the cluster contains both remote-backed and non-remote nodes, OpenSearch restricts where shard copies can be allocated. A replica can be allocated to a remote-backed node only if its primary shard is on a remote-backed node, and a shard that is stored in a remote-backed index can be allocated only to remote-backed nodes. As a result, some replica shards remain unassigned during migration, and some shards can have only one active copy. To keep every shard available, move shards off each node before you stop it.

Preparing to migrate

Review Upgrading OpenSearch for recommendations about backing up your configuration files and creating a snapshot of the cluster state and indexes before you make any changes to your OpenSearch cluster.

Before migrating to remote-backed storage, upgrade to OpenSearch 2.15 or later.

Before upgrading to OpenSearch 2.15, take a cluster snapshot and store it remotely. OpenSearch 2.15 nodes cannot revert to document replication. If a migration needs to be undone, perform a fresh OpenSearch installation and restore from the remote snapshot. Storing the snapshot remotely allows you to retrieve and restore it if issues arise during migration.

Performing the upgrade

  1. Verify the health of your OpenSearch cluster before you begin using the Cluster Health API. Resolve any index or shard allocation issues prior to upgrading to ensure that your data is preserved. A status of green indicates that all primary and replica shards are allocated. You can query the _cluster/health API endpoint using a command similar to the following:

    GET "/_cluster/health?pretty"

    You should receive a response similar to the following:

    {
    "cluster_name":"opensearch-dev-cluster",
    "status":"green",
    "timed_out":false,
    "number_of_nodes":4,
    "number_of_data_nodes":4,
    "active_primary_shards":1,
    "active_shards":4,
    "relocating_shards":0,
    "initializing_shards":0,
    "unassigned_shards":0,
    "delayed_unassigned_shards":0,
    "number_of_pending_tasks":0,
    "number_of_in_flight_fetch":0,
    "task_max_waiting_in_queue_millis":0,
    "active_shards_percent_as_number":100.0
}

  2. Set the cluster.remote_store.compatibility_mode setting to mixed to allow remote-backed storage nodes to join the cluster:

    PUT "/_cluster/settings?pretty"
{
    "persistent": {
        "cluster.remote_store.compatibility_mode": "mixed"
    }
}

    You should receive a response similar to the following:

    {
  "acknowledged" : true,
  "persistent" : {
    "cluster" : {
      "remote_store" : {
        "compatibility_mode" : "mixed"
      }
    }
  },
  "transient" : { }
}

  3. Review your cluster and identify the first node to be migrated.

  4. Exclude the node from shard allocation so that OpenSearch moves its shards to other nodes. Replace <node-name> with the name of the node:

    PUT "/_cluster/settings?pretty"
{
    "persistent": {
        "cluster.routing.allocation.exclude._name": "<node-name>"
    }
}

  5. Wait for shard relocation to finish:

    GET "/_cluster/health?wait_for_no_relocating_shards=true&wait_for_no_initializing_shards=true&pretty"

    Confirm that the response contains "timed_out" : false. If the response contains "timed_out" : true, shards are still relocating. Repeat the request until relocation finishes.

  6. List the shards in the cluster and review the shards that remain on the node:

    GET "/_cat/shards?v&h=index,shard,prirep,state,node&s=index,shard"

    Some shards can remain on the node because no other node is eligible to hold them. For example, a replica remains on the node when its primary shard is on a non-remote node and every other non-remote node already holds a copy of the shard. Before you stop the node, confirm that each shard remaining on the node has a STARTED copy on another node. Stopping a node that holds the only copy of a shard makes that shard unavailable until the node rejoins the cluster. To learn why a shard cannot be moved, use the Cluster Allocation Explain API.

  7. Disable shard replication to prevent shard replicas from being created while the node is offline:

    PUT "/_cluster/settings?pretty"
{
    "persistent": {
        "cluster.routing.allocation.enable": "primaries"
    }
}

    You should receive a response similar to the following:

    {
  "acknowledged" : true,
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "primaries"
        }
      }
    }
  },
  "transient" : { }
}

  8. Perform the following flush operation on the cluster to commit transaction log entries to the Lucene index:

    POST "/_flush?pretty"

    You should receive a response similar to the following:

    {
  "_shards" : {
    "total" : 4,
    "successful" : 4,
    "failed" : 0
  }
}

  9. Provide the remote store repository details as node attributes in opensearch.yml, as shown in the following example:

    # Repository name
node.attr.remote_store.segment.repository: my-repo-1
node.attr.remote_store.translog.repository: my-repo-2
node.attr.remote_store.state.repository: my-repo-3
   
# Segment repository settings
node.attr.remote_store.repository.my-repo-1.type: s3
node.attr.remote_store.repository.my-repo-1.settings.bucket: <Bucket Name 1>
node.attr.remote_store.repository.my-repo-1.settings.base_path: <Bucket Base Path 1>
node.attr.remote_store.repository.my-repo-1.settings.region: us-east-1
   
# Translog repository settings
node.attr.remote_store.repository.my-repo-2.type: s3
node.attr.remote_store.repository.my-repo-2.settings.bucket: <Bucket Name 2>
node.attr.remote_store.repository.my-repo-2.settings.base_path: <Bucket Base Path 2>
node.attr.remote_store.repository.my-repo-2.settings.region: us-east-1
   
# Enable Remote cluster state cluster setting
cluster.remote_store.state.enabled: true
   
# Remote cluster state repository settings
node.attr.remote_store.repository.my-repo-3.type: s3
node.attr.remote_store.repository.my-repo-3.settings.bucket: <Bucket Name 3>
node.attr.remote_store.repository.my-repo-3.settings.base_path: <Bucket Base Path 3>
node.attr.remote_store.repository.my-repo-3.settings.region: <Bucket region>

  10. Stop the node you are migrating. Do not delete the volume associated with the container when you delete the container. The new OpenSearch container will use the existing volume. Deleting the volume will result in data loss.

  11. Deploy a new container running the same version of OpenSearch and mapped to the same volume as the container you deleted.

  12. Query the _cat/nodes endpoint after OpenSearch is running on the new node to confirm that it has joined the cluster:

    GET "/_cat/nodes?v&h=name,node.role,master"

  13. Reenable shard replication:

    PUT "/_cluster/settings?pretty"
{
    "persistent": {
        "cluster.routing.allocation.enable": "all"
    }
}

    You should receive a response similar to the following:

    {
  "acknowledged" : true,
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"
        }
      }
    }
  },
  "transient" : { }
}

  14. Wait for shard recovery to finish:

    GET "/_cluster/health?wait_for_no_initializing_shards=true&wait_for_no_relocating_shards=true&pretty"

    Confirm that the response contains "timed_out" : false. If the response contains "timed_out" : true, repeat the request until recovery finishes. The cluster status can remain yellow until all nodes are migrated because replicas that cannot be allocated during migration remain unassigned.

  15. After the first node joins the cluster as a remote-backed node, set cluster.migration.direction to remote_store. This setting allocates new indexes to remote-backed data nodes and allows primary shards to relocate from non-remote nodes to remote-backed nodes. Set this setting only after a remote-backed node has joined the cluster. If no remote-backed node is available, primary shards of new indexes cannot be allocated, and the cluster status becomes red:

    PUT "/_cluster/settings?pretty"
{
    "persistent": {
        "cluster.migration.direction": "remote_store"
    }
}

    You should receive a response similar to the following:

    {
  "acknowledged" : true,
  "persistent" : {
    "cluster" : {
      "migration" : {
        "direction" : "remote_store"
      }
    }
  },
  "transient" : { }
}

  16. Repeat steps 4 through 14 for each remaining node in your cluster. In step 4, specify the name of the next node. For the last non-remote node, OpenSearch moves all of its shards to remote-backed nodes, so the node holds no shards when you stop it.

  17. Remove the node exclusion:

    PUT "/_cluster/settings?pretty"
{
    "persistent": {
        "cluster.routing.allocation.exclude._name": null
    }
}

  18. Confirm that the cluster is healthy by using the Cluster Health API, as shown in the following command:

    GET "/_cluster/health?wait_for_status=green&pretty"

    You should receive a response similar to the following:

    {
  "cluster_name" : "opensearch-dev-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "discovered_master" : true,
  "active_primary_shards" : 1,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

  19. Clear the remote_store.compatibility_mode and migration.direction settings by using the following command so that non-remote nodes are not allowed to join the cluster:

    PUT "/_cluster/settings?pretty"
{
    "persistent": {
        "cluster.remote_store.compatibility_mode": null,
        "cluster.migration.direction": null
    }
}

    You should receive a response similar to the following:

    {
  "acknowledged" : true,
  "persistent" : { },
  "transient" : { }
}

The migration to the remote store is now complete.

Use the following cluster settings to enable migration to a remote-backed cluster.

Field Data type Description
cluster.remote_store.compatibility_mode String When set to strict, only allows the creation of either non-remote or remote nodes, depending on the initial cluster type. When set to mixed, allows both remote and non-remote nodes to join the cluster. Default is strict.
cluster.migration.direction String When set to remote_store, creates new shards only on remote-backed storage nodes and allows primary shards to relocate from non-remote nodes to remote-backed nodes. Default is None.
350 characters left

Have a question? .

Want to contribute? or .