Playbook: Elasticsearch 6.8 to OpenSearch 3.5 on GCP
This playbook describes migrating a self-managed Elasticsearch 6.8 cluster to an OpenSearch 3.5 cluster on Google Cloud Platform (GCP) using Migration Assistant running on Google Kubernetes Engine (GKE). The source and target are accessed over the migration cluster’s virtual private cloud (VPC) network, and snapshots are stored in Google Cloud Storage.
Use this playbook when you run search infrastructure on GCP and need a repeatable, snapshot-based migration with optional zero-downtime Capture and Replay.
The playbook includes every migration stage and assumes that Migration Assistant is already deployed on GKE. For information about deploying Migration Assistant, see Deploy on Google Kubernetes Engine (GKE).
Example placeholders
Replace the following placeholders with values from your environment.
| Placeholder | Meaning |
|---|---|
<PROJECT_ID> | The GCP project ID hosting the migration environment. |
<REGION> | The GCP region, such as us-central1. |
<CLUSTER_NAME> | The GKE cluster name (from terraform output -raw cluster_name). |
<SOURCE_ENDPOINT> | The Elasticsearch endpoint reachable from the migration VPC. |
<SOURCE_USERNAME>, <SOURCE_PASSWORD> | The source cluster credentials. |
<TARGET_ENDPOINT> | The OpenSearch 3.5 endpoint reachable from the migration VPC. |
<TARGET_USERNAME>, <TARGET_PASSWORD> | The target cluster credentials. |
<PILOT_INDEX_NAME> | A small, noncritical index for the pilot run. |
The GKE Terraform module creates the Cloud Storage bucket for snapshots and makes it available to the workflow through the migrations-default-gcs-config ConfigMap. You do not need to configure the bucket manually. This playbook refers to the bucket as <BUCKET_NAME>.
Estimated timing
The following table provides estimated durations for each phase of the migration.
| Phase | Typical duration |
|---|---|
Source preparation (installing the repository-gcs plugin, configuring Cloud Storage access, registering the snapshot repository) | 15–25 minutes |
| Deploying Migration Assistant on GKE | 15–25 minutes |
| Building the workflow configuration and testing connectivity | 5–10 minutes |
| Running a pilot migration | 15–20 minutes |
| Running the full backfill-only migration | Depends on data size and target ingest capacity |
| Running the full zero-downtime migration | Backfill duration plus replay synchronization time |
For zero-downtime migrations, replay duration depends on the volume of traffic accumulated during backfill and the speedupFactor configured for the Replayer. A typical starting value is 1.5 to 2.0. Monitor the target cluster to confirm it can sustain the replay rate.
Before you start
In addition to the general playbook prerequisites, verify the following before starting the migration.
Source cluster requirements
Your source cluster must meet all of the following requirements:
- It is reachable from the VPC where Migration Assistant runs.
- You know the source endpoint and authentication method.
- It can write snapshots to Google Cloud Storage.
- It contains the
repository-gcsplugin. Elasticsearch 8.0 and later bundle Cloud Storage support, but Elasticsearch 6.8 requires the plugin to be installed on every node.
If the source uses self-signed certificates, plan to set allowInsecure: true in the workflow configuration.
Target cluster requirements
Your OpenSearch 3.5 cluster must meet all of the following requirements:
- It already exists.
- It is reachable from the migration VPC.
- You know the target endpoint and authentication method.
Infrastructure requirements
The following infrastructure is required:
- Migration Assistant deployed on GKE. See Deploy on Google Kubernetes Engine (GKE).
- The
gcloudCLI andkubectl, authenticated for the migration cluster. - If the source or target must be reached over private connectivity, see Private networking on GKE.
Step 1: Choose the migration type
Select one of the following migration types before configuring the workflow.
Option A: Planned downtime
For a planned downtime migration, follow these steps:
- Pause writes to Elasticsearch.
- Create a snapshot.
- Migrate metadata.
- Backfill documents.
- Validate the target.
- Point clients to OpenSearch.
If you are unsure which approach to select, use planned downtime because it involves fewer components and lower risk.
Option B: Zero downtime
For a zero-downtime migration, follow these steps:
- Start the capture.
- Route clients to the capture proxy.
- Create a snapshot.
- Migrate metadata.
- Backfill documents.
- Replay captured traffic until the target is synchronized with the source.
- Validate the target.
- Switch clients to OpenSearch.
If your application depends on auto-generated document IDs, test Capture and Replay thoroughly before selecting the zero-downtime approach. Auto-generated IDs are not preserved during replay.
Step 2: Prepare the source cluster
Backfill reads from a snapshot, so the source cluster needs the Cloud Storage snapshot plugin, access to the bucket, and a registered repository.
Install the repository-gcs plugin
Migration Assistant uses snapshots for backfill, so the source must support Cloud Storage snapshots. Verify the current plugin state:
curl -sk -u <SOURCE_USERNAME>:<SOURCE_PASSWORD> https://<SOURCE_ENDPOINT>:9200/_cat/plugins?v
If repository-gcs is missing, install it on every source node and restart:
bin/elasticsearch-plugin install --batch repository-gcs
If your source runs in Kubernetes, a common pattern is to install the plugin using an init container so it persists across pod restarts. For general information, see the Elasticsearch documentation for the repository-gcs plugin.
Grant the source access to Cloud Storage
The source cluster writes snapshot files directly to the Cloud Storage bucket, so it needs credentials with write access to that bucket. Configure the repository-gcs client credentials on the source according to your environment (for example, a service account key added to the Elasticsearch keystore). Grant that identity read and write access to objects in the migration bucket (roles/storage.objectAdmin).
Register a snapshot repository
Read the bucket name that the GKE Terraform module provisioned, then register it as a snapshot repository on the source. From a shell with access to the migration cluster, read the bucket name from the ConfigMap:
kubectl get configmap migrations-default-gcs-config -n ma \
-o jsonpath='{.data.GCS_BUCKET_NAME}'
Register the repository on the source cluster (this is the same gcs repository body that the workflow requires):
curl -sk -u <SOURCE_USERNAME>:<SOURCE_PASSWORD> \
-X PUT "https://<SOURCE_ENDPOINT>:9200/_snapshot/migration" \
-H "Content-Type: application/json" \
-d '{"type":"gcs","settings":{"bucket":"<BUCKET_NAME>","base_path":"snapshots"}}'
A successful response returns {"acknowledged":true}.
Step 3: Connect to the Migration Console and create secrets
Open a shell in the Migration Console pod:
kubectl exec -it migration-console-0 -n ma -- /bin/bash
Create Kubernetes secrets for the source and target credentials, which the workflow references by name:
kubectl create secret generic source-credentials \
--from-literal=username=<SOURCE_USERNAME> \
--from-literal=password=<SOURCE_PASSWORD> -n ma
kubectl create secret generic target-credentials \
--from-literal=username=<TARGET_USERNAME> \
--from-literal=password=<TARGET_PASSWORD> -n ma
Step 4: Build the workflow configuration
Load the version-matched sample and edit it:
workflow configure sample --load
workflow configure edit
The following configuration is the backfill-only (planned downtime) workflow. It sets the source and target endpoints, references the credential secrets, and points the snapshot repository at the Cloud Storage bucket:
sourceClusters:
source:
endpoint: https://<SOURCE_ENDPOINT>:9200
allowInsecure: true
version: ES 6.8.23
authConfig:
basic:
secretName: source-credentials
snapshotInfo:
repos:
migration:
repoPathUri: gs://<BUCKET_NAME>/snapshots
snapshots:
snap1:
config:
createSnapshotConfig: {}
repoName: migration
targetClusters:
target:
endpoint: https://<TARGET_ENDPOINT>:9200
allowInsecure: true
authConfig:
basic:
secretName: target-credentials
snapshotMigrationConfigs:
- fromSource: source
toTarget: target
perSnapshotConfig:
snap1:
- metadataMigrationConfig: {}
documentBackfillConfig:
podReplicas: 1
maxConnections: 50
documentsPerBulkRequest: 10000
For a zero-downtime migration, add a traffic section with a capture proxy and a Replayer. On GKE, the capture proxy’s load balancer is internal by default, so its ingress stays on the VPC network:
traffic:
proxies:
capture-proxy:
source: source
proxyConfig:
listenPort: 9201
podReplicas: 2
replayers:
replay1:
fromProxy: capture-proxy
toTarget: target
dependsOnSnapshotMigrations:
- source: source
snapshot: snap1
replayerConfig:
speedupFactor: 2
# Cap in-flight requests so a throttled target cannot drive the
# replayer out of memory. Raise gradually while watching the target.
maxConcurrentRequests: 500
Set version to your exact source version. Use allowInsecure: true only when the source or target uses self-signed certificates.
When you save and exit, the CLI validates the configuration against the workflow schema and prompts you to fix any errors.
Step 5: Test connectivity
Confirm that the Migration Console can reach both clusters:
console clusters connection-check
Resolve any connectivity or authentication failure before continuing.
Step 6: Run the pilot migration
Run a small pilot before the full migration to catch mapping, authentication, and throughput issues early. Restrict the pilot to a small, noncritical index (<PILOT_INDEX_NAME>), then submit and monitor:
workflow submit
workflow manage
Step 7: Validate the pilot migration
Compare document counts and sample a few documents on the target:
console clusters cat-indices
console clusters curl target /<PILOT_INDEX_NAME>/_count
console clusters curl target "/<PILOT_INDEX_NAME>/_search?size=5&pretty"
Step 8: Run the full migration
After the pilot succeeds, widen the configuration to the full index set and submit again.
Planned downtime path
- Pause writes to the source.
- Run
workflow submitand monitor by runningworkflow manage. - Wait for the workflow to reach
Succeeded. - Continue to validation.
Zero-downtime path
- Confirm capture is running and clients are routed through the capture proxy so live writes are recorded.
- Run
workflow submitand monitor by runningworkflow manage. - After backfill completes, the Replayer drains captured traffic to the target. Monitor until the target is synchronized with the source and the document counts stabilize.
- Continue to validation.
For large migrations, consider applying target-side write tuning during the load: increase primary shards on hot indexes, and set refresh_interval: -1 and number_of_replicas: 0 while loading, then restore them afterward. Watch the target’s write thread pool for rejections and lower Replayer concurrency if rejections appear.
Step 9: Validate and switch traffic
Validate the target before switching clients:
console clusters cat-indices
console clusters curl target /_cat/count?h=count
Confirm per-index document counts match between source and target. When you are confident, switch clients to the OpenSearch target by updating DNS, load balancer configuration, or application connection strings.
Step 10: Keep the source available as a fallback
Keep the source cluster available as a rollback option until production traffic has been stable on the target (24–72 hours is a common window).
Step 11: Remove the migration infrastructure
When you no longer need Migration Assistant for rollback, replay, or comparison, remove it. Do not remove it immediately after cutover. For the readiness checklist and removal commands, see Removing migration infrastructure.
Troubleshooting
The following issues occur most often on the GCP path.
Snapshot registration fails with repository type [gcs] does not exist
The source is missing the repository-gcs plugin. Install it on every source node and restart (see Step 2).
Source cannot write to the bucket
Confirm the source’s repository-gcs client credentials are configured and that the identity has object read and write access to the migration bucket.
Pilot succeeds from the console but the workflow fails
The Migration Console and the workflow executor pods can have different network or identity access. Confirm both can reach the source and target and that any required credentials are available to both.