Link Search Menu Expand Document Documentation Menu
Documentation

Transform dense_vector fields to knn_vector

This guide explains how Migration Assistant automatically handles the transformation of Elasticsearch’s dense_vector field type to OpenSearch’s knn_vector field type during migration.

Overview

The dense_vector field type was introduced in Elasticsearch 7.x for storing dense vectors used in machine learning and similarity search applications. When migrating from Elasticsearch 7.x to OpenSearch, Migration Assistant automatically converts dense_vector fields to OpenSearch’s equivalent knn_vector type.

This transformation includes mapping the vector configuration parameters and enabling the necessary OpenSearch k-NN plugin settings.

To determine whether an Elasticsearch cluster uses dense_vector field types, make a call to your source cluster’s GET /_mapping API. In the migration console, run console clusters curl source_cluster "/_mapping". If you see "type":"dense_vector", then this transformation is applicable and these fields will be automatically transformed during migration.

Compatibility

The dense_vector to knn_vector transformation applies to:

  • Source clusters: Elasticsearch 7.x+
  • Target clusters: OpenSearch 1.x+
  • Automatic conversion: No configuration required

Automatic conversion logic

Migration Assistant performs the following transformations when converting dense_vector to knn_vector fields.

Field type transformation

  • Changes type: "dense_vector" to type: "knn_vector"
  • Maps dims parameter to dimension
  • Converts similarity metrics to OpenSearch space types
  • Configures the Hierarchical Navigable Small World (HNSW) algorithm with the Lucene engine

Similarity mapping

The transformation maps Elasticsearch similarity functions to OpenSearch space types:

  • cosine → cosinesimil
  • dot_product → innerproduct
  • l2 (default) → l2

Index settings

When dense_vector fields are converted, Migration Assistant automatically performs the following operations:

  • Enables the k-NN plugin by setting index.knn: true
  • Ensures proper index configuration for vector search

Migration output

During the migration process, you’ll see this transformation in the output:

Transformations:
   dense_vector to knn_vector:
      Convert field data type dense_vector to OpenSearch knn_vector

Transformation behavior

Source field type Target field type 
{
  "properties": {
    "embedding": {
      "type": "dense_vector",
      "dims": 128,
      "similarity": "cosine"
    }
  }
}
 
{
  "properties": {
    "embedding": {
      "type": "knn_vector",
      "dimension": 128,
      "method": {
        "name": "hnsw",
        "engine": "lucene",
        "space_type": "cosinesimil",
        "parameters": {
          "encoder": {
            "name": "sq"
          }
        }
      }
    }
  }
}

HNSW algorithm parameters

The transformation automatically configures the HNSW algorithm with the following options:

  • engine: lucene (OpenSearch default)
  • encoder: sq (scalar quantization for memory efficiency)
  • method: hnsw (approximate nearest neighbor search)

Index options mapping

Elasticsearch index_options are mapped to OpenSearch HNSW parameters:

  • m → m (maximum number of connections per node)
  • ef_construction → ef_construction (size of dynamic candidate list)

Index settings

When any dense_vector fields are converted, the following index setting is automatically added:

{
  "settings": {
    "index.knn": true
  }
}

Behavior differences

Migration Assistant automatically transforms all dense_vector fields during metadata migration. The k-NN plugin must be installed and enabled on the target OpenSearch cluster. Note: Most OpenSearch distributions include the k-NN plugin in which case no action is needed.

Query compatibility

After migration, vector search queries need to be updated:

  • Elasticsearch uses script_score queries with vector functions.
  • OpenSearch uses native knn query syntax.

Elasticsearch query example:

{
  "query": {
    "script_score": {
      "query": {"match_all": {}},
      "script": {
        "source": "cosineSimilarity(params.query_vector, 'embedding') + 1.0",
        "params": {"query_vector": [0.1, 0.2, 0.3]}
      }
    }
  }
}

OpenSearch query example:

{
  "query": {
    "knn": {
      "embedding": {
        "vector": [0.1, 0.2, 0.3],
        "k": 10
      }
    }
  }
}

Troubleshooting

If you encounter issues with dense_vector conversion:

  1. Verify the k-NN plugin – Ensure the k-NN plugin is installed and enabled on your target OpenSearch cluster: 
    GET /_cat/plugins
  2. Check migration logs – Review the detailed migration logs for any warnings or errors: 
    tail /shared-logs-output/migration-console-default/*/metadata/*.log
  3. Validate mappings – After migration, verify that the field types have been correctly converted: 
    GET /your-index/_mapping
  4. Test vector search – Verify that vector search functionality works with sample queries: 
    POST /your-index/_search
{
  "query": {
    "knn": {
      "embedding": {
        "vector": [0.1, 0.2, 0.3],
        "k": 5
      }
    }
  }
}
  5. Monitor performance – Vector search performance may differ between Elasticsearch and OpenSearch. Monitor query performance and adjust HNSW parameters if needed.
350 characters left

Have a question? .

Want to contribute? or .