Link Search Menu Expand Document Documentation Menu
Documentation

Multimodal search using ML inference processors

This tutorial shows you how to search text and image data using the Amazon Bedrock Titan Multimodal Embeddings model (amazon.titan-embed-image-v1) and ML inference processors. Using the processors, you can search an index by text, by image, or by text and image together, using one search pipeline. For more information about multimodal search, see Multimodal search.

In this tutorial, you’ll use the following OpenSearch components:

For a related example in Python, see the e-commerce demo notebook, which uses ML inference processors to index product images and search them by text or image.

The steps in this tutorial use self-managed OpenSearch, with AWS access keys in the connector credentials. If using Amazon OpenSearch Service, see Using Amazon OpenSearch Service.

Replace the placeholders beginning with the prefix your_ with your own values.

Step 1: Create an embedding model

Follow these steps to create and test the embedding model.

Step 1.1: Create a connector

To create a connector for the model, send the following request. Because you’ll use ML inference processors in this tutorial, you don’t need to specify a pre- or post-processing function in the connector:

POST /_plugins/_ml/connectors/_create
{
  "name": "Amazon Bedrock connector: Titan Multimodal Embeddings",
  "description": "The connector to the Amazon Bedrock Titan Multimodal Embeddings model",
  "version": 1,
  "protocol": "aws_sigv4",
  "credential": {
    "access_key": "your_aws_access_key",
    "secret_key": "your_aws_secret_key",
    "session_token": "your_aws_session_token"
  },
  "parameters": {
    "region": "your_aws_region",
    "service_name": "bedrock",
    "model": "amazon.titan-embed-image-v1"
  },
  "actions": [
    {
      "action_type": "predict",
      "method": "POST",
      "headers": {
        "content-type": "application/json"
      },
      "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
      "request_body": "{\"inputText\": \"${parameters.inputText:-null}\", \"inputImage\": \"${parameters.inputImage:-null}\"}"
    }
  ]
}

The model accepts text in the inputText parameter and a Base64-encoded image in the inputImage parameter. The :-null suffix sets a parameter to null when the request doesn’t provide it, so you can send text, an image, or both.

The response contains a connector ID:

{
  "connector_id": "P8c_JZUB7judm8f4591w"
}

Note the connector ID; you’ll use it in the next step.

Step 1.2: Register and deploy the model

To register and deploy the model, send the following request:

POST /_plugins/_ml/models/_register?deploy=true
{
  "name": "Amazon Bedrock Titan Multimodal Embeddings",
  "function_name": "remote",
  "description": "Multimodal embedding model",
  "connector_id": "P8c_JZUB7judm8f4591w"
}

The response contains the model ID:

{
  "task_id": "nMdJJZUB7judm8f4HN29",
  "status": "CREATED",
  "model_id": "ncdJJZUB7judm8f4HN3P"
}

Note the model ID; you’ll use it in the following steps.

Step 1.3: Test the model

To test the model, send the following request. To generate an embedding for an image, provide a Base64-encoded image in the inputImage parameter. If you provide both inputText and inputImage, the model generates one embedding that represents both inputs:

POST /_plugins/_ml/models/ncdJJZUB7judm8f4HN3P/_predict
{
  "parameters": {
    "inputText": "Say this is a test"
  }
}

Because the connector has no post-processing function, the response contains the model output in the dataAsMap object. The embedding field contains the embedding generated by the model:

{
  "inference_results": [
    {
      "output": [
        {
          "name": "response",
          "dataAsMap": {
            "embedding": [
              0.022338867,
              ...
            ],
            "inputTextTokenCount": 7,
            "message": null,
            "successCount": null,
            "failureCount": null,
            "results": null
          }
        }
      ],
      "status_code": 200
    }
  ]
}

Step 2: Create an ingest pipeline

Create an ingest pipeline containing an ML inference processor. The processor’s input_map passes the document’s name field to the model’s inputText parameter and the image field to the inputImage parameter. The output_map stores the embedding field from the model output in a new document field named multimodal_embedding:

PUT /_ingest/pipeline/ml_inference_pipeline_multimodal
{
  "description": "Generate multimodal embeddings for ingested documents",
  "processors": [
    {
      "ml_inference": {
        "tag": "ml_inference",
        "model_id": "ncdJJZUB7judm8f4HN3P",
        "input_map": [
          {
            "inputText": "name",
            "inputImage": "image"
          }
        ],
        "output_map": [
          {
            "multimodal_embedding": "embedding"
          }
        ],
        "ignore_missing": true,
        "ignore_failure": true
      }
    }
  ]
}

Both input fields are in one input_map object, so the processor invokes the model once per document. Because ignore_missing is true, a document that contains only one of the two fields is still sent to the model, and a document that contains neither is indexed without an embedding.

Step 3: Create an index and ingest data

To create a vector index that uses the ingest pipeline by default, send the following request. The dimension of the multimodal_embedding field matches the model’s default embedding length of 1,024:

PUT /my-multimodal-index
{
  "settings": {
    "index.knn": true,
    "default_pipeline": "ml_inference_pipeline_multimodal"
  },
  "mappings": {
    "properties": {
      "multimodal_embedding": {
        "type": "knn_vector",
        "dimension": 1024
      }
    }
  }
}

Ingest three documents into the index. The first document contains only text, the second contains text and a Base64-encoded image, and the third contains neither a name nor an image field. In the following request, the Base64 string is truncated; replace it with a full Base64-encoded image:

POST _bulk
{ "index": { "_index": "my-multimodal-index", "_id": "1" } }
{ "name": "Central Park", "category": "Park" }
{ "index": { "_index": "my-multimodal-index", "_id": "2" } }
{ "name": "Times Square", "category": "Square", "image": "iVBORw0KGgoAAAANSUhEUgAAAFcAAAAdCAYAAADfC/BmAAAAAXNSR0IArs4c6QAAAGJlWElmTU0AKgAAAA..." }
{ "index": { "_index": "my-multimodal-index", "_id": "3" } }
{ "category": "Bridge" }

To verify that the pipeline generated an embedding from the text and image, retrieve the second document:

GET /my-multimodal-index/_doc/2

The document contains the generated multimodal_embedding field:

{
  "_index": "my-multimodal-index",
  "_id": "2",
  "_version": 1,
  "_seq_no": 1,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "multimodal_embedding": [
      0.026977539,
      ...
    ],
    "image": "iVBORw0KGgoAAAANSUhEUgAAAFcAAAAdCAYAAADfC/BmAAAAAXNSR0IArs4c6QAAAGJlWElmTU0AKgAAAA...",
    "name": "Times Square",
    "category": "Square"
  }
}

The first document contains an embedding generated from the name field only. The third document is indexed without a multimodal_embedding field because it contains no fields in the input_map.

Step 4: Create a search pipeline

Create a search pipeline containing an ML inference search request processor. The processor rewrites the search request into a knn query that uses the embedding generated from the query text, the query image, or both:

PUT /_search/pipeline/multimodal_semantic_search_pipeline
{
  "request_processors": [
    {
      "ml_inference": {
        "tag": "ml_inference",
        "description": "Generates an embedding from the query text or image and runs a k-NN query",
        "model_id": "ncdJJZUB7judm8f4HN3P",
        "query_template": "{\"query\": {\"knn\": {\"multimodal_embedding\": {\"vector\": ${multimodal_embedding}, \"k\": 3}}}}",
        "optional_input_map": [
          {
            "inputText": "ext.ml_inference.text",
            "inputImage": "ext.ml_inference.image"
          }
        ],
        "output_map": [
          {
            "multimodal_embedding": "embedding"
          }
        ],
        "model_config": {},
        "ignore_missing": false,
        "ignore_failure": false
      }
    }
  ]
}

The processor contains the following settings:

  • The optional_input_map maps the ext.ml_inference.text and ext.ml_inference.image fields in the search request to the model’s inputText and inputImage parameters. The processor passes only the fields that are present in the request to the model, so a search request can contain text, an image, or both. A search request must contain at least one of the fields: if it contains neither, the processor sends the model a request without input, and the model returns an error. If you use input_map, both fields must be present in every search request.
  • The output_map stores the embedding field from the model output in the multimodal_embedding variable.
  • The query_template defines the query that replaces the original query in the search request. The template inserts the multimodal_embedding variable as the query vector.

Step 5: Search the index

When you search using the pipeline, the processor replaces the query in the request with the knn query from the query_template. The following examples send a match_all query, which the processor replaces.

Search by text

To search the index by text, provide the query text in the ext.ml_inference.text field:

GET /my-multimodal-index/_search?search_pipeline=multimodal_semantic_search_pipeline
{
  "query": {
    "match_all": {}
  },
  "ext": {
    "ml_inference": {
      "text": "place where recreational activities are done, picnics happen there"
    }
  }
}

The response contains the two documents that have embeddings. The Central Park document is the closest match:

{
  "took": 222,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 0.5640698,
    "hits": [
      {
        "_index": "my-multimodal-index",
        "_id": "1",
        "_score": 0.5640698,
        "_source": {
          "multimodal_embedding": [
            0.01171875,
            ...
          ],
          "name": "Central Park",
          "category": "Park"
        }
      },
      {
        "_index": "my-multimodal-index",
        "_id": "2",
        "_score": 0.5551745,
        "_source": {
          "image": "iVBORw0KGgoAAAANSUhEUgAAAFcAAAAdCAYAAADfC/BmAAAAAXNSR0IArs4c6QAAAGJlWElmTU0AKgAAAA...",
          "multimodal_embedding": [
            0.026977539,
            ...
          ],
          "name": "Times Square",
          "category": "Square"
        }
      }
    ]
  }
}

To view the query that the processor generated, add the verbose_pipeline=true query parameter to the search request:

GET /my-multimodal-index/_search?search_pipeline=multimodal_semantic_search_pipeline&verbose_pipeline=true
{
  "query": {
    "match_all": {}
  },
  "ext": {
    "ml_inference": {
      "text": "place where recreational activities are done, picnics happen there"
    }
  }
}

In addition to the search results, the response contains a processor_results array. The input_data object contains the original search request, and the output_data object contains the rewritten knn query:

{
  ...
  "processor_results": [
    {
      "processor_name": "ml_inference",
      "tag": "ml_inference",
      "duration_millis": 442,
      "status": "success",
      "input_data": {
        "ext": {
          "ml_inference": {
            "text": "place where recreational activities are done, picnics happen there"
          }
        },
        "verbose_pipeline": true,
        "query": {
          "match_all": {
            "boost": 1.0
          }
        }
      },
      "output_data": {
        "query": {
          "knn": {
            "multimodal_embedding": {
              "vector": [
                0.035888672,
                0.014221191,
                ...
              ],
              "boost": 1.0,
              "k": 3
            }
          }
        }
      }
    }
  ]
}

For more information, see Debugging a search pipeline.

Search by image

To search the index by image, provide a Base64-encoded image in the ext.ml_inference.image field. In the following request, the Base64 string is truncated; replace it with a full Base64-encoded image:

GET /my-multimodal-index/_search?search_pipeline=multimodal_semantic_search_pipeline
{
  "query": {
    "match_all": {}
  },
  "ext": {
    "ml_inference": {
      "image": "iVBORw0KGgoAAAANSUhEUgAAAFcAAAAdCAYAAADfC/BmAAAAAXNSR0IArs4c6QAAAGJlWElmTU0AKgAAAA..."
    }
  }
}

The Times Square document, which contains the same image, is the closest match:

{
  "took": 248,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 0.7687805,
    "hits": [
      {
        "_index": "my-multimodal-index",
        "_id": "2",
        "_score": 0.7687805,
        "_source": {
          "image": "iVBORw0KGgoAAAANSUhEUgAAAFcAAAAdCAYAAADfC/BmAAAAAXNSR0IArs4c6QAAAGJlWElmTU0AKgAAAA...",
          "multimodal_embedding": [
            0.026977539,
            ...
          ],
          "name": "Times Square",
          "category": "Square"
        }
      },
      {
        "_index": "my-multimodal-index",
        "_id": "1",
        "_score": 0.4280292,
        "_source": {
          "multimodal_embedding": [
            0.01171875,
            ...
          ],
          "name": "Central Park",
          "category": "Park"
        }
      }
    ]
  }
}

Search by text and image

To search the index by text and image together, provide both the text and image fields:

GET /my-multimodal-index/_search?search_pipeline=multimodal_semantic_search_pipeline
{
  "query": {
    "match_all": {}
  },
  "ext": {
    "ml_inference": {
      "text": "any place in New York",
      "image": "iVBORw0KGgoAAAANSUhEUgAAAFcAAAAdCAYAAADfC/BmAAAAAXNSR0IArs4c6QAAAGJlWElmTU0AKgAAAA..."
    }
  }
}

The model generates one embedding from the text and image, and the response contains the documents ordered by their similarity to that embedding:

{
  "took": 237,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 0.89464855,
    "hits": [
      {
        "_index": "my-multimodal-index",
        "_id": "2",
        "_score": 0.89464855,
        "_source": {
          "image": "iVBORw0KGgoAAAANSUhEUgAAAFcAAAAdCAYAAADfC/BmAAAAAXNSR0IArs4c6QAAAGJlWElmTU0AKgAAAA...",
          "multimodal_embedding": [
            0.026977539,
            ...
          ],
          "name": "Times Square",
          "category": "Square"
        }
      },
      {
        "_index": "my-multimodal-index",
        "_id": "1",
        "_score": 0.6142057,
        "_source": {
          "multimodal_embedding": [
            0.01171875,
            ...
          ],
          "name": "Central Park",
          "category": "Park"
        }
      }
    ]
  }
}

Using Amazon OpenSearch Service

In Amazon OpenSearch Service, the connector uses an AWS Identity and Access Management (IAM) role to invoke the model. Steps 2–5 of this tutorial are the same. To create the connector, follow Steps 1–3 of Semantic search using Amazon Bedrock Titan, making the following changes:

  • In the permissions of the role that invokes the model, set the resource to arn:aws:bedrock:*::foundation-model/amazon.titan-embed-image-v1.
  • In the Python code that creates the connector, use the following connector payload, which replaces the access keys with the IAM role ARN:
payload = {
  "name": "Amazon Bedrock connector: Titan Multimodal Embeddings",
  "description": "The connector to the Amazon Bedrock Titan Multimodal Embeddings model",
  "version": 1,
  "protocol": "aws_sigv4",
  "credential": {
    "roleArn": "your_iam_role_arn_created_in_step1"
  },
  "parameters": {
    "region": "your_aws_region",
    "service_name": "bedrock",
    "model": "amazon.titan-embed-image-v1"
  },
  "actions": [
    {
      "action_type": "predict",
      "method": "POST",
      "headers": {
        "content-type": "application/json"
      },
      "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
      "request_body": "{\"inputText\": \"${parameters.inputText:-null}\", \"inputImage\": \"${parameters.inputImage:-null}\"}"
    }
  ]
}

Then register and deploy the model as described in Step 1.2.

350 characters left

Have a question? .

Want to contribute? or .