Link Search Menu Expand Document Documentation Menu
Documentation

Generating embeddings from arrays of objects

This tutorial shows you how to generate embeddings for arrays of objects. For more information, see Generating embeddings automatically.

Replace the placeholders beginning with the prefix your_ with your own values.

Step 1: Register an embedding model

In this tutorial, you will use the Amazon Titan Text Embeddings models hosted on Amazon Bedrock.

First, follow the Amazon Bedrock Titan blueprint example to register and deploy the model.

Test the model, providing the model ID:

POST /_plugins/_ml/models/your_embedding_model_id/_predict
{
    "parameters": {
        "inputText": "hello world"
    }
}

The response contains inference results:

{
  "inference_results": [
    {
      "output": [
        {
          "name": "sentence_embedding",
          "data_type": "FLOAT32",
          "shape": [ 1536 ],
          "data": [0.7265625, -0.0703125, 0.34765625, ...]
        }
      ],
      "status_code": 200
    }
  ]
}

Step 2: Create an ingest pipeline

Follow the next set of steps to create an ingest pipeline for generating embeddings.

Step 2.1: Create a vector index

First, create a vector index:

PUT my_books
{
  "settings" : {
      "index.knn" : "true",
      "default_pipeline": "bedrock_embedding_pipeline"
  },
  "mappings": {
    "properties": {
      "books": {
        "type": "nested",
        "properties": {
          "title_embedding": {
            "type": "knn_vector",
            "dimension": 1536
          },
          "title": {
            "type": "text"
          },
          "description": {
            "type": "text"
          }
        }
      }
    }
  }
}

Step 2.2: Create an ingest pipeline

Then create an ingest pipeline containing a text_embedding processor. The books.title key in the field_map refers to the title field of each object in the books array, so the processor generates an embedding for each object that contains a title and stores it in that object’s title_embedding field.

To create the pipeline, send the following request:

PUT _ingest/pipeline/bedrock_embedding_pipeline
{
  "processors": [
    {
      "text_embedding": {
        "model_id": "your_embedding_model_id",
        "field_map": {
          "books.title": "title_embedding"
        }
      }
    }
  ]
}

Step 2.3: Simulate the pipeline

First, you’ll test the pipeline on an array that contains two book objects, both with a title field:

POST _ingest/pipeline/bedrock_embedding_pipeline/_simulate
{
  "docs": [
    {
      "_index": "my_books",
      "_id": "1",
      "_source": {
        "books": [
          {
            "title": "first book",
            "description": "This is first book"
          },
          {
            "title": "second book",
            "description": "This is second book"
          }
        ]
      }
    }
  ]
}

The response contains generated embeddings for both objects in their title_embedding fields:

{
  "docs": [
    {
      "doc": {
        "_index": "my_books",
        "_id": "1",
        "_source": {
          "books": [
            {
              "title": "first book",
              "title_embedding": [-1.1015625, 0.65234375, 0.7578125, ...],
              "description": "This is first book"
            },
            {
              "title": "second book",
              "title_embedding": [-0.65234375, 0.21679688, 0.7265625, ...],
              "description": "This is second book"
            }
          ]
        },
        "_ingest": {
          "timestamp": "2024-05-28T16:16:50.538929413Z"
        }
      }
    }
  ]
}

Next, you’ll test the pipeline on an array that contains two book objects, one with a title field and one without:

POST _ingest/pipeline/bedrock_embedding_pipeline/_simulate
{
  "docs": [
    {
      "_index": "my_books",
      "_id": "1",
      "_source": {
        "books": [
          {
            "title": "first book",
            "description": "This is first book"
          },
          {
            "description": "This is second book"
          }
        ]
      }
    }
  ]
}

The response contains generated embeddings for the object that contains the title field:

{
  "docs": [
    {
      "doc": {
        "_index": "my_books",
        "_id": "1",
        "_source": {
          "books": [
            {
              "title": "first book",
              "title_embedding": [-1.1015625, 0.65234375, 0.7578125, ...],
              "description": "This is first book"
            },
            {
              "description": "This is second book"
            }
          ]
        },
        "_ingest": {
          "timestamp": "2024-05-28T16:19:03.942644042Z"
        }
      }
    }
  ]
}

Step 2.4: Test data ingestion

Ingest one document:

PUT my_books/_doc/1
{
  "books": [
    {
      "title": "first book",
      "description": "This is first book"
    },
    {
      "title": "second book",
      "description": "This is second book"
    }
  ]
}

Get the document:

GET my_books/_doc/1

The response contains the generated embeddings:

{
  "_index": "my_books",
  "_id": "1",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "books": [
      {
        "description": "This is first book",
        "title": "first book",
        "title_embedding": [-1.1015625, 0.65234375, 0.7578125, ...]
      },
      {
        "description": "This is second book",
        "title": "second book",
        "title_embedding": [-0.65234375, 0.21679688, 0.7265625, ...]
      }
    ]
  }
}

You can also ingest several documents in bulk and test the generated embeddings by calling the Get Document API:

POST _bulk
{ "index" : { "_index" : "my_books" } }
{ "books" : [{"title": "first book", "description": "This is first book"}, {"title": "second book", "description": "This is second book"}] }
{ "index" : { "_index" : "my_books" } }
{ "books" : [{"title": "third book", "description": "This is third book"}, {"description": "This is fourth book"}] }

350 characters left

Have a question? .

Want to contribute? or .