Link Search Menu Expand Document Documentation Menu
Documentation

Connectors for third-party ML platforms

Introduced 2.9

Connectors facilitate access to models hosted on third-party machine learning (ML) platforms.

OpenSearch provides connectors for several platforms, for example:

  • Amazon SageMaker allows you to host and manage the lifecycle of text embedding models, powering semantic search queries in OpenSearch. When connected, Amazon SageMaker hosts your models and OpenSearch is used to query inferences. This benefits Amazon SageMaker users who value its functionality, such as model monitoring, serverless hosting, and workflow automation for continuous training and deployment.
  • OpenAI ChatGPT enables you to invoke an OpenAI chat model from inside an OpenSearch cluster.
  • Cohere allows you to use data from OpenSearch to power the Cohere large language models.
  • Amazon Bedrock supports models like Bedrock Titan Embeddings, which can drive semantic search and retrieval-augmented generation in OpenSearch.
  • Google Cloud Vertex AI enables you to invoke Vertex AI models, such as Gemini and text embedding models, from inside an OpenSearch cluster.

Connector blueprints

Creating a connector requires two pieces of information: the request and response format for your model, and the authentication method for your platform:

  • The request and response format is specified in a connector blueprint, which defines the set of fields (the request body) to provide when creating a connector for a specific platform and model. To find a pre-built blueprint for your platform and model, see OpenSearch-provided connector blueprints. For descriptions of all connector fields or to create a blueprint for a platform or model that OpenSearch does not provide, see Connector blueprints.

  • The authentication method is specified in the connector’s protocol field and is determined by the platform. For more information, see Connector authentication.

Creating a connector

You can provision connectors in two ways:

  1. Create a standalone connector: A standalone connector can be reused by multiple model registrations in OpenSearch that share the same external endpoint and configuration. Standalone connectors require access to both the connector and the model in OpenSearch as well as the third-party platform. Standalone connectors are saved in a connector index.

  2. Create a connector for a specific externally hosted model: Alternatively, you can create a connector that can only be used with the model for which it was created. To access such a connector, you only need access to the model itself because the connection is established inside the model. These connectors are saved in the model index.

If you need to connect to a different external model (for example, switching from gpt-4o-mini to gpt-4o), we recommend creating a separate standalone connector. Alternatively, advanced users can override connector parameters at predict time if the connector blueprint uses placeholders. For more information, see Connector blueprints.

If using Python, you can create connectors using the opensearch-py-ml client CLI. The CLI automates many configuration steps, making setup faster and reducing the chance of errors. For more information about using the CLI, see the CLI documentation.

Creating a standalone connector

To create a standalone connector, send a request to the connectors/_create endpoint and provide all of the parameters described in Connector blueprints:

POST /_plugins/_ml/connectors/_create
{
    "name": "OpenAI Chat Connector",
    "description": "The connector to public OpenAI model service for gpt-4o-mini",
    "version": 1,
    "protocol": "http",
    "parameters": {
        "endpoint": "api.openai.com",
        "model": "gpt-4o-mini"
    },
    "credential": {
        "openAI_key": "<openai_key>"
    },
    "actions": [
        {
            "action_type": "predict",
            "method": "POST",
            "url": "https://${parameters.endpoint}/v1/chat/completions",
            "headers": {
                "Authorization": "Bearer ${credential.openAI_key}"
            },
            "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }"
        }
    ]
}

Creating a connector for a specific model

To create a connector for a specific model, provide all of the parameters described in Connector blueprints within the connector object of a request to the models/_register endpoint:

POST /_plugins/_ml/models/_register
{
    "name": "openAI-gpt-4o-mini model with a connector",
    "function_name": "remote",
    "model_group_id": "lEFGL4kB4ubqQRzegPo2",
    "description": "test model",
    "connector": {
        "name": "OpenAI Connector",
        "description": "The connector to public OpenAI model service for gpt-4o-mini",
        "version": 1,
        "protocol": "http",
        "parameters": {
            "endpoint": "api.openai.com",
            "max_tokens": 7,
            "temperature": 0,
            "model": "gpt-4o-mini"
        },
        "credential": {
            "openAI_key": "<openai_key>"
        },
        "actions": [
            {
                "action_type": "predict",
                "method": "POST",
                "url": "https://${parameters.endpoint}/v1/chat/completions",
                "headers": {
                    "Authorization": "Bearer ${credential.openAI_key}"
                },
                "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": [{\"role\": \"user\", \"content\": \"${parameters.prompt}\"}], \"max_tokens\": ${parameters.max_tokens}, \"temperature\": ${parameters.temperature} }"
            }
        ]
    }
}

Updating connector credentials

In some cases, you may need to update the credentials used to connect to an externally hosted model, such as an expiring API key. To do this without undeploying the model, provide the new credentials in an update request.

Connector for a specific model

To update credentials for a connector linked to a specific model, provide the new credentials in the following request:

PUT /_plugins/_ml/models/{model_id}
{
  "connectors": {
    "credential": {
      "openAI_key": "<new_openai_key>"
    }
  }
}

Standalone connector

To update credentials for a standalone connector, provide the new credentials in the following request:

PUT /_plugins/_ml/connectors/{connector_id}
{
  "credential": {
    "openAI_key": "<new_openai_key>"
  }
}

Next steps

350 characters left

Have a question? .

Want to contribute? or .