Link Search Menu Expand Document Documentation Menu

Kinesis source for pull-based ingestion

Introduced 3.0

This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, join the discussion on the OpenSearch forum.

The kinesis source lets an index consume records from an Amazon Kinesis data stream. Each OpenSearch shard reads from one Kinesis shard. For more information, see Stream partitioning.

Prerequisites

Before you create an index that uses the Kinesis source, install the ingestion-kinesis plugin on every node using bin/opensearch-plugin install ingestion-kinesis.

Creating an index

The following request creates an index that pulls data from a Kinesis stream in the segment replication mode. For other available modes, see Ingestion modes:

PUT /my-index
{
  "settings": {
    "ingestion_source": {
      "type": "kinesis",
      "pointer.init.reset": "earliest",
      "param": {
        "stream": "test",
        "region": "us-east-1",
        "access_key": "<access_key>",
        "secret_key": "<secret_key>"
      }
    },
    "index.number_of_shards": 1,
    "index.number_of_replicas": 1,
    "index": {
      "replication.type": "SEGMENT"
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "age": {
        "type": "integer"
      }
    }
  }
}

For all ingestion_source settings, see Ingestion source settings.

Source parameters

The following table lists the source parameters. Specify them in the param object of ingestion_source.

Parameter Required Description
stream Yes The Kinesis stream name.
region Yes The AWS Region.
access_key Yes The AWS access key.
secret_key Yes The AWS secret key.
endpoint_override No A custom endpoint for the Kinesis service.

When you reset a Kinesis consumer by offset, provide a Kinesis sequence number as the offset. For more information, see Resume Ingestion API.

350 characters left

Have a question? .

Want to contribute? or .