Kinesis source for pull-based ingestion
Introduced 3.0
This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, join the discussion on the OpenSearch forum.
The kinesis source lets an index consume records from an Amazon Kinesis data stream. Each OpenSearch shard reads from one Kinesis shard. For more information, see Stream partitioning.
Prerequisites
Before you create an index that uses the Kinesis source, install the ingestion-kinesis plugin on every node using bin/opensearch-plugin install ingestion-kinesis.
Creating an index
The following request creates an index that pulls data from a Kinesis stream in the segment replication mode. For other available modes, see Ingestion modes:
PUT /my-index
{
"settings": {
"ingestion_source": {
"type": "kinesis",
"pointer.init.reset": "earliest",
"param": {
"stream": "test",
"region": "us-east-1",
"access_key": "<access_key>",
"secret_key": "<secret_key>"
}
},
"index.number_of_shards": 1,
"index.number_of_replicas": 1,
"index": {
"replication.type": "SEGMENT"
}
},
"mappings": {
"properties": {
"name": {
"type": "text"
},
"age": {
"type": "integer"
}
}
}
}
For all ingestion_source settings, see Ingestion source settings.
Source parameters
The following table lists the source parameters. Specify them in the param object of ingestion_source.
| Parameter | Required | Description |
|---|---|---|
stream | Yes | The Kinesis stream name. |
region | Yes | The AWS Region. |
access_key | Yes | The AWS access key. |
secret_key | Yes | The AWS secret key. |
endpoint_override | No | A custom endpoint for the Kinesis service. |
When you reset a Kinesis consumer by offset, provide a Kinesis sequence number as the offset. For more information, see Resume Ingestion API.