Kafka source for pull-based ingestion
Introduced 3.0
The kafka source lets an index consume messages from an Apache Kafka topic. Each OpenSearch shard reads from one partition of the topic. For more information, see Stream partitioning.
Prerequisites
Before you create an index that uses the Kafka source, install the ingestion-kafka plugin on every node using bin/opensearch-plugin install ingestion-kafka.
Creating an index
The following request creates an index that pulls data from a Kafka topic in the segment replication mode. For other available modes, see Ingestion modes:
PUT /my-index
{
"settings": {
"ingestion_source": {
"type": "kafka",
"pointer.init.reset": "earliest",
"param": {
"topic": "test",
"bootstrap_servers": "localhost:49353"
}
},
"index.number_of_shards": 1,
"index.number_of_replicas": 1,
"index": {
"replication.type": "SEGMENT"
}
},
"mappings": {
"properties": {
"name": {
"type": "text"
},
"age": {
"type": "integer"
}
}
}
}
For all ingestion_source settings, see Ingestion source settings.
Source parameters
The following table lists the source parameters. Specify them in the param object of ingestion_source.
| Parameter | Required | Description |
|---|---|---|
topic | Yes | The Kafka topic to consume from. |
bootstrap_servers | Yes | The Kafka server addresses. |
Optionally, you can provide additional standard Kafka consumer parameters (such as fetch.min.bytes). These parameters are passed directly to the Kafka consumer.