Getting started with OpenSearch Data Prepper
OpenSearch Data Prepper is a server-side data collector that receives data from your applications, transforms it, and sends it to OpenSearch. Data Prepper runs separately from OpenSearch and is not included in the OpenSearch installation packages.
Pipelines and events
Data Prepper runs pipelines. Each pipeline reads data from a source, transforms or drops events using one or more processors, and writes the result to one or more sinks. Between the source and the processors, events wait in a buffer. For example, a pipeline can receive log lines over HTTP, parse each line into fields, and write the result to an OpenSearch index. A single Data Prepper instance can run multiple pipelines.
An event is a single data item, such as a log line, that moves through a pipeline. Each event consists of key-value pairs called fields, similar to a JSON object. When a pipeline writes to OpenSearch, each event becomes a document. Other sinks, such as Amazon S3, serialize events into files. Processors read and modify the fields of each event before the event reaches a sink.
Configuration files
Data Prepper uses the following configuration files.
| File | Required | Description |
|---|---|---|
| Pipeline configuration | Yes | One or more YAML files that define the pipelines to run, including their sources, processors, and sinks. Data Prepper does not provide a default pipeline and does not start if no pipeline configuration file exists. For more information, see Configuring pipelines. |
data-prepper-config.yaml | No | Settings for the Data Prepper server, such as the port, TLS, and authentication for the core APIs. Data Prepper includes a default server configuration file. |
log4j2-rolling.properties | No | The Log4j 2 logging configuration. Data Prepper includes a default logging configuration file. |
Because a pipeline is required, you define at least one pipeline before you start Data Prepper.
Next steps
To get started with Data Prepper, follow these steps in order:
Install Data Prepper using the Docker image or a Linux archive and learn how it’s configured.
Learn about the pipeline structure and the server and logging configuration files.
Start Data Prepper using a minimal pipeline, send a test event, and then stop Data Prepper.
Build a pipeline that parses raw Apache access logs, writes them to an OpenSearch index, and lets you search the parsed fields.