OpenSearch Data Prepper
OpenSearch Data Prepper is a server-side data collector capable of filtering, enriching, transforming, normalizing, and aggregating data for downstream analysis and visualization. Data Prepper is the preferred data ingestion tool for OpenSearch. It is recommended for most data ingestion use cases in OpenSearch and for processing large, complex datasets.
With Data Prepper you can build custom pipelines to improve the operational view of applications. Two common use cases for Data Prepper are trace analytics and log analytics. Trace analytics can help you visualize event flows and identify performance problems. Log analytics equips you with tools to enhance your search capabilities, conduct comprehensive analysis, and gain insights into your applications’ performance and behavior.
Key concepts
Data Prepper ingests data through pipelines, which process data as events. An event is a single data item, such as a log line or a trace span. When a pipeline writes to OpenSearch, each event becomes a document. Each pipeline reads events from a source, transforms or drops them using processors, and writes the result to one or more sinks. For more information, see Getting started with OpenSearch Data Prepper.