Ingesting observability data
Before you can analyze observability data in OpenSearch Dashboards, your applications must send it to OpenSearch or to a backend that OpenSearch can query. Logs and traces are stored in OpenSearch indexes. Metrics are stored in Prometheus, which OpenSearch queries through a Prometheus data source, or stored in OpenSearch indexes.
The setups described in the documentation use the following components:
- Applications instrumented using OpenTelemetry SDKs produce traces, metrics, and logs.
- Log collectors, such as Fluent Bit, read log files and send log lines.
- The OpenTelemetry Collector receives data from applications and routes it by type.
- OpenSearch Data Prepper parses and enriches logs and traces and writes them to OpenSearch indexes.
- Prometheus stores metrics. OpenSearch queries Prometheus through a Prometheus data source, so the Prometheus server must be reachable from the OpenSearch nodes.
Ingestion paths
The following table lists the typical paths that observability data takes to OpenSearch or Prometheus. Find the type of data that you want to ingest and then follow the setup instructions for its path.
| Data | Path | Setup |
|---|---|---|
| Logs | Log collector or OpenTelemetry Collector → Data Prepper → OpenSearch index OR Log collector → OpenSearch index | Getting started with log analytics (Fluent Bit tutorial) Ingesting log data into OpenSearch Log analytics OTel logs source Fluent Bit OpenSearch output Logstash |
| Traces | Instrumented application → OpenTelemetry Collector → Data Prepper → OpenSearch span and service map indexes | Getting started with trace analytics (Jaeger HotROD tutorial) Trace analytics with Data Prepper OTel trace source |
| Metrics | Instrumented application → OpenTelemetry Collector → Prometheus OR Instrumented application → OpenTelemetry Collector → Data Prepper → OpenSearch index | Connecting Prometheus to OpenSearch Metric analytics OTel metrics source |
| Logs, traces, and metrics for APM | Instrumented application → OpenTelemetry Collector → Data Prepper → OpenSearch indexes (logs and traces) and Prometheus (metrics) | Configuring telemetry ingestion |
| Agent traces | Application instrumented using the OpenSearch generative AI observability SDK → OpenTelemetry Collector → Data Prepper → OpenSearch index | Instrumenting your application |
Trace analytics and APM require the data that their Data Prepper pipelines create, so follow the setup instructions for the app that you plan to use.
Related documentation
- To install preconfigured dashboards and visualizations for common data sources, see Integrations.
- For the index naming conventions and field mappings that some classic observability apps require, see Simple Schema for Observability.