Query metrics

Introduced 2.16

Key query metrics, such as aggregation types, query types, latency, and resource usage per query type, are captured along the search path by using the OpenTelemetry (OTel) instrumentation framework. The telemetry data can be consumed using OTel metrics exporters.

Configuring query metric generation

To configure query metric generation, use the following steps.

Step 1: Install the Query Insights plugin

For information about installing the Query Insights plugin, see Installing the Query Insights plugin.

Step 2: Install the OpenTelemetry plugin

For information about installing the OpenTelemetry plugin, see Distributed tracing.

Step 3: Enable query metrics

Enable query metrics by configuring the following opensearch.yml settings:

telemetry.feature.metrics.enabled: true
search.query.metrics.enabled: true

The following is a complete sample configuration that includes a telemetry configuration:

# Enable query metrics feature
search.query.metrics.enabled: true
telemetry.feature.metrics.enabled: true

# OTel-related configuration
opensearch.experimental.feature.telemetry.enabled: true
telemetry.tracer.sampler.probability: 1.0
telemetry.feature.tracer.enabled: true

Alternatively, you can configure query metric generation using the API:

PUT _cluster/settings
{
  "persistent" : {
    "search.query.metrics.enabled" : true
  }
}

Configure the export of metrics and traces using a gRPC exporter. For more information, see Exporters. You can skip this step if you use the default logging exporter:

telemetry.otel.tracer.span.exporter.class: io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter
telemetry.otel.metrics.exporter.class: io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter

Metrics

Query metrics provide the following measurements:

• The number of queries per query type (for example, the number of match or regex queries)
• The number of queries per aggregation type (for example, the number of terms aggregation queries)
• The number of queries per sort order (for example, the number of ascending and descending sort queries)
• Histograms of latency for each query type, aggregation type, and sort order
• Histograms of cpu for each query type, aggregation type, and sort order
• Histograms of memory for each query type, aggregation type, and sort order

Default logging exporter

By default, if no gRPC exporters are configured, then the metrics and traces are exported to log files. The data is saved in the opensearch/logs directory in the following files:

• opensearch_otel_metrics.log
• opensearch_otel_traces.log