Metrics
This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated GitHub issue.
The metrics framework records counters, histograms, and gauges for cluster operations and exports them using the telemetry-otel plugin. Histograms support percentiles, which the Stats APIs do not provide, so you can identify outliers in latency and resource usage. Plugins and extensions can add their own metrics using the metrics framework APIs.
Enabling metrics
To enable metrics, first enable the telemetry feature flag and install the telemetry-otel plugin. For more information, see Enabling telemetry.
Then add the following setting to opensearch.yml on each node and restart the node:
telemetry.feature.metrics.enabled: true
Configuring metrics export
The telemetry-otel plugin aggregates metrics locally and exports them at a configured interval. To configure the export, specify the following static settings in opensearch.yml.
| Setting | Description |
|---|---|
telemetry.otel.metrics.publish.interval | The interval at which metrics are exported. Default is 60s. |
telemetry.otel.metrics.exporter.class | The exporter that persists the metrics. Valid values are io.opentelemetry.exporter.logging.LoggingMetricExporter, which writes metrics to the <cluster-name>_otel_metrics.log file in the logs directory, and io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter, which sends metrics to an OpenTelemetry Collector using gRPC. Default is io.opentelemetry.exporter.logging.LoggingMetricExporter. |
To send metrics to an OpenTelemetry Collector, install the collector on each node and add the following setting to opensearch.yml. By default, the exporter sends metrics to the http://localhost:4317/ endpoint:
telemetry.otel.metrics.exporter.class: io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter
Supported metric types
The metrics framework supports the following metric types.
| Type | Description |
|---|---|
| Counter | Records a value that only increases, such as the number of errors, the number of bytes processed, or the total number of requests. |
| UpDown counter | Records a value that can increase or decrease, such as the number of open connections or active requests. |
| Histogram | Records the distribution of values, such as request latencies, so that you can calculate percentiles and find outliers. |
| Asynchronous gauge | Records the current value each time the metric is read, such as CPU or memory utilization. |
Monitoring machine learning workflows
Introduced 3.1
ML Commons publishes metrics about machine learning (ML) operations to the metrics framework. Every 5 minutes, a periodic job also collects and exports the state of the models and agents in the cluster.
The periodic job captures the following attributes.
| Resource | Captured attributes |
|---|---|
| Model | Deployment type (remote, pretrained, or custom), service provider, algorithm, model name, and model type |
| Agent | Large language model (LLM) interface, model deployment type, service provider, model type, memory type, and model identifier |
The following is an example of captured model metrics:
{is_hidden=false, service_provider=openai, model=gpt-4o-mini, type=llm, deployment=remote, algorithm=REMOTE}
The following is an example of captured agent metrics:
{_llm_interface=bedrock/converse/claude, model_deployment=remote, is_hidden=false, model_service_provider=bedrock, model_type=llm, memory_type=conversation_index, model=us.anthropic.claude-sonnet-4-6, type=CONVERSATIONAL}
To enable ML metrics, first enable metrics. Then add the following settings to opensearch.yml and restart the node:
plugins.ml_commons.metrics_collection_enabled: true
plugins.ml_commons.metrics_static_collection_enabled: true
The plugins.ml_commons.metrics_collection_enabled setting enables ML operation metrics and is static. The plugins.ml_commons.metrics_static_collection_enabled setting enables the periodic job and is dynamic, so you can also update it using the Cluster Settings API.