OTLP source

The otlp source is a unified OpenTelemetry source that follows the OpenTelemetry Protocol (OTLP) specification and can receive logs, metrics, and traces through a single endpoint. This source consolidates the functionality of the individual otel_logs_source, otel_metrics_source, and otel_trace_source sources, providing a streamlined approach to ingesting all OpenTelemetry telemetry signals.

The otlp source supports both the OTLP/gRPC and OTLP/HTTP protocols. For OTLP/HTTP, only Protobuf encoding is supported. This makes it compatible with a wide range of OpenTelemetry collectors and instrumentation libraries.

Configuration

You can configure the otlp source with the following options.

If an individual output format (for example, logs_output_format) is set, it takes precedence over the generic output_format for that signal type. If neither is set, the default is otel.

SSL/TLS configuration

You can configure SSL/TLS in the otlp source with the following options.

Authentication configuration

By default, the otlp source runs without authentication. You can configure authentication using the following options.

To explicitly disable authentication, specify the following settings:

source:
  otlp:
    authentication:
      unauthenticated:

To enable HTTP Basic authentication, specify the following settings:

source:
  otlp:
    authentication:
      http_basic:
        username: my-user
        password: my_s3cr3t

This plugin uses pluggable authentication for gRPC servers. To provide custom authentication, create a plugin that implements GrpcAuthenticationProvider.

Retry information

You can set retry behavior using the retry_info setting, specifying how long to wait for the next request when backpressure occurs. The retry mechanism applies exponential backoff with a configurable maximum delay:

source:
  otlp:
    retry_info:
      min_delay: 100ms  # defaults to 100ms
      max_delay: 2s     # defaults to 2s

Usage

The following examples demonstrate how to configure and use the otlp source in various scenarios.

Basic configuration

To get started with the otlp source, create a pipeline.yaml file with the following minimal configuration:

pipeline:
  source:
    otlp:
      ssl: false
  sink:
    - stdout:

Routing telemetry signals

One of the key features of the otlp source is its ability to route different telemetry signals (logs, metrics, and traces) to different processors or sinks based on your specific needs. Routing is determined by metadata using the getEventType() function:

version: "2"
otel-telemetry:
  source:
    otlp:
      ssl: false
  route:
    - traces: 'getEventType() == "TRACE"'
    - logs: 'getEventType() == "LOG"'
    - metrics: 'getEventType() == "METRIC"'
  sink:
    - opensearch:
        routes:
          - logs
        hosts: [ "https://opensearch:9200" ]
        index: logs-%{yyyy.MM.dd}
        username: admin
        password: yourStrongPassword123!
        insecure: true
    - pipeline:
        name: traces-raw
        routes:
          - traces
    - pipeline:
        name: otel-metrics
        routes:
          - metrics

traces-raw:
  source:
    pipeline:
      name: otel-telemetry
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        index_type: trace-analytics-raw
        username: admin
        password: yourStrongPassword123!
        insecure: true

otel-metrics:
  source:
    pipeline:
      name: otel-telemetry
  processor:
    - otel_metrics:
        calculate_histogram_buckets: true
        calculate_exponential_histogram_buckets: true
        exponential_histogram_max_allowed_scale: 10
        flatten_attributes: false
  sink:
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        index: metrics-otel-%{yyyy.MM.dd}
        username: admin
        password: yourStrongPassword123!
        insecure: true

Using OpenSearch output format

To generate data in the OpenSearch format for all telemetry signals, specify the following settings:

source:
  otlp:
    output_format: opensearch

To use different output formats for different signal types, specify the following settings:

source:
  otlp:
    logs_output_format: opensearch
    metrics_output_format: otel
    traces_output_format: opensearch

Configuring with SSL/TLS

To enable SSL/TLS with local certificates, specify the following settings:

source:
  otlp:
    ssl: true
    ssl_certificate_file: "/path/to/certificate.crt"
    ssl_key_file: "/path/to/private-key.key"

To use the ACM, specify the following settings:

source:
  otlp:
    ssl: true
    use_acm_cert_for_ssl: true
    acm_certificate_arn: "arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012"
    aws_region: "us-east-1"

Metrics

The otlp source includes the following metrics for monitoring its performance and health.

Counters

The following counters track request activity and errors in the otlp source.

Timers

The following timers track request activity and errors in the otlp source.

Distribution summaries

The following distribution summaries track request activity and errors in the otlp source.

Migrating from individual OpenTelemetry sources

If you’re using separate otel_logs_source, otel_metrics_source, or otel_trace_source sources, you can migrate to the unified otlp source by following these steps:

1. Replace all three sources with a single otlp source.
2. Use routing configuration to direct different signal types to their appropriate pipelines.
3. Change the port numbers if needed (the otlp source uses port 21893 by default).

Migration example

The following example demonstrates how to consolidate separate OpenTelemetry log, metric, and trace sources into a single otlp source.

Consider a setup where logs, metrics, and traces are configured separately:

logs-pipeline:
  source:
    otel_logs_source:
      port: 21892
  sink:
    - opensearch:
        index: logs

metrics-pipeline:
  source:
    otel_metrics_source:
      port: 21891
  sink:
    - opensearch:
        index: metrics

traces-pipeline:
  source:
    otel_trace_source:
      port: 21890
  sink:
    - opensearch:
        index: traces

You can consolidate logs, metrics, and traces into a single otlp source as follows:

otlp-pipeline:
  source:
    otlp:
      port: 21893
  route:
    - logs: 'getEventType() == "LOG"'
    - metrics: 'getEventType() == "METRIC"'
    - traces: 'getEventType() == "TRACE"'
  sink:
    - opensearch:
        routes: 
          - logs
        index: logs
    - opensearch:
        routes:
          - metrics
        index: metrics
    - opensearch:
        routes:
          - traces
        index: traces

Related pages

• OTel logs source
• OTel metrics source
• OTel trace source
• getEventType()