Link Search Menu Expand Document Documentation Menu
Documentation

Pipeline latency tuning guide

This section provides the most effective configurations for reducing end-to-end latency and contains ready-to-use examples.

There are two latency types:

  • Ingest latency: The amount of time from the moment the source receives data until the sink sends it to OpenSearch or another destination.
  • Searchable latency: The amount of time until data becomes visible in OpenSearch search results. This is capped by the index’s refresh_interval.

Low-latency configuration

The following table lists configurations that can be adjusted to improve latency.

Component Setting Why it matters Low‑latency starting point Trade‑offs
Pipeline loop workers in each pipeline Increasing parallelism reduces queueing in CPU- or I/O-bound pipelines. Typically set to the number of CPU cores; increase if sinks are I/O bound. Higher CPU usage and more concurrent requests sent to sinks.
Pipeline loop delay The pause between buffer reads. 0–10ms to pull data as soon as possible. Lower delays reduce latency but increase polling overhead, context switches, and CPU activations. Adjust to balance latency and CPU usage.
Bounded blocking buffer batch_size Determines how many records are processed per batch; smaller batches flush sooner. 64–256 Smaller batches reduce throughput and increase the number of requests.
Peer Forwarder batch_size, request_timeout Batch size and request timeouts impact hop-to-hop latency. Keep batch_size moderate, for example, 48–128. Too small a batch_size reduces throughput, and a request_timeout that is too short can cause retries or timeouts under load.
Peer Forwarder forwarder configuration Limits queueing before forwarding. Use low timeouts, for example, 50–200 ms. Shorter timeouts increase in-flight requests and connections, adding CPU, memory, TLS handshake, and context-switch overhead. Longer timeouts can cause queue buildup and higher tail latency.
Aggregate processors group_duration Determines how long events wait for the aggregation window to close. Prefer removing aggregation; if needed, keep the window short, for example, 5s. Smaller windows can break grouping semantics.
OpenSearch sink bulk_size (MiB) Determines the size of bulk requests; smaller bulks flush sooner. 1–5 MiB Very small bulk_size increases the number of bulk requests, adds HTTP/TLS overhead, causes more threadpool contention, produces smaller Lucene batches, and lowers throughput. Very large bulk_size increases the time needed to fill a batch and can cause memory spikes, larger retries, and higher p95/p99 latency.
OpenSearch sink index.refresh_interval Controls how frequently indexed data becomes searchable. 1s (default) Lower values increase segment churn and indexing overhead.

The delay processor adds latency by design. Avoid it in low‑latency pipelines.

Low‑latency configuration examples

The following are low-latency configuration examples.

Logs: Prioritize sub‑second ingest

You can use the following example when you need sub‑second end‑to‑end ingest for HTTP log intake. It minimizes batching and avoids heavy processors so events flush to OpenSearch quickly while keeping CPU costs predictable:

logs-low-latency:
  workers: 4
  delay: 0
  source:
    http:
      port: 2021
      path: /logs
      ssl: true
      sslKeyCertChainFile: certs/dp.crt
      sslKeyFile: certs/dp.key
  buffer:
    bounded_blocking:
      buffer_size: 4096
      batch_size: 128
  processor: []   # keep light, avoid heavy aggregation
  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        index_type: log-analytics
        bulk_size: 1         # MiB; smaller -> lower latency, lower throughput
        max_retries: 8

Traces with peer forwarding minimizing cross‑node wait

You can use the following example configuration to minimize cross‑node wait for distributed tracing. The data-prepper-config.yaml file enables low‑latency, mTLS‑secured peer forwarding with fast timeouts, and the pipelines.yml file splits ingestion from raw indexing so that the forwarding hop stays lightweight.

Create the following data-prepper-config.yaml file:


ssl: true
serverPort: 4900
keyStoreFilePath: certs/dp1-admin.p12   # or .jks
keyStorePassword: changeit
privateKeyPassword: changeit

# Or disable ssl on top-level admin/metrics server
#ssl: false
#serverPort: 4900

authentication:
  http_basic:
    username: myuser
    password: "mys3cr3t"
  # or disable http_basic authentication
  #unauthenticated:
  
peer_forwarder:

  ssl: true
  ssl_certificate_file: certs/dp1-peer.crt
  ssl_key_file: certs/dp1-peer.key
  authentication:
    mutual_tls: {}
  port: 4994 # Default
  # choose one discovery mode
  # Discovery mode: dns
  discovery_mode: dns
  domain_name: data-prepper.your-domain.local

  # discovery_mode: static
  #port: 4994
  #static_endpoints: ["dp1", "dp2"]

  # lower batching/wait
  batch_size: 96
  request_timeout: 1000   # ms

Create the following pipelines.yml file:

traces-low-latency:
  workers: 4
  delay: 0
  source:
    otel_trace_source:
      port: 21890
      ssl: true
      sslKeyCertChainFile: certs/dp.crt
      sslKeyFile: certs/dp.key
  buffer:
    bounded_blocking:
      buffer_size: 4096
      batch_size: 96
  processor:
    - trace_peer_forwarder: {}
  sink:
    - pipeline:
        name: raw-trace-pipeline   # feed the next pipeline

raw-trace-pipeline:
  source:
    pipeline:
      name: traces-low-latency     # consumes from above
  processor:
    - otel_traces:
  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        insecure: true
        username: admin
        password: admin_password
        index_type: trace-analytics-raw

350 characters left

Have a question? .

Want to contribute? or .