Configuring OpenSearch Data Prepper

You can customize your OpenSearch Data Prepper configuration by editing the data-prepper-config.yaml file in your Data Prepper installation. The following configuration options are independent from pipeline configuration options.

Data Prepper configuration

Use the following options to customize your Data Prepper configuration.

Peer Forwarder options

The following section details various configuration options for peer forwarding.

General options for peer forwarding

The following table lists the general configuration for peer forwarding.

TLS/SSL options for Peer Forwarder

Use the following options to enable and configure TLS/SSL for Peer Forwarder, including certificate sources (files, S3, or ACM) and verification behavior.

Authentication options for Peer Forwarder

Use the following option to specify how Peer Forwarder authenticates requests between peers.

Circuit breakers

Data Prepper includes a circuit breaker to help prevent Java heap memory exhaustion. This is particularly useful for pipelines that use stateful processors, which retain data in memory beyond the buffers.

When a circuit breaker is triggered, Data Prepper rejects incoming data routed into buffers until the breaker resets.

The following table lists the available circuit breaker configuration blocks.

Heap circuit breaker

The heap circuit breaker configures Data Prepper to reject incoming data when the JVM heap reaches a specified usage threshold. The heap parameter supports the following values.

Extension plugins

Data Prepper supports user‑configurable extension plugins. Extension plugins provide a reusable configuration shared across pipeline plugins (sources, buffers, processors, or sinks).

AWS extension plugin

The aws extension provides the secrets configuration, which integrates with AWS Secrets Manager to securely manage sensitive configuration values in your pipelines. The following example shows a basic configuration:

extensions:
  aws:
    secrets:
      <YOUR_SECRET_CONFIG_ID_1>:
        secret_id: <YOUR_SECRET_ID_1>
        region: <YOUR_REGION_1>
        sts_role_arn: <YOUR_STS_ROLE_ARN_1>
        refresh_interval: <YOUR_REFRESH_INTERVAL>
        disable_refresh: false
      <YOUR_SECRET_CONFIG_ID_2>:
        # ...

Secrets

The secrets configuration supports the following parameters.

Reference secrets

In pipelines.yaml, reference secret values in plugin settings using:

• Plaintext: ${{aws_secrets:<YOUR_SECRET_CONFIG_ID>}}
• JSON key: ${{aws_secrets:<YOUR_SECRET_CONFIG_ID>:<YOUR_KEY>}}

You can use the following setting types for secret substitution: string, number, long, short, integer, double, float, Boolean, or character.

The following is an example data-prepper-config.yaml with two secret configuration IDs (host-secret-config and credential-secret-config):

extensions:
  aws:
    secrets:
      host-secret-config:
        secret_id: <YOUR_SECRET_ID_1>
        region: <YOUR_REGION_1>
        sts_role_arn: <YOUR_STS_ROLE_ARN_1>
        refresh_interval: <YOUR_REFRESH_INTERVAL_1>
      credential-secret-config:
        secret_id: <YOUR_SECRET_ID_2>
        region: <YOUR_REGION_2>
        sts_role_arn: <YOUR_STS_ROLE_ARN_2>
        refresh_interval: <YOUR_REFRESH_INTERVAL_2>

You can then reference these secrets in pipelines.yaml as follows:

sink:
  - opensearch:
      hosts: ["${{aws_secrets:host-secret-config}}"]
      username: "${{aws_secrets:credential-secret-config:username}}"
      password: "${{aws_secrets:credential-secret-config:password}}"
      index: "test-migration"

Automatically refreshing secrets

For each secret configuration, Data Prepper polls the latest value at a regular interval to support rotating secrets in AWS Secrets Manager. Refreshed values are used by plugins that can refresh their connections/authentication (for example, sinks). For multiple secret configurations, a jitter of up to 60s is applied across all configurations during the initial polling.