HTTP source

The http plugin accepts HTTP requests from clients. The following table describes options you can use to configure the http source.

Ingestion

Clients should send HTTP POST requests to the endpoint /log/ingest.

The http protocol only supports the JSON UTF-8 codec for incoming requests, for example, [{"key1": "value1"}, {"key2": "value2"}].

Example

The following examples demonstrate different configurations that can be used with the http source.

Minimal HTTP source

The following is the minimal configuration using all default values:

minimal-http-pipeline:
  source:
    http:
  sink:
    - stdout: {}

You can test this pipeline using the following command:

curl -s "http://localhost:2021/log/ingest" \
  -H "Content-Type: application/json" \
  --data '[{"msg":"one"},{"msg":"two"}]'

You should see the following output in the Data Prepper logs:

{"msg":"one"}
{"msg":"two"}

Custom path using the pipeline name and health check

The following example uses a custom path, configures a custom port, and enables health checks:

audit-pipeline:
  source:
    http:
      port: 2022
      path: "/${pipelineName}/logs"  # -> /audit-pipeline/logs
      health_check_service: true
      unauthenticated_health_check: true
  sink:
    - stdout: {}

You can use the following command to check the pipeline health:

curl -s "http://localhost:2022/health"

You can ingest data using the following command:

curl -s "http://localhost:2022/audit-pipeline/logs" \
  -H "Content-Type: application/json" \
  --data '[{"event":"login","user":"alice"}]'

Basic authentication on the source

The following example configures a custom port and path, enables health checks, and configures basic authentication:

secure-intake-pipeline:
  source:
    http:
      port: 2023
      path: /ingest
      authentication:
        http_basic:
          username: ingest
          password: s3cr3t
      health_check_service: true
      unauthenticated_health_check: true
  sink:
    - stdout: {}
    - opensearch:
        hosts: ["https://opensearch:9200"]
        insecure: true
        username: admin
        password: admin_password
        index_type: custom
        index: demo-%{yyyy.MM.dd}

You can test this pipeline using the following command:

curl -s -u ingest:s3cr3t "http://localhost:2023/ingest" \
  -H "Content-Type: application/json" \
  --data '[{"service":"web","status":"ok"}]'

Metrics

The http source includes the following metrics.

Counters

• requestsReceived: Measures the total number of requests received by the /log/ingest endpoint.
• requestsRejected: Measures the total number of requests rejected (429 response status code) by the HTTP Source plugin.
• successRequests: Measures the total number of requests successfully processed (200 response status code) the by HTTP Source plugin.
• badRequests: Measures the total number of requests with either an invalid content type or format processed by the HTTP Source plugin (400 response status code).
• requestTimeouts: Measures the total number of requests that time out in the HTTP source server (415 response status code).
• requestsTooLarge: Measures the total number of requests where the size of the event is larger than the buffer capacity (413 response status code).
• internalServerError: Measures the total number of requests processed by the HTTP Source with a custom exception type (500 response status code).

Timers

• requestProcessDuration: Measures the latency of requests processed by the HTTP Source plugin in seconds.

Distribution summaries

• payloadSize: Measures the incoming request payload size in bytes.