Link Search Menu Expand Document Documentation Menu
Documentation

Date processor

The date processor adds a default timestamp to an event, parses timestamp fields, and converts timestamp information to the International Organization for Standardization (ISO) 8601 format. This timestamp information can be used as an event timestamp.

Configuration

The following table describes the options you can use to configure the date processor.

Option Required Type Description
match Conditionally Match The date match configuration. This option cannot be defined at the same time as from_time_received. There is no default value.
from_time_received Conditionally Boolean When true, the timestamp from the event metadata, which is the time at which the source receives the event, is added to the event data. This option cannot be defined at the same time as match. Default is false.
date_when No String Specifies under what condition the date processor should perform matching. Default is no condition.
to_origination_metadata No Boolean When true, the matched time is also added to the event’s metadata as an instance of Instant. Default is false.
destination No String The field used to store the timestamp parsed by the date processor. Can be used with both match and from_time_received. Default is @timestamp.
output_format No String Determines the format of the timestamp added to an event. Default is yyyy-MM-dd'T'HH:mm:ss.SSSXXX.
source_timezone No String The time zone used to parse dates, including when the zone or offset cannot be extracted from the value. If the zone or offset are part of the value, then the time zone is ignored. A list of all the available time zones is contained in the TZ database name column of the list of database time zones.
destination_timezone No String The time zone used for storing the timestamp in the destination field. A list of all the available time zones is contained in the TZ database name column of the list of database time zones.
locale No String The location used for parsing dates. Commonly used for parsing month names (MMM). The value can contain language, country, or variant fields in IETF BCP 47, such as en-US, or a string representation of the locale object, such as en_US. A full list of locale fields, including language, country, and variant, can be found in the language subtag registry. Default is Locale.ROOT.

Match

Option Required Type Description
key Yes String Represents the event key against which to match patterns. Required if match is configured.
patterns Yes List A list of possible patterns that the timestamp value of the key can have. The patterns are based on a sequence of letters and symbols. The patterns support all the patterns listed in the Java DatetimeFormatter reference. The timestamp value also supports epoch_second, epoch_milli, and epoch_nano values, which represent the timestamp as the number of seconds, milliseconds, and nanoseconds since the epoch. Epoch values always use the UTC time zone.

Metrics

The following table describes common Abstract processor metrics.

Metric name Type Description
recordsIn Counter Metric representing the ingress of records to a pipeline component.
recordsOut Counter Metric representing the egress of records from a pipeline component.
timeElapsed Timer Metric representing the time elapsed during execution of a pipeline component.

The date processor includes the following custom metrics.

  • dateProcessingMatchSuccessCounter: Returns the number of records that match at least one pattern specified by the match configuration option.
  • dateProcessingMatchFailureCounter: Returns the number of records that did not match any of the patterns specified by the patterns match configuration option.

Pattern reference

The following table provides common DateTime patterns.

Pattern Example Description
yyyy-MM-dd'T'HH:mm:ss.SSSXXX 2023-03-15T14:30:45.123-05:00 ISO 8601 format with time zone
dd/MMM/yyyy:HH:mm:ss Z 15/Mar/2023:14:30:45 -0500 Apache Common Log Format
MMM dd HH:mm:ss Mar 15 14:30:45 Syslog format
yyyy-MM-dd HH:mm:ss 2023-03-15 14:30:45 Standard SQL format
MM/dd/yyyy HH:mm:ss 03/15/2023 14:30:45 US format

Epoch patterns

The following table provides supported epoch patterns.

Pattern Description Example value
epoch_second Seconds since Unix epoch (1970-01-01 00:00:00 UTC) 1678902000
epoch_milli Milliseconds since Unix epoch 1678902000123
epoch_micro Microseconds since Unix epoch 1678902000123456
epoch_nano Nanoseconds since Unix epoch 1678902000123456789

Only one epoch pattern is allowed per pattern list. To support multiple epoch formats, use separate date processors.

Examples

The following are examples of different configurations.

The examples don’t use security and are for demonstration purposes only. We strongly recommend configuring SSL before using these examples in production.

Add the default timestamp to an event using ingestion time

The following date processor configuration can be used to add a default timestamp in the @timestamp field applied to all events:

date-ingestion-pipeline:
  source:
    http:
      path: /events
      ssl: false

  processor:
    - date:
        from_time_received: true
        destination: "@timestamp"

  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        insecure: true
        username: admin
        password: admin_pass
        index_type: custom
        index: date-ingestion-%{yyyy.MM.dd}

You can test this pipeline using the following command:

curl -sS -X POST "http://localhost:2021/events" \
  -H "Content-Type: application/json" \
  -d '[
        {"app":"web-server","env":"prod","message":"User login successful","level":"info","user_id":"12345"},
        {"app":"database","env":"stage","message":"Connection timeout","level":"error","retry_count":3},
        {"app":"api-gateway","env":"dev","message":"Request processed","level":"debug","response_time_ms":145}
      ]'

The documents stored in OpenSearch contain the following information:

{
  ...
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "date-ingestion-2025.10.13",
        "_id": "984A3ZkBoQvfnsgqa4od",
        "_score": 1,
        "_source": {
          "app": "web-server",
          "env": "prod",
          "message": "User login successful",
          "level": "info",
          "user_id": "12345",
          "@timestamp": "2025-10-13T09:56:23.090Z"
        }
      },
      {
        "_index": "date-ingestion-2025.10.13",
        "_id": "-M4A3ZkBoQvfnsgqa4od",
        "_score": 1,
        "_source": {
          "app": "database",
          "env": "stage",
          "message": "Connection timeout",
          "level": "error",
          "retry_count": 3,
          "@timestamp": "2025-10-13T09:56:23.095Z"
        }
      },
      {
        "_index": "date-ingestion-2025.10.13",
        "_id": "-c4A3ZkBoQvfnsgqa4od",
        "_score": 1,
        "_source": {
          "app": "api-gateway",
          "env": "dev",
          "message": "Request processed",
          "level": "debug",
          "response_time_ms": 145,
          "@timestamp": "2025-10-13T09:56:23.096Z"
        }
      }
    ]
  }
}

Parse a timestamp to convert its format and time zone

The following example demonstrates how the date processor can sequentially parse different timestamp formats (such as epoch milliseconds, epoch seconds, ISO 8601, or Apache Common Log Format) using multiple processors, each handling one specific format:

date-multi-format-pipeline:
  source:
    http:
      path: /events
      ssl: false

  processor:
    # Handle epoch formats (separate processors for different epoch types)
    - date:
        match:
          - key: event_time
            patterns: ["epoch_milli"]
        destination: "@timestamp"
        output_format: yyyy-MM-dd'T'HH:mm:ss.SSSXXX
        source_timezone: UTC
        destination_timezone: America/New_York
        locale: en_US
        date_when: "/event_time != null"
    
    - date:
        match:
          - key: event_time
            patterns: ["epoch_second"]
        destination: "@timestamp"
        output_format: yyyy-MM-dd'T'HH:mm:ss.SSSXXX
        source_timezone: UTC
        destination_timezone: America/New_York
        locale: en_US
        date_when: "/event_time != null and /@timestamp == null"
    
    # Handle ISO format variations in a single processor
    - date:
        match:
          - key: event_time
            patterns: ["yyyy-MM-dd'T'HH:mm:ss.SSSXXX"]
        destination: "@timestamp"
        output_format: yyyy-MM-dd'T'HH:mm:ss.SSSXXX
        source_timezone: UTC
        destination_timezone: America/New_York
        locale: en_US
        date_when: "/event_time != null and /@timestamp == null"
    
    # Handle Apache format variations in a single processor
    - date:
        match:
          - key: event_time
            patterns: ["dd/MMM/yyyy:HH:mm:ss Z"]
        destination: "@timestamp"
        output_format: yyyy-MM-dd'T'HH:mm:ss.SSSXXX
        source_timezone: UTC
        destination_timezone: America/New_York
        locale: en_US
        date_when: "/event_time != null and /@timestamp == null"

  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        insecure: true
        username: admin
        password: admin_pass
        index_type: custom
        index: date-multi-format-%{yyyy.MM.dd}

You can test this pipeline using the following command:

curl -sS -X POST "http://localhost:2021/events" \
  -H "Content-Type: application/json" \
  -d '[
        {"app":"payment-service","event_time":"1678902000123","transaction_id":"txn_001","amount":99.99,"currency":"USD"},
        {"app":"user-service","event_time":"1678902000","user_id":"user_123","action":"login","ip":"192.168.1.100"},
        {"app":"analytics","event_time":"2023-03-15T14:30:45.123-05:00","event_type":"page_view","page":"/home","session_id":"sess_456"},
        {"app":"web-server","event_time":"15/Mar/2023:14:30:45 -0500","client_ip":"10.0.0.1","method":"GET","url":"/api/users","status":200}
      ]'

The documents stored in OpenSearch contain the following information:

{
  ...
  "hits": {
    "total": {
      "value": 4,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "date-multi-format-2025.10.13",
        "_id": "8M4A3ZkBoQvfnsgqaorB",
        "_score": 1,
        "_source": {
          "app": "payment-service",
          "event_time": "1678902000123",
          "transaction_id": "txn_001",
          "amount": 99.99,
          "currency": "USD",
          "@timestamp": "2023-03-15T13:40:00.123-04:00"
        }
      },
      {
        "_index": "date-multi-format-2025.10.13",
        "_id": "8c4A3ZkBoQvfnsgqaorB",
        "_score": 1,
        "_source": {
          "app": "user-service",
          "event_time": "1678902000",
          "user_id": "user_123",
          "action": "login",
          "ip": "192.168.1.100",
          "@timestamp": "2023-03-15T13:40:00.000-04:00"
        }
      },
      {
        "_index": "date-multi-format-2025.10.13",
        "_id": "8s4A3ZkBoQvfnsgqaorB",
        "_score": 1,
        "_source": {
          "app": "analytics",
          "event_time": "2023-03-15T14:30:45.123-05:00",
          "event_type": "page_view",
          "page": "/home",
          "session_id": "sess_456",
          "@timestamp": "2023-03-15T15:30:45.123-04:00"
        }
      },
      {
        "_index": "date-multi-format-2025.10.13",
        "_id": "884A3ZkBoQvfnsgqaorB",
        "_score": 1,
        "_source": {
          "app": "web-server",
          "event_time": "15/Mar/2023:14:30:45 -0500",
          "client_ip": "10.0.0.1",
          "method": "GET",
          "url": "/api/users",
          "status": 200,
          "@timestamp": "2023-03-15T15:30:45.000-04:00"
        }
      }
    ]
  }
}

Timestamp formats with different day spacing

The following example demonstrates how the date processor handles variable syslog timestamp formats with different day spacing using pattern variations:

date-syslog-pipeline:
  source:
    http:
      path: /events
      ssl: false

  processor:
    - date:
        match:
          - key: syslog_timestamp
            patterns: ["MMM d HH:mm:ss", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
        destination: "@timestamp"
        output_format: yyyy-MM-dd'T'HH:mm:ss.SSSXXX
        source_timezone: America/Los_Angeles
        destination_timezone: UTC
        locale: en_US
        date_when: /syslog_timestamp != null
        origination_timestamp_to_metadata: true

  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        insecure: true
        username: admin
        password: admin_password
        index_type: custom
        index: date-syslog-%{yyyy.MM.dd}

You can test the pipeline using the following command:

curl -sS -X POST "http://localhost:2021/events" \
  -H "Content-Type: application/json" \
  -d '[
        {"host":"web-server-01","syslog_timestamp":"Mar 15 14:30:45","facility":"daemon","severity":"info","message":"sshd[1234]: Accepted publickey for user1"},
        {"host":"database-02","syslog_timestamp":"Mar  5 09:15:30","facility":"auth","severity":"warning","message":"mysql[5678]: Access denied for user '\''root'\''@'\''localhost'\''"},
        {"host":"app-server-03","syslog_timestamp":"Mar 25 23:45:12","facility":"local0","severity":"error","message":"nginx[9012]: upstream timed out (110: Connection timed out)"},
        {"host":"cache-server-04","message":"redis[3456]: Background saving terminated with success"}
      ]'

The documents stored in OpenSearch contain the following information:

{
  ...
  "hits": {
    "total": {
      "value": 4,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "date-syslog-2025.10.13",
        "_id": "5s4A3ZkBoQvfnsgqZIrV",
        "_score": 1,
        "_source": {
          "host": "web-server-01",
          "syslog_timestamp": "Mar 15 14:30:45",
          "facility": "daemon",
          "severity": "info",
          "message": "sshd[1234]: Accepted publickey for user1",
          "@timestamp": "2025-03-15T21:30:45.000Z"
        }
      },
      {
        "_index": "date-syslog-2025.10.13",
        "_id": "584A3ZkBoQvfnsgqZIrV",
        "_score": 1,
        "_source": {
          "host": "database-02",
          "syslog_timestamp": "Mar  5 09:15:30",
          "facility": "auth",
          "severity": "warning",
          "message": "mysql[5678]: Access denied for user 'root'@'localhost'",
          "@timestamp": "2025-03-05T17:15:30.000Z"
        }
      },
      {
        "_index": "date-syslog-2025.10.13",
        "_id": "6M4A3ZkBoQvfnsgqZIrV",
        "_score": 1,
        "_source": {
          "host": "app-server-03",
          "syslog_timestamp": "Mar 25 23:45:12",
          "facility": "local0",
          "severity": "error",
          "message": "nginx[9012]: upstream timed out (110: Connection timed out)",
          "@timestamp": "2025-03-26T06:45:12.000Z"
        }
      },
      {
        "_index": "date-syslog-2025.10.13",
        "_id": "6c4A3ZkBoQvfnsgqZIrV",
        "_score": 1,
        "_source": {
          "host": "cache-server-04",
          "message": "redis[3456]: Background saving terminated with success"
        }
      }
    ]
  }
}
350 characters left

Have a question? .

Want to contribute? or .