Link Search Menu Expand Document Documentation Menu
Documentation

Delete entries processor

The delete_entries processor removes entries, such as key-value pairs, from an event. Use the with_keys field to specify the exact keys to delete. To delete keys that match a regular expression pattern, use the with_keys_regex field. You can prevent deletion of specific events when using regular expressions by configuring the exclude_from_regex field.

The only way to configure both with_keys and with_keys_regex in the same delete_entries processor is by using the entries field.

Configuration

You can configure the delete_entries processor with the following options.

Option Required Description
with_keys No An array that specifies the keys of the entries to delete.
with_keys_regex No An array of regular expression (regex) patterns used to match the keys of entries to delete.
exclude_from_delete No A set of entries to exclude from deletion when using the with_keys_regex configuration.
entries No A list of entries to delete from the event.
delete_when No Defines the condition under which the deletion is performed. For example, value="/some_key == null" deletes the key only if /some_key is null or does not exist.
delete_from_element_when No Defines the condition that determines whether a key–value pair should be removed from each element in the list specified by iterate_on. The condition is evaluated for each element, and deletion occurs only if the element’s key matches one defined in with_keys or with_keys_regex and satisfies the condition.
iterate_on No Specifies the key of the list field that contains objects to iterate over. The processor applies any configured deletion rules, such as with_keys, with_keys_regex, or delete_from_element_when, to each element in the list.

Usage

To get started, create the following pipeline.yaml file:

pipeline:
  source:
    ...
  ....
  processor:
    - delete_entries:
        with_keys: [ "message" ]
  sink:

Next, create a log file named logs_json.log and replace the path in the file source of your pipeline.yaml file with that filepath. For more information, see Configuring OpenSearch Data Prepper.

For example, before you run the delete_entries processor, if the logs_json.log file contains the following event record:

{"message": "hello", "message2": "goodbye"}

When you run the delete_entries processor, it parses the message into the following output:

{"message2": "goodbye"}

Deleting keys that match a pattern

First, create the following pipeline.yaml file:

pipeline:
  source:
    ...
  ....
  processor:
    - delete_entries:
        with_keys_regex: [ "^tes.*" ]
        exclude_from_delete: [ "test" ]
  sink:

If your logs_json.log file contains the following event record:

{"test": "friends", "test2": "are", "test3": "kind"}

When you run the delete_entries processor, it parses the message into the following output:

{"test": "friends"}

Combining multiple deletion rules

First, create the following pipeline.yaml file:

pipeline:
  source:
    ...
  ....
  processor:
    - delete_entries:
        entries:
          - with_keys: [ "message" ]
          - with_keys_regex: [ "^tes.*" ]
            exclude_from_delete: [ "test" ]
  sink:

If your logs_json.log file contains the following event record:

{"message": "hello", "message2": "goodbye", "test": "friends", "test2": "are", "test3": "kind"}

When you run the delete_entries processor, it parses the message into the following output:

{"message2": "goodbye","test": "friends"}

If the with_keys, with_keys_regex, or exclude_from_delete values do not match any event keys, then no action occurs.

350 characters left

Have a question? .

Want to contribute? or .