Ingest processors
Ingest processors are a core component of ingest pipelines. They preprocess documents before indexing. For example, you can remove fields, extract values from text, convert data formats, or append additional information.
OpenSearch provides a standard set of ingest processors within your OpenSearch installation. For a list of processors available in OpenSearch, use the Nodes Info API operation:
GET /_nodes/ingest?filter_path=nodes.*.ingest.processors
Every processor in a pipeline must be installed on all nodes in the cluster. If a processor is missing from any node, the Create or Update Pipeline API request fails. If you use processors provided by a plugin, list the plugin in the plugin.mandatory setting so that a node without the plugin doesn’t start.
To set up and deploy ingest processors, make sure you have the necessary permissions and access rights. For more information, see API permissions.
Supported processors
Processor types and their required or optional parameters vary depending on your specific use case. OpenSearch supports the following ingest processors. For tutorials on using these processors in an OpenSearch pipeline, go to each processor’s respective documentation.
| Processor type | Description |
|---|---|
append | Adds one or more values to a field in a document. |
bytes | Converts a human-readable byte value to its value in bytes. |
community_id | Generates a community ID flow hash algorithm for the network flow tuples. |
convert | Changes the data type of a field in a document. |
copy | Copies an entire object in an existing field to another field. |
csv | Extracts CSVs and stores them as individual fields in a document. |
date | Parses dates from fields and then uses the date or timestamp as the timestamp for a document. |
date_index_name | Indexes documents into time-based indexes based on a date or timestamp field in a document. |
dissect | Extracts structured fields from a text field using a defined pattern. |
dot_expander | Expands a field with dots into an object field. |
drop | Drops a document without indexing it or raising any errors. |
fail | Raises an exception and stops the execution of a pipeline. |
fingerprint | Generates a hash value for either certain specified fields or all fields in a document. |
foreach | Applies another processor to each element of an array field in a document. |
geojson-feature | Indexes GeoJSON data into a geospatial field. |
grok | Parses and structures unstructured data using pattern matching. |
gsub | Replaces or deletes substrings within a string field of a document. |
html_strip | Removes HTML tags from a text field and returns the plain text content. |
ip2geo | Adds information about the geographical location of an IPv4 or IPv6 address. |
join | Concatenates each element of an array into a single string using a separator character between each element. |
json | Converts a JSON string into a structured JSON object. |
kv | Automatically parses key-value pairs in a field. |
lowercase | Converts text in a specific field to lowercase letters. |
ml_inference | Runs a machine learning model registered in ML Commons and adds the model outputs to the document. |
pipeline | Runs an inner pipeline. |
remove | Removes fields from a document. |
remove_by_pattern | Removes fields from a document by field pattern. |
rename | Renames an existing field. |
script | Runs an inline or stored script on incoming documents. |
set | Sets the value of a field to a specified value. |
sort | Sorts the elements of an array in ascending or descending order. |
sparse_encoding | Generates a sparse vector/token and weights from text fields for neural sparse search using sparse retrieval. |
split | Splits a field into an array using a separator character. |
text_chunking | Splits long documents into smaller chunks. |
text_embedding | Generates vector embeddings from text fields for semantic search. |
text_image_embedding | Generates combined vector embeddings from text and image fields for multimodal neural search. |
trim | Removes leading and trailing white space from a string field. |
uppercase | Converts text in a specific field to uppercase letters. |
urldecode | Decodes a string from URL-encoded format. |
user_agent | Extracts details from the user agent sent by a browser to its web requests. |
Common processor parameters
Each processor has its own parameters. Additionally, all processors support the following optional parameters.
| Parameter | Type | Description |
|---|---|---|
description | String | A description of the processor. |
if | String or object | A Painless script condition, specified either as a string or as a script object containing source, lang, and params. The processor runs only if the condition evaluates to true. For more information, see Conditional execution. |
ignore_failure | Boolean | If true, the pipeline continues to run if the processor fails. Default is false. For more information, see Handling pipeline failures. |
on_failure | Array of processor objects | Processors to run if the processor fails. For more information, see Handling pipeline failures. |
tag | String | An identifier for the processor. The tag appears in error metadata and in ingest statistics returned by the Nodes Stats API, which helps you identify a specific processor in a pipeline. For more information, see Using processor tags for monitoring and debugging. |
The following lowercase processor normalizes the coupon_code field. It runs only for documents in which the coupon code is longer than four characters, and if it fails, it records the error message in the coupon_error field so that the pipeline continues:
{
"lowercase": {
"description": "Normalizes coupon codes",
"tag": "normalize-coupon",
"if": "ctx.coupon_code instanceof String && ctx.coupon_code.length() > 4",
"field": "coupon_code",
"on_failure": [
{
"set": {
"field": "coupon_error",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}
}
Processor limit settings
You can limit the number of ingest processors using the cluster setting cluster.ingest.max_number_processors. The total number of processors includes both the number of processors and the number of on_failure processors.
The default value for cluster.ingest.max_number_processors is Integer.MAX_VALUE. Adding a higher number of processors than the value configured in cluster.ingest.max_number_processors will throw an IllegalStateException.
Batch-enabled processors
Some processors support batch ingestion—they can process multiple documents at the same time as a batch. These batch-enabled processors usually provide better performance when using batch processing. For batch processing, use the Bulk API and provide a batch_size parameter. All batch-enabled processors have a batch mode and a single-document mode. When you ingest documents using the PUT method, the processor functions in single-document mode and processes documents in series. Only the text_embedding and sparse_encoding processors are batch enabled. All other processors process documents one at a time.
Selectively enabling processors
Processors defined by the ingest-common module can be selectively enabled by providing the ingest.common.processors.allowed static setting in opensearch.yml. If not provided, then all processors are enabled by default. Specifying an empty list disables all processors. If the setting is changed to remove previously enabled processors, then any pipeline using a disabled processor will fail after node restart when the new setting takes effect.