Regex conditionals
Ingest pipelines support conditional logic using regular expressions (regex) with the Painless scripting language. For more information, see Painless scripting language. This allows fine-grained control over which documents get processed based on the structure and contents of text fields. Regex can be used within the if parameter to evaluate string patterns. This is especially useful for matching IP formats, validating email addresses, identifying UUIDs, or processing logs with specific keywords.
Regular expressions can be expensive to evaluate. To limit their CPU consumption, OpenSearch limits the number of characters that a Painless regular expression can read, based on the script.painless.regex.enabled and script.painless.regex.limit-factor settings. For more information, see Controlling regular expressions.
Example: Email domain filtering
The following pipeline uses regex to identify users from the @example.com email domain and tag those documents accordingly:
PUT _ingest/pipeline/tag_example_com_users
{
"processors": [
{
"set": {
"field": "user_domain",
"value": "example.com",
"if": "ctx.email != null && ctx.email =~ /@example\\.com$/"
}
}
]
}
Use the following request to simulate the pipeline:
POST _ingest/pipeline/tag_example_com_users/_simulate
{
"docs": [
{ "_source": { "email": "alice@example.com" } },
{ "_source": { "email": "bob@another.com" } }
]
}
Only the first document has user_domain added:
{
"docs": [
{
"doc": {
"_source": {
"email": "alice@example.com",
"user_domain": "example.com"
}
}
},
{
"doc": {
"_source": {
"email": "bob@another.com"
}
}
}
]
}
Example: Detect IPv6 addresses
The following pipeline uses regex to identify and flag IPv6-formatted addresses:
PUT _ingest/pipeline/ipv6_flagger
{
"processors": [
{
"set": {
"field": "ip_type",
"value": "IPv6",
"if": "ctx.ip != null && ctx.ip =~ /^[a-fA-F0-9:]+$/ && ctx.ip.contains(':')"
}
}
]
}
Use the following request to simulate the pipeline:
POST _ingest/pipeline/ipv6_flagger/_simulate
{
"docs": [
{ "_source": { "ip": "2001:0db8:85a3:0000:0000:8a2e:0370:7334" } },
{ "_source": { "ip": "192.168.0.1" } }
]
}
The first document contains an added ip_type field set to IPv6:
{
"docs": [
{
"doc": {
"_source": {
"ip": "2001:0db8:85a3:0000:0000:8a2e:0370:7334",
"ip_type": "IPv6"
}
}
},
{
"doc": {
"_source": {
"ip": "192.168.0.1"
}
}
}
]
}
Example: Validate UUID strings
The following pipeline uses regex to verify whether a session_id field contains a valid UUID:
PUT _ingest/pipeline/uuid_checker
{
"processors": [
{
"set": {
"field": "valid_uuid",
"value": true,
"if": "ctx.session_id != null && ctx.session_id =~ /^[a-f0-9]{8}-[a-f0-9]{4}-4[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$/"
}
}
]
}
Use the following request to simulate the pipeline:
POST _ingest/pipeline/uuid_checker/_simulate
{
"docs": [
{ "_source": { "session_id": "550e8400-e29b-41d4-a716-446655440000" } },
{ "_source": { "session_id": "invalid-uuid-1234" } }
]
}
The first document is tagged with a new valid_uuid field:
{
"docs": [
{
"doc": {
"_source": {
"session_id": "550e8400-e29b-41d4-a716-446655440000",
"valid_uuid": true
}
}
},
{
"doc": {
"_source": {
"session_id": "invalid-uuid-1234"
}
}
}
]
}
Using string methods for simple matches
String methods are less expensive to evaluate than regular expressions. When a string method can perform the same match, use the string method. For example, the following condition is equivalent to the email domain example but uses the endsWith method:
PUT _ingest/pipeline/tag_example_com_users
{
"processors": [
{
"set": {
"field": "user_domain",
"value": "example.com",
"if": "ctx.email != null && ctx.email.endsWith('@example.com')"
}
}
]
}
Other string methods that can replace common regular expressions include startsWith and contains.