Mapping explosion

Mapping explosion occurs when an index accumulates an excessive number of fields, which can lead to performance degradation, memory issues, and cluster instability. This situation commonly arises when using dynamic mapping with highly variable document structures, where each new document introduces additional fields that are automatically added to the index mapping.

When OpenSearch encounters new fields in documents, it automatically creates mappings for these fields through dynamic mapping. While this feature provides flexibility, it can become problematic in scenarios such as:

• Log data with varying structures: Different log sources may include unique fields, leading to rapid field proliferation.
• User-generated content: Applications that allow users to define custom fields or attributes.
• Nested object structures: Documents with deeply nested objects that contain many subfields.
• Time-series data: Metrics or events that include dynamic field names based on timestamps or identifiers.

As the number of fields grows, several issues can emerge:

• Increased memory consumption for storing field mappings
• Slower query performance due to larger mapping structures
• Potential out-of-memory errors during indexing or searching
• Difficulty in cluster recovery scenarios

Mapping limit settings

OpenSearch provides several index-level settings to prevent mapping explosion by limiting various aspects of mapping growth. These settings can be configured when creating an index or updated for existing indexes:

PUT /my-index/_settings
{
  "index.mapping.total_fields.limit": 2000
}

The following table lists all available mapping limit settings. All settings are dynamic. For more information, see Dynamic settings.

Best practices

To avoid mapping explosion, follow these guidelines.

Use explicit mappings

Define explicit mappings whenever possible instead of relying on dynamic mapping:

PUT /logs
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      },
      "message": {
        "type": "text"
      },
      "level": {
        "type": "keyword"
      },
      "source": {
        "type": "keyword"
      }
    }
  }
}

Configure dynamic mapping templates

Use dynamic templates to control how new fields are mapped:

PUT /logs
{
  "mappings": {
    "dynamic_templates": [
      {
        "strings_as_keywords": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}

Use the flat_object field type

For documents with arbitrary key-value pairs, use the flat_object field type instead of allowing dynamic mapping:

PUT /products
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "attributes": {
        "type": "flat_object"
      }
    }
  }
}

Disable dynamic mapping

For indexes with well-defined schemas, disable dynamic mapping entirely:

PUT /structured-data
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "id": {
        "type": "keyword"
      },
      "value": {
        "type": "double"
      }
    }
  }
}

Monitoring and maintenance

Regular monitoring helps you detect mapping growth early and take action before it affects cluster performance. You can monitor field mappings in the following ways.

Check the current field count

Monitor the number of fields in your indexes:

GET /my-index/_mapping

You can also use the Cluster Stats API to get field count information:

GET /_cluster/stats

Identify problematic indexes

Use index statistics to find indexes with high field counts:

GET /_cat/indices?v&h=index,docs.count,store.size,pri.store.size&s=store.size:desc

Clean up unused fields

For indexes with dynamic mapping enabled, regularly review and clean up fields that are no longer needed by reindexing with a more restrictive mapping.

Recovery from mapping explosion

If an index has already experienced mapping explosion:

1. Determine which fields are actually needed.
2. Create a new index with explicit mappings and appropriate limits.
3. Reindex the data using the Reindex API, filtering out unnecessary fields.
4. Update aliases to point to the new index.
5. Delete the old index once the migration is complete:

POST /_reindex
{
  "source": {
    "index": "old-index"
  },
  "dest": {
    "index": "new-index"
  },
  "script": {
    "source": "ctx._source.remove('unwanted_field')"
  }
}