Diversified sampler aggregation

The diversified_sampler aggregation is a filtering aggregation that limits subaggregation processing to a sample of top-scoring documents while ensuring the sample contains diverse content. It extends the sampler aggregation by deduplicating documents that share a common field value, preventing any single category from dominating the sample.

Because the sample contains only the top-scoring documents, the subaggregation results focus on the most relevant matches and are not diluted by a long tail of weak matches. This aggregation is also useful when you need to ensure fair representation across different groups—for example, preventing a single prolific author from skewing analytics results or ensuring geographic diversity in location-based analysis. It also reduces the cost of expensive subaggregations like significant_terms by producing useful results from a smaller, more representative sample.

Parameters

The diversified_sampler aggregation takes the following parameters.

Execution hint

The following table lists the valid execution_hint values.

By default, OpenSearch uses global_ordinals if the field has ordinals and map otherwise. All three options produce the same sample unless bytes_hash treats two different values as duplicates because their hashes collide. They differ only in the type of value held in memory, and each option holds up to shard_size values. OpenSearch ignores the execution_hint if the chosen option does not apply to the field type, and an unrecognized value causes the request to fail.

The diversified_sampler aggregation can be nested under a terms aggregation. If the terms aggregation uses the breadth_first collect mode, OpenSearch replays the relevance scores of the deferred documents, so the sample is still based on the top-scoring documents.

Example: Deduplicating by field

A sample of the top-matching documents can be dominated by a few prolific sources, such as a customer who places many similar orders. In the sample ecommerce data, one customer placed 103 of the 646 orders containing a dress. A plain sampler with a shard_size of 100 contains orders from only 16 customers.

The following example samples the top-matching orders containing a dress but allows only one order for each customer_id, so that the subaggregations reflect many customers equally:

GET /opensearch_dashboards_sample_data_ecommerce/_search
{
  "size": 0,
  "query": {
    "match": {
      "products.product_name": "dress"
    }
  },
  "aggs": {
    "unbiased_sample": {
      "diversified_sampler": {
        "shard_size": 100,
        "field": "customer_id"
      },
      "aggs": {
        "customers": {
          "cardinality": {
            "field": "customer_id"
          }
        },
        "top_manufacturers": {
          "terms": {
            "field": "manufacturer.keyword",
            "size": 3
          }
        }
      }
    }
  }
}

The sample contains 22 orders, one from each of the 22 customers who ordered a dress:

{
  "aggregations": {
    "unbiased_sample": {
      "doc_count": 22,
      "top_manufacturers": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 17,
        "buckets": [
          {
            "key": "Tigress Enterprises",
            "doc_count": 7
          },
          {
            "key": "Low Tide Media",
            "doc_count": 6
          },
          {
            "key": "Oceanavigations",
            "doc_count": 6
          }
        ]
      },
      "customers": {
        "value": 22
      }
    }
  }
}

Example: Deduplicating by script

Use a script to diversify on a combination of values. The following example allows at most three orders for each combination of customer_gender and day_of_week:

GET /opensearch_dashboards_sample_data_ecommerce/_search
{
  "size": 0,
  "query": {
    "match": {
      "products.product_name": "dress"
    }
  },
  "aggs": {
    "unbiased_sample": {
      "diversified_sampler": {
        "shard_size": 100,
        "max_docs_per_value": 3,
        "script": {
          "lang": "painless",
          "source": "doc['customer_gender'].value + '-' + doc['day_of_week'].value"
        }
      },
      "aggs": {
        "by_day": {
          "terms": {
            "field": "day_of_week",
            "size": 7
          }
        }
      }
    }
  }
}

The sample contains 27 orders: at most three for each combination of gender and day of the week. Because each day can contain orders from both female and male customers, some days contain more than three orders:

{
  "aggregations": {
    "unbiased_sample": {
      "doc_count": 27,
      "by_day": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Thursday",
            "doc_count": 5
          },
          {
            "key": "Friday",
            "doc_count": 4
          },
          {
            "key": "Sunday",
            "doc_count": 4
          },
          {
            "key": "Tuesday",
            "doc_count": 4
          },
          {
            "key": "Wednesday",
            "doc_count": 4
          },
          {
            "key": "Monday",
            "doc_count": 3
          },
          {
            "key": "Saturday",
            "doc_count": 3
          }
        ]
      }
    }
  }
}

Response body fields

The following table lists the response body fields.

Limitations

The diversified_sampler aggregation has the following limitations:

• The field or script must produce a single value per document. Using a multi-valued field, such as manufacturer.keyword in the sample ecommerce data, causes the request to fail. To diversify on all values of a multi-valued field, use a script that combines them into one value, for example, doc['manufacturer.keyword'].hashCode().
• Deduplication is applied independently on each shard, so documents with the same value on different shards are not deduplicated against each other.
• There is no specialized syntax for geographic or date-based diversity values (such as 7d or 10km). To diversify by geographic region or time interval, write a script that buckets the raw values—for example, (int)(doc['geoip.location'].lat / 10) for latitude bands or doc['order_date'].value.dayOfWeek for day-of-week grouping.