Variable width histogram aggregation

The variable_width_histogram aggregation divides a numeric field’s values into a target number of buckets whose widths adapt to the data. Bucket boundaries are derived by clustering the values, so dense parts of the value range are split into narrow buckets and sparse parts are covered by wide ones. Use this aggregation for unevenly distributed data, for which a fixed interval in a histogram aggregation produces either many nearly empty buckets or a few buckets holding almost every document.

Each shard buffers the first initial_buffer values it collects, sorts them, and divides them into a number of initial clusters equal to three-quarters of shard_size. It assigns each remaining value to the nearest cluster, unless the value lies more than twice the average distance between neighboring cluster centroids away from all of them and the shard holds fewer than shard_size clusters, in which case the shard starts a new cluster. The coordinating node collects the clusters from all shards and repeatedly merges the two clusters with the closest centroids until buckets remain.

Parameters

The variable_width_histogram aggregation takes the following parameters.

Example

The following example groups ecommerce order totals into five variable-width buckets:

GET /opensearch_dashboards_sample_data_ecommerce/_search
{
  "size": 0,
  "aggs": {
    "price_buckets": {
      "variable_width_histogram": {
        "field": "taxful_total_price",
        "buckets": 5
      }
    }
  }
}

Example response

{
  "took": 17,
  "timed_out": false,
  "terminated_early": true,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 4675,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "price_buckets": {
      "buckets": [
        {
          "min": 6.98828125,
          "key": 59.14907207464146,
          "max": 105.46875,
          "doc_count": 3835
        },
        {
          "min": 105.46875,
          "key": 139.096796875,
          "max": 229.5,
          "doc_count": 800
        },
        {
          "min": 229.5,
          "key": 247.11111111111111,
          "max": 304.0,
          "doc_count": 27
        },
        {
          "min": 304.0,
          "key": 318.4,
          "max": 308.0,
          "doc_count": 5
        },
        {
          "min": 308.0,
          "key": 563.25,
          "max": 2250.0,
          "doc_count": 8
        }
      ]
    }
  }
}

The buckets covering the crowded low end of the price range are a few tens of dollars wide, whereas the last bucket spans nearly $2,000 to cover the few highest orders.

Example: Nesting a subaggregation

Like other bucket aggregations, variable_width_histogram accepts subaggregations. The following example computes the average number of items ordered in each price bucket:

GET /opensearch_dashboards_sample_data_ecommerce/_search
{
  "size": 0,
  "aggs": {
    "price_buckets": {
      "variable_width_histogram": {
        "field": "taxful_total_price",
        "buckets": 3
      },
      "aggs": {
        "avg_quantity": {
          "avg": {
            "field": "total_quantity"
          }
        }
      }
    }
  }
}

The response contains the subaggregation result in each bucket:

{
  "aggregations": {
    "price_buckets": {
      "buckets": [
        {
          "min": 6.98828125,
          "key": 73.45130757286513,
          "max": 246.0,
          "doc_count": 4649,
          "avg_quantity": {
            "value": 2.1559475155947516
          }
        },
        {
          "min": 246.0,
          "key": 281.9166666666667,
          "max": 370.0,
          "doc_count": 24,
          "avg_quantity": {
            "value": 2.7083333333333335
          }
        },
        {
          "min": 393.0,
          "key": 1321.5,
          "max": 2250.0,
          "doc_count": 2,
          "avg_quantity": {
            "value": 1.5
          }
        }
      ]
    }
  }
}

Response body fields

The following table lists the response body fields.

Limitations

The variable_width_histogram aggregation has the following limitations:

• The aggregation cannot be nested inside a parent aggregation that collects more than one bucket. A terms, histogram, range, or filters parent returns the following error:

  [variable_width_histogram] cannot be nested inside an aggregation that collects more than a single bucket.
  

  Single-bucket parents, such as filter, global, and nested, are supported.

• The aggregation cannot run as a child of a nested aggregation when it has a subaggregation that requires document scores, such as top_hits. This combination returns an error.
• The keyed parameter is not supported, so buckets are always returned as an array.
• The min and max bounds are approximate. While merging clusters, OpenSearch can leave two clusters whose bounds overlap as separate buckets if their centroids are far apart. It then sets the boundary between the two buckets to the midpoint of the overlap, so a bound is not necessarily a value present in the data, and the lower bucket holds more values than its bounds indicate while the upper bucket holds fewer. Lowering a bucket’s max in this way can place it below the bucket’s own key, which is the mean of the values in the bucket. The fourth bucket in the first example response reports a key of 318.4 against a max of 308.0 for this reason. This merging step runs on a single-shard index as well as on a multi-shard one.
• Buckets are not contiguous. Each bucket’s bounds are the smallest and largest values it holds, so a gap in the data appears as a gap between adjacent buckets. In the subaggregation example response, one bucket ends at 370.0 and the next begins at 393.0.
• Bucket bounds are sensitive to outliers. A few extreme values stretch a bucket across a wide range of the data: the last bucket in the first example response spans 308.0 to 2250.0 to hold 8 orders.
• Bucket boundaries depend on the order in which each shard collects values, so they can change as segments merge or as documents are added, even when the value distribution stays the same.