Link Search Menu Expand Document Documentation Menu
Documentation

Top N queries

Monitoring the top N queries using query insights allows you to gain real-time visibility into the queries with the highest latency or resource consumption in a specified time period (for example, the last hour).

Configuring top N query monitoring

You can configure top N query monitoring by the following metric types:

  • latency
  • cpu
  • memory

Each metric has a set of corresponding settings:

For example, to enable top N query monitoring by CPU usage, set search.insights.top_queries.cpu.enabled to true. For more information about ways to specify dynamic settings, see Dynamic settings.

It’s important to exercise caution when enabling this feature because it can consume system resources.

Enabling top N query monitoring

When you install the query-insights plugin, top N query monitoring is enabled by default. To disable top N query monitoring, update the dynamic cluster settings for the desired metric types. For example, to disable monitoring of top N queries by latency, update the search.insights.top_queries.latency.enabled setting:

PUT _cluster/settings
{
  "persistent" : {
    "search.insights.top_queries.latency.enabled" : false
  }
}

Configuring the window size

To configure the monitoring window size, update the window_size setting for the desired metric type. The default window_size is 5m. For example, to collect the top N queries by latency in a 60-minute window, update the search.insights.top_queries.latency.window_size setting:

PUT _cluster/settings
{
  "persistent" : {
    "search.insights.top_queries.latency.window_size" : "60m"
  }
}

Configuring the value of N

To configure the value of N, update the top_n_size setting for the desired metric type. The default top_n_size is 10. For example, to collect the top 20 queries by latency, update the insights.top_queries.latency.top_n_size setting:

PUT _cluster/settings
{
  "persistent" : {
    "search.insights.top_queries.latency.top_n_size" : 20
  }
}

Monitoring current top N queries

You can use the Insights API endpoint to retrieve the top N queries for the current time window. This API returns top N latency results by default.

GET /_insights/top_queries

Query parameters

The following table lists the available query parameters. All query parameters are optional.

Parameter Data type Description
type String The metric type for which to retrieve top N query data. Results will be sorted in descending order based on this metric. Valid values are latency, cpu, and memory. Default is latency.
from String The start of the time range for fetching historical top N queries. For more information, see Monitoring historical top N queries.
to String The end of the time range for fetching historical top N queries. For more information, see Monitoring historical top N queries.
id String The ID of a specific top query record to retrieve.
verbose Boolean Indicates whether to return verbose output. Default is true.

Example response

Response 
{
  "top_queries" : [
    {
      "timestamp" : 1745021834451,
      "id" : "36506bd2-7bca-4a0a-a6b8-f3e7db2b0745",
      "group_by" : "NONE",
      "indices" : [
        "my-index-0"
      ],
      "source" : {
        "size" : 20,
        "query" : {
          "bool" : {
            "must" : [
              {
                "match_phrase" : {
                  "message" : {
                    "query" : "document",
                    "slop" : 0,
                    "zero_terms_query" : "NONE",
                    "boost" : 1.0
                  }
                }
              },
              {
                "match" : {
                  "user.id" : {
                    "query" : "userId",
                    "operator" : "OR",
                    "prefix_length" : 0,
                    "max_expansions" : 50,
                    "fuzzy_transpositions" : true,
                    "lenient" : false,
                    "zero_terms_query" : "NONE",
                    "auto_generate_synonyms_phrase_query" : true,
                    "boost" : 1.0
                  }
                }
              }
            ],
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        }
      },
      "task_resource_usages" : [
        {
          "action" : "indices:data/read/search[phase/query]",
          "taskId" : 28,
          "parentTaskId" : 27,
          "nodeId" : "BBgWzu8QR0qDkR0G45aw8w",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 22664000,
            "memory_in_bytes" : 6604536
          }
        },
        {
          "action" : "indices:data/read/search",
          "taskId" : 27,
          "parentTaskId" : -1,
          "nodeId" : "BBgWzu8QR0qDkR0G45aw8w",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 119000,
            "memory_in_bytes" : 3920
          }
        }
      ],
      "node_id" : "BBgWzu8QR0qDkR0G45aw8w",
      "phase_latency_map" : {
        "expand" : 0,
        "query" : 23,
        "fetch" : 0
      },
      "labels" : {
        "X-Opaque-Id" : "query-label-1"
      },
      "search_type" : "query_then_fetch",
      "total_shards" : 1,
      "measurements" : {
        "memory" : {
          "number" : 6608456,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "latency" : {
          "number" : 24,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "cpu" : {
          "number" : 22783000,
          "count" : 1,
          "aggregationType" : "NONE"
        }
      }
    },
    {
      "timestamp" : 1745021826937,
      "id" : "86e161d0-e982-48c2-b8da-e3a3763f2e36",
      "group_by" : "NONE",
      "indices" : [
        "my-index-*"
      ],
      "source" : {
        "size" : 20,
        "query" : {
          "term" : {
            "user.id" : {
              "value" : "userId",
              "boost" : 1.0
            }
          }
        }
      },
      "task_resource_usages" : [
        {
          "action" : "indices:data/read/search[phase/query]",
          "taskId" : 26,
          "parentTaskId" : 25,
          "nodeId" : "BBgWzu8QR0qDkR0G45aw8w",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 11020000,
            "memory_in_bytes" : 4292272
          }
        },
        {
          "action" : "indices:data/read/search",
          "taskId" : 25,
          "parentTaskId" : -1,
          "nodeId" : "BBgWzu8QR0qDkR0G45aw8w",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 1032000,
            "memory_in_bytes" : 115816
          }
        }
      ],
      "node_id" : "BBgWzu8QR0qDkR0G45aw8w",
      "phase_latency_map" : {
        "expand" : 0,
        "query" : 15,
        "fetch" : 1
      },
      "labels" : { },
      "search_type" : "query_then_fetch",
      "total_shards" : 1,
      "measurements" : {
        "memory" : {
          "number" : 4408088,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "latency" : {
          "number" : 23,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "cpu" : {
          "number" : 12052000,
          "count" : 1,
          "aggregationType" : "NONE"
        }
      }
    }
  ]
}

If your query returns no results, ensure that top N query monitoring is enabled for the target metric type and that search requests were made within the current time window.

Monitoring historical top N queries

To query historical top N results, specify a time range with the from and to parameters in ISO 8601 format: YYYY-MM-DD'T'HH:mm:ss.SSSZ. For example, to retrieve the top N queries from August 25, 2024, at 15:00 UTC to August 30, 2024, at 17:00 UTC, send the following request:

GET /_insights/top_queries?from=2024-08-25T15:00:00.000Z&to=2024-08-30T17:00:00.000Z

To view historical query data, the exporter type must be set to local_index. For more information, see Configuring a local index exporter.

Exporting top N query data

You can configure your desired exporter to export top N query data to different sinks, allowing for better monitoring and analysis of your OpenSearch queries. Currently, the following exporters are supported:

Configuring a debug exporter

To use the debug exporter, set the exporter type to debug:

PUT _cluster/settings
{
  "persistent" : {
      "search.insights.top_queries.exporter.type" : "debug"
  }
}

Configuring a local index exporter

The default exporter is local_index. A local index exporter allows you to save top N query data to indexes that are automatically created in your OpenSearch domain. Query Insights creates these indexes following the naming pattern top_queries-YYYY.MM.dd-hashcode, where hashcode is a 5-digit number generated based on the current UTC date. A new index is created daily. For historical top N lookups using the Top Queries API or the Query Insights dashboard, you must enable the local index exporter.

To use the local index exporter, set the exporter type to local_index:

PUT _cluster/settings
{
  "persistent" : {
    "search.insights.top_queries.exporter.type" : "local_index"
  }
}

Use the delete_after_days setting (integer) to specify the number of days after which local indexes are automatically deleted. Query Insights runs a job once per day at 00:05 UTC to delete top N local indexes older than the specified number of days. The default value for delete_after_days is 7, with valid values ranging from 1 to 180.

For example, to delete local indexes older than 10 days, send the following request:

PUT _cluster/settings
{
  "persistent" : {
    "search.insights.top_queries.exporter.delete_after_days" : "10"
  }
}

350 characters left

Have a question? .

Want to contribute? or .