Anatomy of a workload
A workload is a specification of one or more benchmarking scenarios. All workloads contain the following files and directories:
workload.json: Contains all of the workload settings.index.json: Contains the document mappings and parameters as well as index settings.files.txt: Contains the data corpora file names._test-procedures: Most workloads contain only one default test procedure, which is configured indefault.json._operations: Contains all of the operations used in test procedures.workload.py: Adds more dynamic functionality to the test.
workload.json
A workload.json file usually includes the following elements:
indices: Defines the relevant indexes and index templates used for the workload. To create an index, specify itsname. To add definitions to your index, use thebodyoption and point it to the JSON file containing the index definitions.corpora: Defines all document corpora used for the workload, including the source files that contain the documents and the number of documents in each file.operations: Optional. Lists the OpenSearch API operations performed by the workload and how they are parameterized. For example, you can list an operation namedcreate-indexthat creates an index in the benchmark cluster to which OpenSearch Benchmark can write documents.schedule: Defines operations and the order in which the operations run inline. Alternatively, you can useoperationsto group operations and thetest_proceduresparameter to specify the order of operations.
The following example workload shows these elements working together. You can run this workload in your own benchmark configuration:
{
"description": "Tutorial benchmark for OpenSearch Benchmark",
"indices": [
{
"name": "movies",
"body": "index.json"
}
],
"corpora": [
{
"name": "movies",
"documents": [
{
"source-file": "movies-documents.json",
"document-count": 11658903, # Fetch document count from command line
"uncompressed-bytes": 1544799789 # Fetch uncompressed bytes from command line
}
]
}
],
"schedule": [
{
"operation": {
"operation-type": "create-index"
}
},
{
"operation": {
"operation-type": "cluster-health",
"request-params": {
"wait_for_status": "green"
},
"retry-until-success": true
}
},
{
"operation": {
"operation-type": "bulk",
"bulk-size": 5000
},
"warmup-time-period": 120,
"clients": 8
},
{
"operation": {
"name": "query-match-all",
"operation-type": "search",
"body": {
"query": {
"match_all": {}
}
}
},
"iterations": 1000,
"target-throughput": 100
}
]
}
index.json
The index.json file defines the data mappings, indexing parameters, and index settings for workload documents during create-index operations.
When OpenSearch Benchmark creates an index for the workload, it uses the index settings and mappings template in the index.json file. Mappings in the index.json file are based on the mappings of a single document from the workload’s corpus, which is stored in the files.txt file. The following is an example of the index.json file for the nyc_taxis workload. You can customize the fields, such as number_of_shards, number_of_replicas, query_cache_enabled, and requests_cache_enabled.
{
"settings": {
"index.number_of_shards": {{number_of_shards | default(1)}},
"index.number_of_replicas": {{number_of_replicas | default(0)}},
"index.queries.cache.enabled": {{query_cache_enabled | default(false) | tojson}},
"index.requests.cache.enable": {{requests_cache_enabled | default(false) | tojson}}
},
"mappings": {
"_source": {
"enabled": {{ source_enabled | default(true) | tojson }}
},
"properties": {
"surcharge": {
"scaling_factor": 100,
"type": "scaled_float"
},
"dropoff_datetime": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"trip_type": {
"type": "keyword"
},
"mta_tax": {
"scaling_factor": 100,
"type": "scaled_float"
},
"rate_code_id": {
"type": "keyword"
},
"passenger_count": {
"type": "integer"
},
"pickup_datetime": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"tolls_amount": {
"scaling_factor": 100,
"type": "scaled_float"
},
"tip_amount": {
"type": "half_float"
},
"payment_type": {
"type": "keyword"
},
"extra": {
"scaling_factor": 100,
"type": "scaled_float"
},
"vendor_id": {
"type": "keyword"
},
"store_and_fwd_flag": {
"type": "keyword"
},
"improvement_surcharge": {
"scaling_factor": 100,
"type": "scaled_float"
},
"fare_amount": {
"scaling_factor": 100,
"type": "scaled_float"
},
"ehail_fee": {
"scaling_factor": 100,
"type": "scaled_float"
},
"cab_color": {
"type": "keyword"
},
"dropoff_location": {
"type": "geo_point"
},
"vendor_name": {
"type": "text"
},
"total_amount": {
"scaling_factor": 100,
"type": "scaled_float"
},
"trip_distance": {%- if trip_distance_mapping is defined %} {{ trip_distance_mapping | tojson }} {%- else %} {
"scaling_factor": 100,
"type": "scaled_float"
}{%- endif %},
"pickup_location": {
"type": "geo_point"
}
},
"dynamic": "strict"
}
}
files.txt
The files.txt file lists the files that store the workload data, which are typically stored in a zipped JSON file.
_operations and _test-procedures
To make the workload more human-readable, _operations and _test-procedures are separated into two directories.
The _operations directory contains a default.json file that lists all of the supported operations that the test procedure can use. Some workloads, such as nyc_taxis, contain an additional .json file that lists feature-specific operations, such as snapshot operations. The following JSON example shows a list of operations from the nyc_taxis workload:
{
"name": "index",
"operation-type": "bulk",
"bulk-size": {{bulk_size | default(10000)}},
"ingest-percentage": {{ingest_percentage | default(100)}}
},
{
"name": "update",
"operation-type": "bulk",
"bulk-size": {{bulk_size | default(10000)}},
"ingest-percentage": {{ingest_percentage | default(100)}},
"conflicts": "{{conflicts | default('random')}}",
"on-conflict": "{{on_conflict | default('update')}}",
"conflict-probability": {{conflict_probability | default(25)}},
"recency": {{recency | default(0)}}
},
{
"name": "wait-until-merges-finish",
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
},
{
"name": "default",
"operation-type": "search",
"body": {
"query": {
"match_all": {}
}
}
},
{
"name": "range",
"operation-type": "search",
"body": {
"query": {
"range": {
"total_amount": {
"gte": 5,
"lt": 15
}
}
}
}
},
{
"name": "distance_amount_agg",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"bool": {
"filter": {
"range": {
"trip_distance": {
"lt": 50,
"gte": 0
}
}
}
}
},
"aggs": {
"distance_histo": {
"histogram": {
"field": "trip_distance",
"interval": 1
},
"aggs": {
"total_amount_stats": {
"stats": {
"field": "total_amount"
}
}
}
}
}
}
},
{
"name": "autohisto_agg",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "01/01/2015",
"lte": "21/01/2015",
"format": "dd/MM/yyyy"
}
}
},
"aggs": {
"dropoffs_over_time": {
"auto_date_histogram": {
"field": "dropoff_datetime",
"buckets": 20
}
}
}
}
},
{
"name": "date_histogram_agg",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "01/01/2015",
"lte": "21/01/2015",
"format": "dd/MM/yyyy"
}
}
},
"aggs": {
"dropoffs_over_time": {
"date_histogram": {
"field": "dropoff_datetime",
"calendar_interval": "day"
}
}
}
}
},
{
"name": "date_histogram_calendar_interval",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "2015-01-01 00:00:00",
"lt": "2016-01-01 00:00:00"
}
}
},
"aggs": {
"dropoffs_over_time": {
"date_histogram": {
"field": "dropoff_datetime",
"calendar_interval": "month"
}
}
}
}
},
{
"name": "date_histogram_calendar_interval_with_tz",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "2015-01-01 00:00:00",
"lt": "2016-01-01 00:00:00"
}
}
},
"aggs": {
"dropoffs_over_time": {
"date_histogram": {
"field": "dropoff_datetime",
"calendar_interval": "month",
"time_zone": "America/New_York"
}
}
}
}
},
{
"name": "date_histogram_fixed_interval",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "2015-01-01 00:00:00",
"lt": "2016-01-01 00:00:00"
}
}
},
"aggs": {
"dropoffs_over_time": {
"date_histogram": {
"field": "dropoff_datetime",
"fixed_interval": "60d"
}
}
}
}
},
{
"name": "date_histogram_fixed_interval_with_tz",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "2015-01-01 00:00:00",
"lt": "2016-01-01 00:00:00"
}
}
},
"aggs": {
"dropoffs_over_time": {
"date_histogram": {
"field": "dropoff_datetime",
"fixed_interval": "60d",
"time_zone": "America/New_York"
}
}
}
}
},
{
"name": "date_histogram_fixed_interval_with_metrics",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "2015-01-01 00:00:00",
"lt": "2016-01-01 00:00:00"
}
}
},
"aggs": {
"dropoffs_over_time": {
"date_histogram": {
"field": "dropoff_datetime",
"fixed_interval": "60d"
},
"aggs": {
"total_amount": { "stats": { "field": "total_amount" } },
"tip_amount": { "stats": { "field": "tip_amount" } },
"trip_distance": { "stats": { "field": "trip_distance" } }
}
}
}
}
},
{
"name": "auto_date_histogram",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "2015-01-01 00:00:00",
"lt": "2016-01-01 00:00:00"
}
}
},
"aggs": {
"dropoffs_over_time": {
"auto_date_histogram": {
"field": "dropoff_datetime",
"buckets": "12"
}
}
}
}
},
{
"name": "auto_date_histogram_with_tz",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "2015-01-01 00:00:00",
"lt": "2016-01-01 00:00:00"
}
}
},
"aggs": {
"dropoffs_over_time": {
"auto_date_histogram": {
"field": "dropoff_datetime",
"buckets": "13",
"time_zone": "America/New_York"
}
}
}
}
},
{
"name": "auto_date_histogram_with_metrics",
"operation-type": "search",
"body": {
"size": 0,
"query": {
"range": {
"dropoff_datetime": {
"gte": "2015-01-01 00:00:00",
"lt": "2016-01-01 00:00:00"
}
}
},
"aggs": {
"dropoffs_over_time": {
"auto_date_histogram": {
"field": "dropoff_datetime",
"buckets": "12"
},
"aggs": {
"total_amount": { "stats": { "field": "total_amount" } },
"tip_amount": { "stats": { "field": "tip_amount" } },
"trip_distance": { "stats": { "field": "trip_distance" } }
}
}
}
}
},
{
"name": "desc_sort_tip_amount",
"operation-type": "search",
"index": "nyc_taxis",
"body": {
"query": {
"match_all": {}
},
"sort" : [
{"tip_amount" : "desc"}
]
}
},
{
"name": "asc_sort_tip_amount",
"operation-type": "search",
"index": "nyc_taxis",
"body": {
"query": {
"match_all": {}
},
"sort" : [
{"tip_amount" : "asc"}
]
}
}
The _test-procedures directory contains a default.json file that sets the order of operations performed by the workload. Similar to the _operations directory, the _test-procedures directory can also contain feature-specific test procedures, such as searchable_snapshots.json for nyc_taxis. The following examples show the searchable snapshots test procedures for nyc_taxis:
{
"name": "searchable-snapshot",
"description": "Measuring performance for Searchable Snapshot feature. Based on the default test procedure 'append-no-conflicts'.",
"schedule": [
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {%- if index_settings is defined %} {{ index_settings | tojson }} {%- else %}{
"index.codec": "best_compression",
"index.refresh_interval": "30s",
"index.translog.flush_threshold_size": "4g"
}{%- endif %}
}
},
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "nyc_taxis",
"request-params": {
"wait_for_status": "{{ cluster_health | default('green') }}",
"wait_for_no_relocating_shards": "true"
},
"retry-until-success": true
}
},
{
"operation": "index",
"warmup-time-period": 240,
"clients": {{ bulk_indexing_clients | default(8) }},
"ignore-response-error-level": "{{ error_level | default('non-fatal') }}"
},
{
"name": "refresh-after-index",
"operation": "refresh"
},
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200
{%- if force_merge_max_num_segments is defined %},
"max-num-segments": {{ force_merge_max_num_segments | tojson }}
{%- endif %}
}
},
{
"name": "refresh-after-force-merge",
"operation": "refresh"
},
{
"operation": "wait-until-merges-finish"
},
{
"operation": "create-snapshot-repository"
},
{
"operation": "delete-snapshot"
},
{
"operation": "create-snapshot"
},
{
"operation": "wait-for-snapshot-creation"
},
{
"operation": {
"name": "delete-local-index",
"operation-type": "delete-index"
}
},
{
"operation": "restore-snapshot"
},
{
"operation": "default",
"warmup-iterations": 50,
"iterations": 100
{%- if not target_throughput %}
,"target-throughput": 3
{%- elif target_throughput is string and target_throughput.lower() == 'none' %}
{%- else %}
,"target-throughput": {{ target_throughput | tojson }}
{%- endif %}
{%-if search_clients is defined and search_clients %}
,"clients": {{ search_clients | tojson}}
{%- endif %}
},
{
"operation": "range",
"warmup-iterations": 50,
"iterations": 100
{%- if not target_throughput %}
,"target-throughput": 0.7
{%- elif target_throughput is string and target_throughput.lower() == 'none' %}
{%- else %}
,"target-throughput": {{ target_throughput | tojson }}
{%- endif %}
{%-if search_clients is defined and search_clients %}
,"clients": {{ search_clients | tojson}}
{%- endif %}
},
{
"operation": "distance_amount_agg",
"warmup-iterations": 50,
"iterations": 50
{%- if not target_throughput %}
,"target-throughput": 2
{%- elif target_throughput is string and target_throughput.lower() == 'none' %}
{%- else %}
,"target-throughput": {{ target_throughput | tojson }}
{%- endif %}
{%-if search_clients is defined and search_clients %}
,"clients": {{ search_clients | tojson}}
{%- endif %}
},
{
"operation": "autohisto_agg",
"warmup-iterations": 50,
"iterations": 100
{%- if not target_throughput %}
,"target-throughput": 1.5
{%- elif target_throughput is string and target_throughput.lower() == 'none' %}
{%- else %}
,"target-throughput": {{ target_throughput | tojson }}
{%- endif %}
{%-if search_clients is defined and search_clients %}
,"clients": {{ search_clients | tojson}}
{%- endif %}
},
{
"operation": "date_histogram_agg",
"warmup-iterations": 50,
"iterations": 100
{%- if not target_throughput %}
,"target-throughput": 1.5
{%- elif target_throughput is string and target_throughput.lower() == 'none' %}
{%- else %}
,"target-throughput": {{ target_throughput | tojson }}
{%- endif %}
{%-if search_clients is defined and search_clients %}
,"clients": {{ search_clients | tojson}}
{%- endif %}
}
]
}
Workload examples
The following examples show complete workload.json files that you can adapt for your own workloads.
Running unthrottled
In the following example, OpenSearch Benchmark runs an unthrottled bulk index operation for 1 hour against the movies index:
{
"description": "Tutorial benchmark for OpenSearch Benchmark",
"indices": [
{
"name": "movies",
"body": "index.json"
}
],
"corpora": [
{
"name": "movies",
"documents": [
{
"source-file": "movies-documents.json",
"document-count": 11658903, # Fetch document count from command line
"uncompressed-bytes": 1544799789 # Fetch uncompressed bytes from command line
}
]
}
],
"schedule": [
{
"operation": "bulk",
"warmup-time-period": 120,
"time-period": 3600,
"clients": 8
}
]
}
Workload with a single task
The following workload runs a benchmark with a single task: a match_all query. Because no clients are indicated, only one client is used. According to the schedule, the workload runs the match_all query at 10 operations per second with 1 client, uses 100 iterations to warm up, and uses the next 100 iterations to measure the benchmark:
{
"description": "Tutorial benchmark for OpenSearch Benchmark",
"indices": [
{
"name": "movies",
"body": "index.json"
}
],
"corpora": [
{
"name": "movies",
"documents": [
{
"source-file": "movies-documents.json",
"document-count": 11658903, # Fetch document count from command line
"uncompressed-bytes": 1544799789 # Fetch uncompressed bytes from command line
}
]
}
],
"schedule": [
{
"operation": {
"operation-type": "search",
"index": "_all",
"body": {
"query": {
"match_all": {}
}
}
},
"warmup-iterations": 100,
"iterations": 100,
"target-throughput": 10
}
]
}
Next steps
- For the data, cluster requirements, and query types of each prepackaged workload, see Workload types.
- For information about creating your own workload, see Creating custom workloads.