ml

The ml command applies machine learning (ML) algorithms from the ML Commons plugin to the search results returned by a PPL command. It supports various ML operations, including anomaly detection and clustering. The command can perform train, predict, or combined train-and-predict operations, depending on the algorithm and specified action.

To use the ml command, plugins.calcite.enabled must be set to false.

The ml command supports the following algorithms:

• Random Cut Forest (RCF) for anomaly detection, with support for both time-series and non-time-series data

• K-means for clustering data points into groups

Syntax

The ml command supports different syntax options, depending on the algorithm.

Anomaly detection for time-series data

Use this syntax to detect anomalies in time-series data. This method uses the RCF algorithm optimized for sequential data patterns:

ml action='train' algorithm='rcf' <number_of_trees> <shingle_size> <sample_size> <output_after> <time_decay> <anomaly_rate> <time_field> <date_format> <time_zone>

Parameters

The fixed-in-time RCF algorithm supports the following parameters.

Anomaly detection for non-time-series data

Use this syntax to detect anomalies in data where the order doesn’t matter. This method uses the RCF algorithm optimized for independent data points:

ml action='train' algorithm='rcf' <number_of_trees> <sample_size> <output_after> <training_data_size> <anomaly_score_threshold>

Parameters

The batch RCF algorithm supports the following parameters.

K-means clustering

Use this syntax to group data points into clusters based on similarity:

ml action='train' algorithm='kmeans' <centroids> <iterations> <distance_type>

Parameters

The k-means clustering algorithm supports the following parameters.

Example 1: Time-series anomaly detection

This example trains an RCF model and uses it to detect anomalies in time-series ridership data:

source=nyc_taxi
| fields value, timestamp
| ml action='train' algorithm='rcf' time_field='timestamp'
| where value=10844.0

The query returns the following results:

Example 2: Time-series anomaly detection by category

This example trains an RCF model and uses it to detect anomalies in time-series ridership data across multiple category values:

source=nyc_taxi
| fields category, value, timestamp
| ml action='train' algorithm='rcf' time_field='timestamp' category_field='category'
| where value=10844.0 or value=6526.0

The query returns the following results:

Example 3: Non-time-series anomaly detection

This example trains an RCF model and uses it to detect anomalies in non-time-series ridership data:

source=nyc_taxi
| fields value
| ml action='train' algorithm='rcf'
| where value=10844.0

The query returns the following results:

Example 4: Non-time-series anomaly detection by category

This example trains an RCF model and uses it to detect anomalies in non-time-series ridership data across multiple category values:

source=nyc_taxi
| fields category, value
| ml action='train' algorithm='rcf' category_field='category'
| where value=10844.0 or value=6526.0

The query returns the following results:

Example 5: K-means clustering of the Iris dataset

This example uses k-means clustering to classify three Iris species (Iris setosa, Iris virginica, and Iris versicolor) based on the combination of four features measured from each sample (the lengths and widths of sepals and petals):

source=iris_data
| fields sepal_length_in_cm, sepal_width_in_cm, petal_length_in_cm, petal_width_in_cm
| ml action='train' algorithm='kmeans' centroids=3

The query returns the following results: