Kuromoji analyzer

The kuromoji analyzer provides morphological analysis for Japanese text using the Kuromoji library backed by the IPAdic dictionary. It segments Japanese sentences into meaningful tokens, removes common stop words and grammatical particles, and returns tokens in their dictionary base form.

To use the kuromoji analyzer or any Kuromoji components, you must first install the analysis-kuromoji plugin.

Installing the plugin

Install the plugin on all nodes and then restart the cluster:

bin/opensearch-plugin install analysis-kuromoji

For more information about installing plugins, see Installing plugins.

Kuromoji plugin components

The analysis-kuromoji plugin provides the following components that you can use independently or combine in custom analyzers.

Analyzers

Tokenizer

Character filter

Token filters

Analysis pipeline

The kuromoji analyzer applies the following components to input text, in order:

1. The cjk_width character filter converts full-width ASCII variants to their standard ASCII equivalents and half-width katakana variants to full-width katakana before tokenization.
2. The kuromoji_tokenizer in search mode segments text using the IPAdic dictionary, splitting long compound words into their sub-tokens.
3. The kuromoji_baseform token filter replaces inflected verb and adjective forms with their dictionary forms (for example, 食べた becomes 食べる).
4. The kuromoji_part_of_speech token filter removes grammatical particles (助詞), auxiliary verbs (助動詞), punctuation (記号), and other stop tags.
5. The ja_stop token filter removes common Japanese stop words.
6. The kuromoji_stemmer token filter removes trailing long vowel marks (ー) from katakana words of four or more characters (for example, コンピューター becomes コンピュータ).
7. The lowercase token filter converts any Latin characters to lowercase.

Parameters

The following table lists the parameters for the kuromoji analyzer.

For the full list of stop words in the built-in stop set, see stopwords.txt in the Lucene repository.

Tokenization modes

The tokenizer mode controls how compound and unknown words are segmented.

Example: Basic usage

The following example creates an index that uses the built-in kuromoji analyzer:

PUT /japanese-index
{
  "mappings": {
    "properties": {
      "content": {
        "type": "text",
        "analyzer": "kuromoji"
      }
    }
  }
}

Test the analyzer with a Japanese sentence meaning “I studied at the Tokyo library”:

POST /_analyze
{
  "analyzer": "kuromoji",
  "text": "東京の図書館で勉強しました"
}

The analyzer returns base-form tokens with particles and auxiliary verbs removed:

{
  "tokens": [
    {
      "token": "東京",
      "start_offset": 0,
      "end_offset": 2,
      "type": "word",
      "position": 0
    },
    {
      "token": "図書館",
      "start_offset": 3,
      "end_offset": 6,
      "type": "word",
      "position": 2
    },
    {
      "token": "勉強",
      "start_offset": 7,
      "end_offset": 9,
      "type": "word",
      "position": 4
    }
  ]
}

Example: Custom Kuromoji analyzer

The following example creates a custom kuromoji analyzer that uses user_dictionary_rules to register inline custom terms:

PUT /japanese-custom-index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_kuromoji": {
          "type": "kuromoji",
          "mode": "search",
          "user_dictionary_rules": [
            "東京スカイツリー,東京 スカイツリー,トウキョウ スカイツリー,カスタム名詞"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "my_kuromoji"
      }
    }
  }
}

Test the custom analyzer with a sentence containing the registered term (“I went to Tokyo Skytree”):

POST /japanese-custom-index/_analyze
{
  "analyzer": "my_kuromoji",
  "text": "東京スカイツリーに行きました"
}

The analyzer uses the custom dictionary rule to segment 東京スカイツリー into 東京 and スカイツリー. The default katakana stemmer (kuromoji_stemmer) removes the trailing long vowel mark from スカイツリー to produce スカイツリ. The kuromoji_part_of_speech filter removes the particle に and the auxiliary verbs まし and た, and kuromoji_baseform normalizes 行き to its base form 行く:

{
  "tokens": [
    {
      "token": "東京",
      "start_offset": 0,
      "end_offset": 2,
      "type": "word",
      "position": 0
    },
    {
      "token": "スカイツリ",
      "start_offset": 2,
      "end_offset": 8,
      "type": "word",
      "position": 1
    },
    {
      "token": "行く",
      "start_offset": 9,
      "end_offset": 11,
      "type": "word",
      "position": 3
    }
  ]
}

Kuromoji completion analyzer

The kuromoji_completion analyzer is designed for autocomplete use cases. It generates both the original Japanese tokens and their romaji (Latin script) reading variants, allowing users to search for Japanese content by typing in either Japanese characters or their romanized equivalents. In query mode, the analyzer also assembles partial input from an input method editor (IME) into a single token before romanizing it.

The following table lists the parameters for the kuromoji_completion analyzer.

Use query mode in the search analyzer so that partial IME input typed by a user is correctly assembled before romanization.

Example: Autocomplete using the Kuromoji completion analyzer

The following example configures an index with different completion analyzers used at index time and search time:

PUT /autocomplete-index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "kuromoji_index_analyzer": {
          "type": "kuromoji_completion",
          "mode": "index"
        },
        "kuromoji_search_analyzer": {
          "type": "kuromoji_completion",
          "mode": "query"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "suggest": {
        "type": "text",
        "analyzer": "kuromoji_index_analyzer",
        "search_analyzer": "kuromoji_search_analyzer"
      }
    }
  }
}

For more details about the completion token filter, see Kuromoji completion token filter.

Using the Kuromoji and ICU plugins together

For the most thorough Japanese text analysis, combine the analysis-kuromoji and analysis-icu plugins. ICU contributes Unicode normalization of full-width and half-width characters and comprehensive character folding that the Kuromoji components alone do not provide.

Both analysis-kuromoji and analysis-icu must be installed before creating an index that uses components from both plugins.

The following example creates a custom analyzer combining both plugins. The icu_normalizer character filter converts full-width characters to their ASCII equivalents before tokenization (for example, ３００ becomes 300), keeping numeric sequences as single tokens. The kuromoji_iteration_mark character filter expands Japanese iteration marks (for example, 時々 becomes 時時) so the tokenizer can look them up in the dictionary correctly:

PUT /japanese-icu-index
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "japanese_icu_analyzer": {
            "char_filter": [
              "icu_normalizer",
              "kuromoji_iteration_mark"
            ],
            "tokenizer": "kuromoji_tokenizer",
            "filter": [
              "kuromoji_baseform",
              "kuromoji_part_of_speech",
              "ja_stop",
              "kuromoji_stemmer",
              "lowercase"
            ]
          }
        }
      }
    }
  }
}

Test the combined analyzer with a sentence containing full-width digits and a long vowel mark (“The computer processes 300 documents”):

POST /japanese-icu-index/_analyze
{
  "analyzer": "japanese_icu_analyzer",
  "text": "コンピューターは３００件の文書を処理します"
}

The analyzer applies the following transformations:

• icu_normalizer converts the full-width digits ３００ to 300 before tokenization.
• kuromoji_stemmer removes the trailing long vowel mark, converting コンピューター to コンピュータ.
• kuromoji_part_of_speech removes the particles は, の, and を, along with the auxiliary verb ます.
• ja_stop removes し because it appears in the _japanese_ stop set.
• 300 is kept as a single token, and 文書 and 処理 are returned in base form.

The response appears as follows:

{
  "tokens": [
    {
      "token": "コンピュータ",
      "start_offset": 0,
      "end_offset": 7,
      "type": "word",
      "position": 0
    },
    {
      "token": "300",
      "start_offset": 8,
      "end_offset": 11,
      "type": "word",
      "position": 2
    },
    {
      "token": "件",
      "start_offset": 11,
      "end_offset": 12,
      "type": "word",
      "position": 3
    },
    {
      "token": "文書",
      "start_offset": 13,
      "end_offset": 15,
      "type": "word",
      "position": 5
    },
    {
      "token": "処理",
      "start_offset": 16,
      "end_offset": 18,
      "type": "word",
      "position": 7
    }
  ]
}

Related documentation

• Kuromoji tokenizer
• Kuromoji iteration mark character filter
• ICU analyzer
• CJK width token filter