You're viewing version 3.8 of the OpenSearch documentation. This version is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.
Kuromoji part of speech token filter
The kuromoji_part_of_speech token filter removes tokens whose part-of-speech (POS) tag matches an entry in a configured list of stop tags. The Kuromoji tokenizer assigns each token an IPAdic POS tag. This filter reads that tag and discards tokens that serve a grammatical function (such as particles, auxiliary verbs, and punctuation) rather than a content function.
Installation
The kuromoji_part_of_speech token filter requires the analysis-kuromoji plugin. For installation instructions, see Kuromoji analyzer.
Parameters
The following table lists the parameters for the kuromoji_part_of_speech token filter.
| Parameter | Data type | Description |
|---|---|---|
stoptags | Array of strings | A list of IPAdic part-of-speech tags to remove. Tokens whose POS tag exactly matches an entry in this list are discarded. Default is the built-in Japanese stop tag set. |
For the full list of available stop tags, see stoptags.txt in the Lucene repository.
Example: Default filter
The following example creates an index with an analyzer that uses the default stop tag list:
PUT /kuromoji-pos-index
{
"settings": {
"analysis": {
"analyzer": {
"pos_filter_analyzer": {
"type": "custom",
"tokenizer": "kuromoji_tokenizer",
"filter": ["kuromoji_part_of_speech"]
}
}
}
}
}
Test the analyzer with a sentence meaning “I eat sushi at the Tokyo restaurant”:
POST /kuromoji-pos-index/_analyze
{
"analyzer": "pos_filter_analyzer",
"text": "東京のレストランで寿司を食べる"
}
The response shows the particles の (genitive), で (locative), and を (accusative) removed:
{
"tokens": [
{
"token": "東京",
"start_offset": 0,
"end_offset": 2,
"type": "word",
"position": 0
},
{
"token": "レストラン",
"start_offset": 3,
"end_offset": 8,
"type": "word",
"position": 2
},
{
"token": "寿司",
"start_offset": 9,
"end_offset": 11,
"type": "word",
"position": 4
},
{
"token": "食べる",
"start_offset": 12,
"end_offset": 15,
"type": "word",
"position": 6
}
]
}
Example: Custom stop tags
The following example creates a filter that removes only auxiliary verbs (助動詞) while keeping all other grammatical tokens:
PUT /kuromoji-custom-pos-index
{
"settings": {
"analysis": {
"filter": {
"auxiliary_verb_filter": {
"type": "kuromoji_part_of_speech",
"stoptags": ["助動詞"]
}
},
"analyzer": {
"auxiliary_filter_analyzer": {
"type": "custom",
"tokenizer": "kuromoji_tokenizer",
"filter": ["auxiliary_verb_filter"]
}
}
}
}
}