Kuromoji part of speech token filter
The kuromoji_part_of_speech token filter removes tokens whose part-of-speech (POS) tag matches an entry in a configured list of stop tags. The Kuromoji tokenizer assigns each token an IPAdic POS tag. This filter reads that tag and discards tokens that serve a grammatical function (such as particles, auxiliary verbs, and punctuation) rather than a content function.
Installation
The kuromoji_part_of_speech token filter requires the analysis-kuromoji plugin. For installation instructions, see Kuromoji analyzer.
Parameters
The following table lists the parameters for the kuromoji_part_of_speech token filter.
| Parameter | Data type | Description |
|---|---|---|
stoptags | Array of strings | A list of IPAdic part-of-speech tags to remove. Tokens whose POS tag exactly matches an entry in this list are discarded. Default is the built-in Japanese stop tag set. |
For the full list of available stop tags, see stoptags.txt in the Lucene repository.
Example: Default filter
The following example creates an index with an analyzer that uses the default stop tag list:
PUT /kuromoji-pos-index
{
"settings": {
"analysis": {
"analyzer": {
"pos_filter_analyzer": {
"type": "custom",
"tokenizer": "kuromoji_tokenizer",
"filter": ["kuromoji_part_of_speech"]
}
}
}
}
}
Test the analyzer with a sentence meaning “I eat sushi at the Tokyo restaurant”:
POST /kuromoji-pos-index/_analyze
{
"analyzer": "pos_filter_analyzer",
"text": "東京のレストランで寿司を食べる"
}
The response shows the particles の (genitive), で (locative), and を (accusative) removed:
{
"tokens": [
{
"token": "東京",
"start_offset": 0,
"end_offset": 2,
"type": "word",
"position": 0
},
{
"token": "レストラン",
"start_offset": 3,
"end_offset": 8,
"type": "word",
"position": 2
},
{
"token": "寿司",
"start_offset": 9,
"end_offset": 11,
"type": "word",
"position": 4
},
{
"token": "食べる",
"start_offset": 12,
"end_offset": 15,
"type": "word",
"position": 6
}
]
}
Example: Custom stop tags
The following example creates a filter that removes only auxiliary verbs (助動詞) while keeping all other grammatical tokens:
PUT /kuromoji-custom-pos-index
{
"settings": {
"analysis": {
"filter": {
"auxiliary_verb_filter": {
"type": "kuromoji_part_of_speech",
"stoptags": ["助動詞"]
}
},
"analyzer": {
"auxiliary_filter_analyzer": {
"type": "custom",
"tokenizer": "kuromoji_tokenizer",
"filter": ["auxiliary_verb_filter"]
}
}
}
}
}