org.languagetool.tokenizers

Interface Tokenizer

All Known Subinterfaces:

CompoundWordTokenizer, SentenceTokenizer

All Known Implementing Classes:

ArabicWordTokenizer, BelarusianWordTokenizer, BretonWordTokenizer, CatalanWordTokenizer, ChineseSentenceTokenizer, ChineseWordTokenizer, DutchWordTokenizer, EnglishWordTokenizer, EsperantoWordTokenizer, FrenchWordTokenizer, GalicianWordTokenizer, GermanCompoundTokenizer, GermanWordTokenizer, GoogleStyleWordTokenizer, GreekWordTokenizer, JapaneseWordTokenizer, KhmerWordTokenizer, MalayalamWordTokenizer, PersianWordTokenizer, PolishWordTokenizer, PortugueseWordTokenizer, RomanianWordTokenizer, RussianWordTokenizer, SimpleSentenceTokenizer, SpanishWordTokenizer, SRXSentenceTokenizer, TagalogWordTokenizer, UkrainianWordTokenizer, WordTokenizer

public interface Tokenizer

Interface for classes that tokenize text into smaller units.

Author:

Daniel Naber

Method Summary

Modifier and Type	Method and Description
List<String>	tokenize(String text)

Method Detail

tokenize

List<String> tokenize(String text)