org.languagetool.rules.en

Class GoogleStyleWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.rules.en.GoogleStyleWordTokenizer

All Implemented Interfaces:

Tokenizer

public class GoogleStyleWordTokenizer
extends WordTokenizer

Tokenize sentences to tokens like Google does for its ngram index. Note: there doesn't seem to be official documentation about the way Google tokenizes there, so this is just an approximation.

Since:

3.2

Constructor Summary

Constructors

Constructor and Description
GoogleStyleWordTokenizer()

Method Summary

Modifier and Type	Method and Description
String	getTokenizingCharacters()
List<String>	tokenize(String text)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GoogleStyleWordTokenizer

public GoogleStyleWordTokenizer()

Method Detail

getTokenizingCharacters

public String getTokenizingCharacters()

Overrides:

getTokenizingCharacters in class WordTokenizer

Returns:

The string containing the characters used by the tokenizer to tokenize words.

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer