org.languagetool.tokenizers.fr

Class FrenchWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.fr.FrenchWordTokenizer

All Implemented Interfaces:

Tokenizer

public class FrenchWordTokenizer
extends WordTokenizer

Tokenizes a sentence into words. Punctuation and whitespace get its own token. Special treatment for hyphens and apostrophes in French.

Author:

Jaume Ortolà

Constructor Summary

Constructors

Constructor and Description
FrenchWordTokenizer()

Method Summary

Modifier and Type	Method and Description
List<String>	tokenize(String text)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, getTokenizingCharacters, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FrenchWordTokenizer

public FrenchWordTokenizer()

Method Detail

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - Text to tokenize

Returns:

List of tokens. Note: a special string

FR_APOS

is used to replace apostrophes, and

FR_HYPHEN

to replace hyphens.