Character N-Gram Tokenization for European Language Text Retrieval

By Paul McNamee, J. Mayfield. Information retrieval (Boston), 2004

388 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/94d2bfc1681ff3293bbc8f8ac22e36ab40760070

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

CompiLIG at SemEval-2017 Task 1: Cross-Language Plagiarism Detection Methods for Semantic Textual Similarity (2017, methodology): “It is based on Mcnamee and Mayﬁeld (2004) work used in information retrieval.”
Multitask Learning for Mental Health Conditions with Limited Social Media Data (2017, methodology): “We use the entire Twitter history of each user as input to the model, and split it into character 1-to-5-grams, which have been shown to capture more information than words for many Twitter text classification tasks (Mcnamee and Mayfield, 2004; Coppersmith et al., 2015a).”
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, background): “Previous work studied the pros and cons of different tokenization strategies, spacing from word- [14] to character-level [44] tokenization.”
An Empirical Analysis of NMT-Derived Interlingual Embeddings and Their Use in Parallel Sentence Identification (2017, methodology): “We borrow two well-known representations from crosslanguage information retrieval to account for syntactic features by means of cosine similarities: (i) character n-grams [39] with n = [2, 5] and (ii) pseudo-cognates.”
Lump at SemEval-2017 Task 1: Towards an Interlingua Semantic Similarity (2017, methodology): “This kind of representations is used on European languages with similar alphabets (McNamee and Mayﬁeld, 2004; Simard et al., 1992).”
Language Identification: The Long and the Short of the Matter (2010, background): “…in a number of contexts, the most immediate application being in multilingual text retrieval, where retrieval results are generally superior if the language of the query is known, and the search is restricted to only those documents predicted to be in that language (McNamee and Mayﬁeld, 2004).”