NLTK: The Natural Language Toolkit

By Edward Loper, Steven Bird. 2002

1,461 citations recorded by the source corpus.

The Natural Language Toolkit is a suite of program modules, data sets and tutorials supporting research and teaching in computational linguistics and natural language processing. NLTK is written in Python and distributed under the GPL open source license. Over the past year the toolkit has been rewritten, simplifying many linguistic data structures and taking advantage of recent enhancements in the Python language. This paper reports on the simplified toolkit and explains how it is used in teaching NLP.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “Two core libraries are NLTK (Loper and Bird, 2002) and Stanford CoreNLP (Manning et al.”
Towards a Better Metric for Evaluating Question Generation Systems (2018, methodology): “We then created noisy versions of these questions using one of the following four methods: Dropping function words: We refer to the list of English function words as deﬁned in NLTK (Loper and Bird, 2002) and drop all such words from the question.”
FEVER: a Large-scale Dataset for Fact Extraction and VERification (2018, methodology): “, 2017) to select sentences using bigram TF-IDF with binning and compared this to a simple unigram TF-IDF implementation using NLTK (Loper and Bird, 2002).”
Does Gender Matter? Towards Fairness in Dialogue Systems (2019, methodology): “When evaluating with the attribute words, we lemmatize the words in the responses through WordNet lemmatizer in NLTK toolkit (Bird, 2006) before matching them with the attribute words.”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “Two core libraries are NLTK (Loper and Bird, 2002) and Stanford CoreNLP (Manning et al.”
Hypothesis Only Baselines in Natural Language Inference (2018, methodology): “We preprocess each recast dataset using the NLTK tokenizer (Loper and Bird, 2002).”
Self-Training for End-to-End Speech Recognition (2019, methodology): “We apply sentence segmentation using the NLTK toolkit [15] and normalize the text by lower-casing, removing punctuation except for the apostrophe, and replacing hyphens with spaces.”
NLTK: The Natural Language Toolkit (2006, background): “NLTK, the Natural Language Toolkit, is a suite of Python modules providing many NLP data types, processing tasks, corpus samples and readers, together with animated algorithms, tutorials, and problem sets (Loper and Bird, 2002).”
Protection Against Reconstruction and Its Applications in Private Federated Learning (2018, methodology): “We refit this pretrained LSTM on a corpus of all user comments on the website Reddit from November 2017 [10], again using the NLTK tokenization procedure [51].”
DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension (2019, background): “3We use the list of stop words from NLTK (Bird and Loper, 2004).”
BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization (2019, methodology): “To identify these entities, we extract non-recursive noun phrases (regex NP → ADJ∗[NN]+) using NLTK (Loper and Bird, 2002).”
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets (2019, methodology): “σ2 and σ3: we use a set of stopwords from NLTK (Loper and Bird 2002) as function words.”
Visual Question Generation as Dual Task of Visual Question Answering (2017, methodology): “ration problem. Therefore, we use top-1 accuracy (Acc@1) and top-5 accuracy (Acc@5) to measure the quality of the predicted answers. Sentence-level BLEU score (Papineni et al. 2002) provided by NLTK (Bird 2006) is employed to evaluate the generated questions (with Method 4 smooth function). Componen”
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (2019, methodology): “For the generic datasets, we use NLTK (Loper and Bird, 2002).”
Reading Tea Leaves: How Humans Interpret Topic Models (2009, background): “Stop words [ 20 ] and terms occurring in fewer than five documents are also removed.”
Transformers: State-of-the-Art Natural Language Processing (2020, neutral): “Two core libraries are NLTK (Loper and Bird, 2002) and Stanford CoreNLP (Manning et al., 2014), which collect a variety of different approaches to NLP in a single package.”
BLEU is Not Suitable for the Evaluation of Text Simplification (2018, methodology): “Sentence-level BLEU scores are computed using NLTK (Loper and Bird, 2002).”