NLTK: The Natural Language Toolkit

By Steven Bird. 2006

5,221 citations recorded by the source corpus.

The Natural Language Toolkit is a suite of program modules, data sets and tutorials supporting research and teaching in computational linguistics and natural language processing. NLTK is written in Python and distributed under the GPL open source license. Over the past year the toolkit has been rewritten, simplifying many linguistic data structures and taking advantage of recent enhancements in the Python language. This paper reports on the simpli-ﬁed toolkit and explains how it is used in teaching NLP.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Natural Language Processing

Citations in the corpus

Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, methodology): “After sentence and word tokenization (with NLTK; Bird, 2006) and lower-casing, we identify all paragraphs longer than 8 sentences and extract a NEXT example from each, as well as pairs of sentences for the ORDER and CONJUNCTION tasks.”
Language Models are Few-Shot Learners (2020, methodology): “s very", "He would be described as", "She would be described as"9. We looked at the adjectives and adverbs in the top 100 most favored words using an off-the-shelf POS tagger [LB02]. We found females were more often described using appearance oriented words such as ”beautifu”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “An off-the-shelf POS tagger (Bird & Loper, 2004) is used to remove stop words and select only descriptive words (i.e. adjectives and adverbs).”
Improving Question Answering with External Knowledge (2019, methodology): “We use the stop word list from NLTK (Bird and Loper, 2004).”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “Our analysis of all three co-occurrence dimensions included an off-the-shelf POS tagger (Bird & Loper, 2004), which we used to remove stop words and select only descriptive words (i.e., adjectives and adverbs).”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, methodology): “Therefore, in these domains we use the named entity analyzer from NLTK [22] to filter out sentences that do not contain a person’s name.”
FEVER: a Large-scale Dataset for Fact Extraction and VERification (2018, methodology): “, 2017) to select sentences and a simple unigram implementation using NLTK (Loper and Bird, 2002) to select l = 5 sentences from the k = 5 highest scoring documents (see Table 3).”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, background): “We lowercase, lemmatize (Bird 2006), and strip excess whitespace from all relationships.”
Does Gender Matter? Towards Fairness in Dialogue Systems (2019, methodology): “When evaluating with the attribute words, we lemmatize the words in the responses through WordNet lemmatizer in NLTK toolkit [27] before matching them with the attribute words.”
A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers (2020, methodology): “We first use Stanford CoreNLP (Manning et al., 2014) to tokenize and tag POSs, and then use NLTK (Bird et al., 2004) to lemmatize each token.”
Hypothesis Only Baselines in Natural Language Inference (2018, methodology): “We preprocess each recast dataset using the NLTK tokenizer (Loper and Bird, 2002).”
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks (2016, methodology): “We use NLTK (Bird, 2006) for tok-enization, and constrain sentence lengths to be between 5 and 70 words.”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, methodology): “For tf-idf, we pre-process all the sentences with WordNet’s lemmatizer [5] and remove stop words.”
Self-Training for End-to-End Speech Recognition (2019, methodology): “We apply sentence segmentation using the NLTK toolkit [15] and normalize the text by lower-casing, removing punctuation except for the apos-trophe, and replacing hyphens with spaces.”
Exploring Models and Data for Image Question Answering (2015, methodology): “In this work we consider the following two simple constraints: (1) Aover-A principle which restricts the movement of a whword inside a noun phrase (NP) (Chomsky, 1973); (2) Our algorithm does not move any wh-word that is contained in a clause constituent.”
Datasets: A Community Library for Natural Language Processing (2021, background): “The inﬂuential NLTK project (Bird, 2006) provided a data library that makes it easy to download and access core datasets.”
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation (2014, methodology): “Taken from the Python Natural Language Toolkit (Bird, 2006).”