SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

By Taku Kudo, John Richardson. 2018

4,200 citations recorded by the source corpus.

This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “We thank Grady Simon, Noah Fiedel, Samuel R. Bowman, Augustus Odena, Daphne Ippolito, Noah Constant, Orhan Firat, Ankur Bapna, and Sebastian Ruder for their comments on this manuscript; Zak Stone and the TFRC team for their support; Austin Tarango for his guidance on dataset creation; Melvin Johnson”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “After tokenization with SentencePiece [17], we obtain 2.78B, 1.09B, 4.75B, 4.30B, and 19.97B subword pieces for Wikipedia, BooksCorpus, Giga5, ClueWeb, and Common Crawl respectively, which are 32.89B in total.”
Training Compute-Optimal Large Language Models (2022, positive): “Preprocessing Input text is tokenized using a SentencePiece tokenizer with a vocabulary of size 32,000.”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “, 2015), using the implementation from SentencePiece (Kudo and Richardson, 2018).”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “, 2016) using the implementation from SentencePiece (Kudo and Richardson, 2018).”
Adversarial Training for Large Neural Language Models (2020, methodology): “es) separated by a special token [SEP]. To address the problem of out-ofvocabulary words, tokens are divided into subword units, using Byte-Pair Encoding (BPE) (Sennrich et al.,2015) or its variants (Kudo and Richardson, 2018), which generates a ﬁxed-size subword vocabulary to compactly represent wo”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “Like BERT, we use a vocabulary size of 30,000, tokenized using SentencePiece (Kudo & Richardson, 2018) as in XLNet (Yang et al., 2019).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “, 2018a) containing around 375M parameters, and a shared source-target sentence-piece model (SPM) (Kudo and Richardson, 2018) vocabulary with 32k tokens.”
Unified Scaling Laws for Routed Language Models (2022, methodology): “Our implementation takes the token ID assigned to the input by the SentencePiece tokenizer [Kudo and Richardson, 2018] and uses the remainder of it divided by E as the expert selection.”
LaMDA: Language Models for Dialog Applications (2022, methodology): “We used the SentencePiece library [90] to tokenize the dataset into 2.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “We use the SentencePiece (Kudo & Richardson, 2018) subword tokenizer with a vocabulary of size of 256K.”
Improving language models by retrieving from trillions of tokens (2021, methodology): “We tokenize the dataset using SentencePiece (Kudo and Richardson, 2018) with a vocabulary of 128,000 tokens.”
Challenges in Detoxifying Language Models (2021, methodology): “It uses a 32,000 subword vocabulary with a SentencePiece tokenizer (Kudo and Richardson, 2018).”
Competition-level code generation with AlphaCode (2022, methodology): “For tokenization we used a SentencePiece tokenizer (Kudo and Richardson, 2018) with a vocabulary size of 8,000 tokens trained on a mix of GitHub and CodeContests data.”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “This model is pretrained on a collection of web documents (including those with computer code), dialog data, and Wikipedia, tokenized into 2.49T BPE tokens with a 32k vocabulary using the SentencePiece library (Kudo & Richardson, 2018).”