Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

By Taku Kudo. 2018

1,379 citations recorded by the source corpus.

Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “We use SentencePiece [Kudo and Richardson, 2018] to encode text as WordPiece tokens [Sennrich et al., 2015; Kudo, 2018].”
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018, methodology): “SentencePiece implements two subword segmentation algorithms, byte-pair-encoding (BPE) (Sennrich et al., 2016) and uni-gram language model (Kudo, 2018), with the extension of direct training from raw sentences.”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “, 2021) or randomly choosing different subword segmentations to improve robustness in low-resource and out-of-domain settings (Kudo, 2018).”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “As in T5, we use Senten-cePiece (Kudo and Richardson, 2018; Kudo, 2018) wordpiece models that are trained with the same language sampling rates used during training.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “We compared two tokenization methods2: Byte-pairencoding algorithm (BPE) (Wu et al., 2016; Sennrich et al., 2016), and SentencePiece Unigram algorithm (Unigram) proposed in Kudo (2018).”
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, methodology): “More precisely, for each language, we train a sentence piece tokenizer (Kudo, 2018) and a language model on data from en ru zh fr de ja es it pt pl nl ko id cs fa ro ar el hu da uk no fi bg he Language the targeted domain.”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, background): “, 2015) produced by a tokenizer like SentencePiece (Kudo, 2018; Kudo & Richardson, 2018).”
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, methodology): “To simplify this process, we create a shared token set across all languages using a Sentence Piece Model (SPM) [15] .”
BPE-Dropout: Simple and Effective Subword Regularization (2019, background): “1) and were chosen to match the ones used in the prior work (Kudo, 2018).”
Unsupervised Cross-lingual Representation Learning at Scale (2019, methodology): “CC-100 increases the amount of data by several orders of magnitude, in particular for low-resource languages. enization directly on raw text data using Sentence Piece (Kudo and Richardson, 2018) with a unigram language model (Kudo, 2018).”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “(Kudo, 2018) and segment text by using SentencePiece7.”
Textless Speech-to-Speech Translation on Real Data (2021, methodology): “We build 2000 unigram subword units (Kudo, 2018) from the ASR decoded text as the target.”
Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “The target text for CTC is encoded as 1k unigram subword units [30] to guarantee that the text sequence length is shorter than the length of the stacked or reduced discrete unit sequence.”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “To tokenize text input, we use a sentence-piece model [43, 44] with a vocabulary size of 64k trained on the sampled pretraining dataset.”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, methodology): “For regularization we use 20% dropout (Srivastava et al., 2014), 10% label smoothing, 1% decoder input sampling, and 1% word piece sampling (Kudo, 2018) following Kahn et al. (2019a).”