An Empirical Study of Mini-Batch Creation Strategies for Neural Machine Translation

By Makoto Morishita, Yusuke Oda, Graham Neubig, Koichiro Yoshino, Katsuhito Sudoh, Satoshi Nakamura. 2017

26 citations recorded by the source corpus.

Training of neural machine translation (NMT) models usually uses mini-batches for efficiency purposes. During the mini-batched training process, it is necessary to pad shorter sentences in a mini-batch to be equal in length to the longest sentence therein for efficient computation. Previous work has noted that sorting the corpus based on the sentence length before making mini-batches reduces the amount of padding and increases the processing speed. However, despite the fact that mini-batch creation is an essential step in NMT training, widely used NMT toolkits implement disparate strategies for doing so, which have not been empirically validated or compared. This work investigates mini-batch creation strategies with experiments over two different datasets. Our results suggest that the choice of a mini-batch creation strategy has a large effect on NMT training and some length-based sorting strategies do not always work well compared with simple shuffling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “There are multiple strategies to batch input and output sequence pairs (Morishita et al., 2017).”
Neural Machine Translation: A Review (2019, result): “Previous studies (Morishita et al., 2017; Neishi et al., 2017) on batch size were limited by the hardware since – in vanilla SGD – the training batch has to fit into the GPU memories.”
SpeechBrain: A General-Purpose Speech Toolkit (2021, methodology): “A more sophisticated approach is to apply dynamic batching [68, 69], where sentences are clustered by length and sampled within the same cluster, a trade-off between random and sorted batching.”
Scheduled Multi-Task Learning: From Syntax to Translation (2018, methodology): “We use mini-batching that limits the number of words in the mini-batch instead of the number of sentences (Morishita et al., 2017).”