Meaningless yet meaningful: Morphology grounded subword-level NMT
By Tamali Banerjee, Pushpak Bhattacharyya. 2018
53 citations recorded by the source corpus.
We explore the use of two independent subsystems Byte Pair Encoding (BPE) and Morfessor as basic units for subword-level neural machine translation (NMT). We show that, for linguistically distant language-pairs Morfessor-based segmentation algorithm produces significantly better quality translation than BPE. However, for close language-pairs BPE-based subword-NMT may translate better than Morfessor-based subword-NMT. We propose a combined approach of these two segmentation algorithms Morfessor-BPE (M-BPE) which outperforms these two baseline systems in terms of BLEU score. Our results are supported by experiments on three language-pairs: English-Hindi, Bengali-Hindi and English-Bengali.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Benoît Sagot
- Ivan Provilkov
- Dmitrii Emelianenko
- Samson Tan
- Chenglei Si
- Seongbo Jang
- Manan Dey
- Arun Raja
In Computer Vision and Pattern Recognition
- Ivan Provilkov
- Dmitrii Emelianenko
- Samson Tan
- Chenglei Si
- Seongbo Jang
- Manan Dey
- Arun Raja
- Joohong Lee
In Machine Translation
- Ivan Provilkov
- Dmitrii Emelianenko
- Seongbo Jang
- Joohong Lee
- Dawoon Jung
- Colin Raffel
- Elena Voita
- Kyubyong Park
Citations in the corpus
- BPE-Dropout: Simple and Effective Subword Regularization (2019, methodology): “Other segmentation algorithms include Creutz and Lagus (2006), Schuster and Nakajima (2012), Chitnis and DeNero (2015), Kunchukuttan and Bhattacharyya (2016), Wu and Zhao (2018), Banerjee and Bhattacharyya (2018).”
- An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks (2020, methodology): “Banerjee and Bhattacharyya (2018) combined an off-the-shelf morphological segmenter and BPE in Hindi and Bengali translations against English.”
- Rule-based Morphological Inflection Improves Neural Terminology Translation (2021, unknown): “In NMT, morphologically-aware segmentation is also useful when translating from or into morphologically complex languages (Huck et al., 2017; Ataman and Federico, 2018; Banerjee and Bhattacharyya, 2018).”
- Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “Banerjee and Bhattacharyya (2018) analyze translations obtained segmenting with Morfessor and BPE, and conclude that a possible improvement depends on the similarity of the languages.”