One Size Does Not Fit All: Comparing NMT Representations of Different Granularities
By Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, Preslav Nakov. 2019
43 citations recorded by the source corpus.
Recent work has shown that contextualized word representations derived from neural machine translation are a viable alternative to such from simple word predictions tasks. This is because the internal understanding that needs to be built in order to be able to translate from one language to another is much more comprehensive. Unfortunately, computational and memory limitations as of present prevent NMT models from using large word vocabularies, and thus alternatives such as subword units (BPE and morphological segmentations) and characters have been used. Here we study the impact of using different kinds of units on the quality of the resulting representations when used to model morphology, syntax, and semantics. We found that while representations derived from subwords are slightly better for modeling syntax, character-based representations are superior for modeling morphology and are also more robust to noisy input.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Nadir Durrani
- Hassan Sajjad
- Fahim Dalvi
- Colin Raffel
In Computer Vision and Pattern Recognition
- Yonatan Belinkov
- Fahim Dalvi
- Hassan Sajjad
- Nadir Durrani
- Colin Raffel
- Mihir Kale
- Lu Jiang
- Rami Al-Rfou
In Natural Language Processing
- Yonatan Belinkov
- Fahim Dalvi
- Hassan Sajjad
- Nadir Durrani
- Mohit Bansal
- Colin Raffel
- Graham Neubig
- Hinrich Schütze
Citations in the corpus
- ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “, 2018), syntactic and semantic tagging tasks (Durrani et al., 2019), and machine translation from a non-English source into English (Shaham and Levy, 2021).”
- Analyzing Redundancy in Pretrained Transformer Models (2020, background): “In the case of sub-words, we pick the last sub-word’s representation (Durrani et al., 2019; Liu et al., 2019).”
- AdvAug: Robust Adversarial Augmentation for Neural Machine Translation (2020, background): “Others also tried to deal with this issue by finding better input representations (Durrani et al., 2019), adding adversarial regularization (Sato et al.”