One-Size-Fits-All Multilingual Models
By Ben Peters, André F. T. Martins. 2020
19 citations recorded by the source corpus.
This paper presents DeepSPIN’s submissions to Tasks 0 and 1 of the SIGMORPHON 2020 Shared Task. For both tasks, we present multilingual models, training jointly on data in all languages. We perform no language-specific hyperparameter tuning – each of our submissions uses the same model for all languages. Our basic architecture is the sparse sequence-to-sequence model with entmax attention and loss, which allows our models to learn sparse, local alignments while still being trainable with gradient-based techniques. For Task 1, we achieve strong performance with both RNN- and transformer-based sparse models. For Task 0, we extend our RNN-based model to a multi-encoder set-up in which separate modules encode the lemma and inflection sequences. Despite our models’ lack of language-specific tuning, they tie for first in Task 0 and place third in Task 1.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yonatan Belinkov
- Percy Liang
- Luke S. Zettlemoyer
- Adina Williams
- Colin Raffel
- Yejin Choi
- Hinrich Schütze
In Computer Vision and Pattern Recognition
- Colin Raffel
- Mihir Kale
- Rami Al-Rfou
- Noah Constant
- Mohit Bansal
- Adam Roberts
- Adina Williams
- Sharan Narang
In Natural Language Processing
- Adina Williams
- Colin Raffel
- Noah Constant
- Mohit Bansal
- Mihir Kale
- Rami Al-Rfou
- Hinrich Schütze
- Yonatan Belinkov
Citations in the corpus
- ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “On inflection, ByT5 matches the best single-model (Peters and Martins, 2020).”
- SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection (2020, methodology): “DeepSpin reimplemented gated sparse two-headed attention model from Peters and Martins (2019) and trained it on”
- The SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion (2020, background): “Técnico and Unbabel produced four submissions (Peters and Martins 2020) based on sparse attention models.”