Tricks for Training Sparse Translation Models

By Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Michael Lewis, Angela Fan. 2021

23 citations recorded by the source corpus.

Multi-task learning with an unbalanced data distribution skews model learning towards high resource tasks, especially when model capacity is fixed and fully shared across all tasks. Sparse scaling architectures, such as BASELayers, provide flexible mechanisms for different tasks to have a variable number of parameters, which can be useful to counterbalance skewed data distributions. We find that that sparse architectures for multilingual machine translation can perform poorly out of the box and propose two straightforward techniques to mitigate this — a temperature heating mechanism and dense pre-training. Overall, these methods improve performance on two multilingual translation benchmarks compared to standard BASELayers and Dense scaling baselines, and in combination, more than 2x model convergence speed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, background): “Dua et al. (2021) proposes a similar approach of ﬁrst training a dense model where each inputs goes to every expert and then adapting it to be sparse.”
Mixture-of-Experts with Expert Choice Routing (2022, methodology): “Hash layers [8, 28] devise hashing techniques on input tokens.”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “[100].”