Diversity and Depth in Per-Example Routing Models

By Prajit Ramachandran, Quoc V. Le. 2018

35 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Unified Scaling Laws for Routed Language Models (2022, background): “However, SMoE models have historically routed datapoints to more than one expert [Shazeer et al., 2017, Lepikhin et al., 2020, Ramachandran and Le, 2018].”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “[21], Ramachandran and Le [28], Fedus et al.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, neutral): “More recent work (Devin, Gupta, Darrell, Abbeel, and Levine, 2017; Frans, Ho, Chen, Abbeel, and Schulman, 2017; Fernando, Banarse, Blundell, Zwols, Ha, Rusu, Pritzel, and Wierstra, 2017; Rosenbaum, Klinger, and Riemer, 2017; Meyerson and Miikkulainen, 2017; Kirsch, Kunze, and Barber, 2018; Ramachand”
Routing Networks with Co-training for Continual Learning (2020, background): “L G ] 9 S ep 2 02 0 creating specialist experts resulting in improved predictive performance compared to non-sparse network architectures (Ramachandran & Le, 2019).”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, methodology): “However, routing networks are a generalization of these approaches [Ramachandran and Le, 2019] and are highly related in particular to the concept of one-shot neural architecture search [Brock et al.”
Scaling Vision with Sparse Mixture of Experts (2021, background): “Scaling up routing depth was marked as a major challenge in [51], which we successfully tackle in our work.”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “A thorough discussion of the strengths and weaknesses of routing based approaches can be found in (Rosenbaum et al., 2019) and (Ramachandran and Le, 2019).”
On the Role of Weight Sharing During Deep Option Learning (2019, background): “…Riemer et al., 2015; Misra et al., 2016; Rusu et al., 2016; Shazeer et al., 2017; Fernando et al., 2017; Rosenbaum et al., 2018; Ruder et al., 2017; Ramachandran & Le, 2018; Cases et al., 2019) that directly addresses catastrophic forgetting, increasing representation stability during learning.”
Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts (2021, background): “[34] Prajit Ramachandran and Quoc V Le.”
Gumbel-Matrix Routing for Flexible Multi-task Learning (2019, methodology): “However, (Ramachandran & Le, 2019) report that their models are often hard to train.”
Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer (2020, background): “Empirical investigation of routing networks, a type of modular networks, has emphasized the importance of two key characteristics in ensuring representational power: the diversity of modules per layer and the depth of the network (i.e., the number of routing paths) (Ramachandran & Le, 2019).”