Multiplicative Interactions and Where to Find Them

By Siddhant M. Jayakumar, Jacob Menick, Wojciech M. Czarnecki, Jonathan Schwarz, Jack W. Rae, Simon Osindero, Yee Whye Teh, Tim Harley, Razvan Pascanu. 2020

160 citations recorded by the source corpus.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Quoc V. Le
2. Hanxiao Liu
3. Zihang Dai
4. David R. So
5. Chen Liang
6. Noam Shazeer
7. Esteban Real
8. Tie-Yan Liu

Citations in the corpus

• Primer: Searching for Efficient Transformers for Language Modeling (2021, background): “Interestingly, the effectiveness of higher order polynomials [37] can also be observed in other effective Transformer nonlinearities, such as GLU [38] variants like ReGLU [39] ( y = Ux (cid:12) max( V x, 0) where (cid:12) is an element-wise product) and point-wise activations like approximate GELU…”
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “(2) Multiplicative interactions (Jayakumar et al., 2020) emerge in a bilinear form: o = aᵀWb, where o is the output, and W is the weight matrix.”
• Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think! (2020, background): “Jayakumar et al. (2020) show that feed-forward neural networks can require a very high number of training examples to learn feature interactions.”
• Neural networks with late-phase weights (2020, background): “Building upon recent work on ensembles with shared parameters (Wen et al., 2020) we explore a family of late-phase weight models involving multiplicative interactions (Jayakumar et al., 2020).”