Learning compositionally through attentive guidance
By Dieuwke Hupkes, Anand Singh, Kris Korrel, Germán Kruszewski, Elia Bruni. 2018
31 citations recorded by the source corpus.
While neural network models have been successfully applied to domains that require substantial generalisation skills, recent studies have implied that they struggle when solving the task they are trained on requires inferring its underlying compositional structure. In this paper, we introduce Attentive Guidance, a mechanism to direct a sequence to sequence model equipped with attention to find more compositional solutions. We test it on two tasks, devised precisely to assess the compositional capabilities of neural models, and we show that vanilla sequence to sequence models with attention overfit the training distribution, while the guided versions come up with compositional solutions that fit the training and testing distributions almost equally well. Moreover, the learned solutions generalise even in cases where the training and testing distributions strongly diverge. In this way, we demonstrate that sequence to sequence models are capable of finding compositional solutions without requiring extra components. These results helps to disentangle the causes for the lack of systematic compositionality in neural networks, which can in turn fuel future work.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Elia Bruni
- Dieuwke Hupkes
- Verna Dankers
- Kris Korrel
- Yann Dubois
- Gautier Dagan
- Yonatan Belinkov
- Marco Baroni
In Computer Vision and Pattern Recognition
In Natural Language Processing
- Elia Bruni
- Dieuwke Hupkes
- Verna Dankers
- Kris Korrel
- Yann Dubois
- Gautier Dagan
- Yonatan Belinkov
- James R. Glass
In Model Architectures
- Elia Bruni
- Dieuwke Hupkes
- Verna Dankers
- Kris Korrel
- Yann Dubois
- Gautier Dagan
- Yonatan Belinkov
- John Hewitt
Citations in the corpus
- Transcoding Compositionally: Using Attention to Find More Generalizable Solutions (2019, methodology): “Some positive results in this direction have been presented (e.g., Hupkes et al., 2018b), but a number of recent papers have argued that, rather than understanding the underlying compositional structure of a problem, RNNs rely on heuristics and exploit biases in the data.”
- Location Attention for Extrapolation to Longer Sequences (2019, methodology): “To focus on the attention mechanisms, our model and the baselines all use a smaller version of the best performing recurrent seq2seq architecture on the lookup table task (Hupkes et al., 2018).”