Visually Grounded Continual Learning of Compositional Phrases

By Xisen Jin, Junyi Du, Arka Sadhu, Ramkant Nevatia, Xiang Ren. 2020

18 citations recorded by the source corpus.

Humans acquire language continually with much more limited access to data samples at a time, as compared to contemporary NLP systems. To study this human-like language acquisition ability, we present VisCOLL, a visually grounded language learning task, which simulates the continual acquisition of compositional phrases from streaming visual scenes. In the task, models are trained on a paired image-caption stream which has shifting object distribution; while being constantly evaluated by a visually-grounded masked language prediction task on held-out test sets. VisCOLL compounds the challenges of continual learning (i.e., learning from continuously shifting data distribution) and compositional generalization (i.e., generalizing to novel compositions). To facilitate research on VisCOLL, we construct two datasets, COCO-shift and Flickr-shift, and benchmark them using different continual learning methods. Results reveal that SoTA continual learning approaches provide little to no improvements on VisCOLL, since storing examples of all possible compositions is infeasible. We conduct further ablations and analysis to guide future work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Vladlen Koltun
2. Hexiang Hu
3. Fei Sha
4. Ozan Sener
5. G. Sukhatme
6. Thomas Scialom
7. Sergey Levine
8. Govind Thattai

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Hexiang Hu
3. Vladlen Koltun
4. Trevor Darrell
5. Jitendra Malik
6. Fei Sha
7. Bernt Schiele
8. Dacheng Tao

In Experimental and Cognitive Psychology

1. Smaranda Muresan
2. Jesse Thomason
3. Qiaozi Gao
4. Tuhin Chakrabarty

Citations in the corpus

• Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “Advances in simulators and large-scale data collection have facilitated moving past static language grounding tasks like visual question answering [4, 23] and captioning [80, 65, 36] to embodied benchmarks like Vision-and-Language Navigation (VLN) [49, 17, 3, 61, 41], embodied question answering [22”
• Drinking From a Firehose: Continual Learning With Web-Scale Natural Language (2020, background): “[72] has created a dataset that evaluate model’s capability of learning the composition of visually grounded phrases under continuous data distribution shift, which has”
• Fine-tuned Language Models are Continual Learners (2022, background): “Jin et al. (2020) show how existing CL algorithms fail at learning compositional phrases.”