Shaping Visual Representations with Language for Few-Shot Classification

By Jesse Mu, Percy Liang, Noah D. Goodman. 2019

49 citations recorded by the source corpus.

By describing the features and abstractions of our world, language is a crucial tool for human learning and a promising source of supervision for machine learning models. We use language to improve few-shot visual classification in the underexplored scenario where natural language task descriptions are available during training, but unavailable for novel tasks at test time. Existing models for this setting sample new descriptions at test time and use those to classify images. Instead, we propose language-shaped learning (LSL), an end-to-end model that regularizes visual representations to predict language. LSL is conceptually simpler, more data efficient, and outperforms baselines in two challenging few-shot domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Others have investigated how grounded language can be used to improve visual representations and classifiers on the ShapeWorld dataset (Kuhnle & Copestake, 2017; Andreas et al., 2017; Mu et al., 2019).”
ALICE: Active Learning with Contrastive Natural Language Explanations (2020, background): “training loop is effective in various settings (Andreas et al., 2018; Mu et al., 2020).”
Can language models learn from explanations in context? (2022, methodology): “…have been used for training or tuning models for computer vision (Hendricks et al., 2016; Schramowski et al., 2020; Majumder et al., 2021; Mu et al., 2020), for mathematical problem solving by program induction (Ling et al., 2017), or relational reasoning and reinforcement learning…”
Natural Language Inference with a Human Touch: Using Human Explanations to Guide Model Attention (2021, background): “…to improve performance on a range of domains, including commonsense reasoning (Rajani et al., 2019), relation extraction (Murty et al., 2020), visual clas-siﬁcation tasks (Liang et al., 2020; Mu et al., 2020), sentiment analysis (Pruthi et al., 2020) and question answering (Pruthi et al., 2020).”
Transforming task representations to perform novel tasks (2020, background): “Furthermore, we considered language as input, but producing language as output (in the form of explanations) can improve understanding and generalization in both humans (53) and neural networks (54).”
Leveraging Language to Learn Program Abstractions and Search Heuristics (2021, methodology): “…representation learning (Frome et al., 2013; Jia & Liang, 2016; Andreas et al., 2017; Ye et al., 2020b; Goyal et al., 2020; Liang et al., 2020; Mu et al., 2019; Luketina et al., 2019), and as a high-level specification for program synthesis tasks (Ye et al., 2020a; Nye et al., 2019;…”
Supervising Model Attention with Human Explanations for Robust Natural Language Inference (2021, background): “Human explanations have been found to improve performance on a range of tasks (Rajani et al. 2019; Andreas, Klein, and Levine 2018; Mu, Liang, and Goodman 2020; Liang, Zou, and Yu 2020); however, this has largely not been the case in NLI (Hase and Bansal 2021; Kumar and Talukdar 2020; Camburu et…”
Tell me why! - Explanations support learning of relational and causal structure (2021, background): “Thus, there has been increasing interest in using explanations as learning signal (e.g. Ross et al., 2017; Mu et al., 2020; Camburu et al., 2018; Schramowski et al., 2020; see related work).”
Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations (2020, background): “Lastly, language-guided XIL [34] is an interesting approach for more natural supervision.”
Tell me why! Explanations support learning relational and causal structure (2021, background): “Indeed, there has been increasing interest in using language or explanations as a learning signal for machines (e.g., Ross et al., 2017; Mu et al., 2020; Camburu et al., 2018; Schramowski et al., 2020; see related work).”