Guide Me: Interacting with Deep Networks

By Christian Rupprecht, Iro Laina, Nassir Navab, Gregory Hager, Federico Tombari. 2018

39 citations recorded by the source corpus.

Interaction and collaboration between humans and intelligent machines has become increasingly important as machine learning methods move into real-world applications that involve end users. While much prior work lies at the intersection of natural language and vision, such as image captioning or image generation from text descriptions, less focus has been placed on the use of language to guide or improve the performance of a learned visual processing algorithm. In this paper, we explore methods to flexibly guide a trained convolutional neural network through user input to improve its performance during inference. We do so by inserting a layer that acts as a spatio-semantic guide into the network. This guide is trained to modify the network's activations, either directly via an energy minimization scheme or indirectly through a recurrent model that translates human language queries to interaction weights. Learning the verbal interaction is fully automatic and does not require manual text annotations. We evaluate the method on two datasets, showing that guiding a pre-trained network can improve performance, and provide extensive insights into the interaction between the guide and the CNN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Self-Refine: Iterative Refinement with Self-Feedback (2023, background): “, 2022a; Yasunaga and Liang, 2020), computer vision (Rupprecht et al., 2018), and other tasks (Bai et al.”
Self-critiquing models for assisting human evaluators (2022, methodology): “Human natural language feedback has been used to improve models in many domains, such as computer vision [RLN18], program synthesis [EHA20, AON21], and summarization [SCC22].”
Training Language Models with Language Feedback (2022, background): “Other work uses language feedback at test time to correct mistakes in a model’s behavior, for e.g. image segmentation (Rupprecht et al., 2018) or code generation (Elgohary et al., 2020; Austin et al., 2021).”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, background): “…2020), as additional supervision (Narang et al., 2020; Hase et al., 2020; Pruthi et al., 2020) or intermediate structured variables (Camburu et al., 2018; Rajani et al., 2019; Wiegreffe et al., 2020), and simply as model inputs (Rupprecht et al., 2018; Co-Reyes et al., 2019; Zhou et al., 2020).”
Learning from Natural Language Feedback (2024, methodology): “In the image domain, language has been used to change certain visual aspects of images (Rupprecht et al., 2018; Tian et al., 2022), while we focus on the language domain.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, methodology): “Natural Language Feedback The technique of using natural language (NL) feedback is widely adopted to improve various tasks (Rupprecht et al., 2018; Scheurer et al., 2022).”
Interactive Object Segmentation With Inside-Outside Guidance (2022, background): “Rupprecht et al. [81], [82] use natural language as feedback for interactive segmentation correction.”
Interactive Full Image Segmentation by Considering All Regions Jointly (2018, background): “In [48] they combine a segmentation network with a language module to allow a human to correct the segmentation by typing feedback in natural language, such as “there are no clouds visible in this image”.”
Training Language Models with Natural Language Feedback (2022, background): “Other work uses language feed-back at test time to correct mistakes in a model’s behavior, for e.g. image segmentation (Rupprecht et al., 2018) or code generation (Elgohary et al., 2020; Austin et al., 2021).”
Medical Image Segmentation Using Deep Learning: A Survey (2020, methodology): “g learn to extract the foreground part of the object from a bounding box. During the test, the CNN can better use the information in the speciﬁc image through an adaptive ﬁne-tuning. Rupprecht et al. [95] proposed a new interactive segmentation method named GM interacting that updates image segmenta”
PhraseClick: Toward Achieving Flexible Interactive Segmentation by Phrase and Click (2020, methodology): “We propose to utilize both interaction types to make selection ﬂexible and robust to handle complex scenarios. to drive the selection [24,35,50,61].”
Training Language Models with Language Feedback at Scale (2023, background): “Other work uses language feedback at test time to correct mistakes in a model’s behavior, e.g., image segmentation (Rupprecht et al., 2018) or code generation (Elgohary et al., 2020; Austin et al., 2021).”