Seeing with Humans: Gaze-Assisted Neural Image Captioning

By Yusuke Sugano, Andreas Bulling. 2016

76 citations recorded by the source corpus.

Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems. Previous works demonstrated the potential of gaze for object-centric tasks, such as object localization and recognition, but it remains unclear if gaze can also be beneficial for scene-centric tasks, such as image captioning. We present a new perspective on gaze-assisted image captioning by studying the interplay between human gaze and the attention mechanism of deep neural networks. Using a public large-scale gaze dataset, we first assess the relationship between state-of-the-art object and scene recognition models, bottom-up visual saliency, and human gaze. We then propose a novel split attention model for image captioning. Our model integrates human gaze information into an attention-based long short-term memory architecture, and allows the algorithm to allocate attention selectively to both fixated and non-fixated image regions. Through evaluation on the COCO/SALICON datasets we show that our method improves image captioning performance and that gaze can complement machine attention for semantic scene understanding tasks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Human-Computer Interaction

Citations in the corpus

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “• SupervisedAttentionKnowing to pay attention to the important information is a key step when extracting useful information from objects such as long text sequences [75, 129], images [130, 149], or knowledge bases [23, 146]).”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, neutral): “However, Sugano et al. [129] claimed that gaze information has not yet been integrated in image captioning methods.”
Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model (2016, background): “An intensive research effort aimed to emulate such selective visual mechanisms, as computational saliency can be applied to a wide range of applications like image retargeting [2], [3], object recognition [4], video compression [5], tracking [6] and other data-dependent tasks such as image captionin”
Human Gaze Assisted Artificial Intelligence: A Review (2020, background): “…the referred object from all proposals (object referring) [Bala-jee Vasudevan et al. , 2018], generating descriptive captions for images [Sugano and Bulling, 2016; Tavakoli et al. , 2017; Cornia et al. , 2018a; Chen and Zhao, 2018; He et al. , 2019b] and videos [Yu et al. , 2017], as…”
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention (2017, background): “In particular, Sugano et al. [42] presented a gaze-assisted attention mechanism for image caption based on human eye fixations ( i.e. the static states of gaze upon a specific location).”
Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention (2020, background): “An especially interesting way of leveraging gaze information was demonstrated by works incorporating human gaze into neural attention mechanisms, for example for image and video captioning [71, 83] or visual question answering [58].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “This idea was first explored by Sugano and Bulling [53] who exploited human eye fixations for image captioning by including normalized fixation histograms over the image as an input to the soft-attention module of [42] and weighing the attended image regions based on whether these are fixated or not”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, background): “nisms [154] fail to focus on regions of interest without prior knowledge, visual stimuli methods [155], [156] alone are not sufficient to generate captions for images.”
Resources and End-to-End Neural Network Models for Arabic Image Captioning (2020, unknown): “…salient regions of the image while generating words (Xu et al., 2015) (Wu and Cohen, 2016) (Jin et al., 2015) (Pedersoli et al., 2016) (Lu et al., 2017) (Liu et al., 2017) (Chen et al., 2017) (Tavakoliy et al., 2017) (Anderson et al., 2017) (Chunseong Park et al., 2017) (Sugano and Bulling, 2016).”
Gaze360: Physically Unconstrained Gaze Estimation in the Wild (2019, background): “Just as a parent’s gaze can help to guide a child’s attention, human gaze fixations have also been found to be useful in helping machines to learn or interact in various contexts [18, 22].”