Visual Attention Model for Name Tagging in Multimodal Social Media

By Di Lu, Leonardo Neves, Vitor R. Carvalho, Ning Zhang, Heng Ji. 2018

304 citations recorded by the source corpus.

Everyday billions of multimodal posts containing both images and text are shared in social media sites such as Snapchat, Twitter or Instagram. This combination of image and text in a single message allows for more creative and expressive forms of communication, and has become increasingly common in such sites. This new paradigm brings new challenges for natural language understanding, as the textual component tends to be shorter, more informal, and often is only understood if combined with the visual context. In this paper, we explore the task of name tagging in multimodal social media posts. We start by creating two new multimodal datasets: the first based on Twitter posts and the second based on Snapchat captions (exclusively submitted to public and crowd-sourced stories). We then propose a novel model architecture based on Visual Attention that not only provides deeper visual understanding on the decisions of the model, but also significantly outperforms other state-of-the-art baseline methods for this task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Francesco Barbieri
2. José Camacho-Collados
3. Aixin Sun
4. Leonardo Neves
5. Chenliang Li
6. Gargi Ghosh
7. Luis Espinosa-Anke
8. Jianfei Yu

In Computer Vision and Pattern Recognition

1. Gargi Ghosh
2. José Camacho-Collados
3. Jianfei Yu
4. Jing Jiang
5. Francesco Barbieri
6. Leonardo Neves
7. Aixin Sun
8. Xiaochang Peng

In Multimodal Learning

1. Aixin Sun
2. Gargi Ghosh
3. José Camacho-Collados
4. Jing Jiang
5. Ringki Das
6. Thoudam Doren Singh

In Information Extraction

1. José Camacho-Collados
2. Jing Jiang

Citations in the corpus

• TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification (2020, background): “There are also other scenarios to be addressed as well, like sequence tagging (Baldwin et al., 2015; Gimpel et al., 2018), multimodality (Schifanella et al., 2016; Lu et al., 2018), and code-switching tasks (Barman et al., 2014; Vilares et al., 2016).”
• Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer (2020, methodology): “Besides, we also consider several competitive multimodal approaches for MNER: (1) GVATT-HBiLSTM-CRF (Lu et al., 2018), a state-of-the-art approach for MNER, which integrates HBiLSTM-CRF with the visual context by proposing a visual attention mechanism followed by a visual gate to obtain word-aware v”
• A Survey on Deep Learning for Named Entity Recognition (2018, background): “, gazetteers [18], [108], lexical similarity [109], linguistic dependency [110] and visual features [111]) into the final representations of words, before feeding into context encoding layers.”
• Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification (2019, methodology): “Attention mechanism was initially proposed in neural machine translation to dynamically adjust the focus on the source sentence (Bahdanau et al., 2014), but its application has been extended to many areas including multimodal fusion (Lu et al., 2018; Ghosal et al., 2018; Bagher Zadeh et al., 2018).”