Adapting BERT for Target-Oriented Multimodal Sentiment Classification

By Jianfei Yu, Jing Jiang. 2019

319 citations recorded by the source corpus.

As an important task in Sentiment Analysis, Target-oriented Sentiment Classification (TSC) aims to identify sentiment polarities over each opinion target in a sentence. However, existing approaches to this task primarily rely on the textual content, but ignoring the other increasingly popular multimodal data sources (e.g., images), which can enhance the robustness of these text-based models. Motivated by this observation and inspired by the recently proposed BERT architecture, we study Target-oriented Multimodal Sentiment Classification (TMSC) and propose a multimodal BERT architecture. To model intra-modality dynamics, we first apply BERT to obtain target-sensitive textual representations. We then borrow the idea from self-attention and design a target attention mechanism to perform target-image matching to derive target-sensitive visual representations. To model inter-modality dynamics, we further propose to stack a set of self-attention layers to capture multimodal interactions. Experimental results show that our model can outperform several highly competitive approaches for TSC and TMSC.

Subfields: Artificial Intelligence, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “Other datasets using internet data include Food101 [80], where the goal is to predict the dish of recipes and images; various versions of Yelp reviews [52]; Walmart and Ferramenta product classification [90, 21]; social media name tagging (Twitter and Snapchat) [49]; social media target-oriented sen”
Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges (2023, methodology): “The sentiment models are trained by minimizing the MSE (Jiang et al. [184].”
VAuLT: Augmenting the Vision-and-Language Transformer with the Propagation of Deep Language Representations (2022, methodology): “We use TWITTER-15 and TWITTER-17 (Yu and Jiang 2019) annotated for the task of Target-oriented Sentiment Classiﬁcation.”
Span-based Localizing Network for Natural Language Video Localization (2020, methodology): “…CTRL (Gao et al., 2017), ACRN (Liu et al., 2018a), TGN (Chen et al., 2018), ACL-K (Ge et al., 2019), QSPN (Xu et al., 2019), SAP (Chen and Jiang, 2019), MAN (Zhang et al., 2019), SM-RL (Wang et al., 2019), RWM-RL (He et al., 2019), L-Net (Chen et al., 2019), ExCL (Ghosh et al., 2019…”