Learning Visually-Grounded Semantics from Contrastive Adversarial Samples

By Freda Shi, Jiayuan Mao, Tete Xiao, Yuning Jiang. 2018

53 citations recorded by the source corpus.

We study the problem of grounding distributional representations of texts on the visual domain, namely visual-semantic embeddings (VSE for short). Begin with an insightful adversarial attack on VSE embeddings, we show the limitation of current frameworks and image-text datasets (e.g., MS-COCO) both quantitatively and qualitatively. The large gap between the number of possible constitutions of real-world semantics and the size of parallel data, to a large extent, restricts the model to establish a strong link between textual semantics and visual concepts. We alleviate this problem by augmenting the MS-COCO image captioning datasets with textual contrastive adversarial samples. These samples are synthesized using language priors of human and the WordNet knowledge base, and enforce the model to ground learned embeddings to concrete concepts within the image. This simple but powerful technique brings a noticeable improvement over the baselines on a diverse set of downstream tasks, in addition to defending known-type adversarial attacks. Codes are available at https://github.com/ExplorerFreda/VSE-C.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “, 2018) Black 7 Syntax Sentiment, entailment (Shi et al., 2018) Black 7 Word Image captioning (Alzantot et al.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “…that learn visual concepts from descriptive languages, such as image-captioning or visually-grounded question-answer pairs (Kiros et al., 2014; Shi et al., 2018; Mao et al., 2016; Vendrov et al., 2016; Ganju et al., 2017), dense language descriptions for scenes (Johnson et al., 2016),…”
Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations (2019, unknown): “` = `sent + ηc`comp + ηo`obj + ηa`attr + ηr`rel (4) In the first 2 epochs, we set ηc, ηo and ηa to 0.”
Visually Grounded Neural Syntax Acquisition (2019, unknown): “…et al., 2011) is a widely studied approach, and has achieved remarkable results on image-caption retrieval (Kiros et al., 2014; Faghri et al., 2018; Shi et al., 2018a), image caption generation (Kiros et al., 2014; Karpathy and Fei-Fei, 2015; Ma et al., 2015), and visual question answering…”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, methodology): “[125] worked on Visual-Semantic Embedding applications, where the MSCOCO dataset is used.”
Large Language Models Can Be Easily Distracted by Irrelevant Context (2023, methodology): “…input perturbations for natural language tasks, including model-agnostic input transformations (Liang et al., 2022; Ravichan-der et al., 2022, inter alia ) and adversarial example generation against individual models (Jia & Liang, 2017; Shi et al., 2018; Morris et al., 2020; Wang et al., 2021).”