Graph Optimal Transport for Cross-Domain Alignment

By Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, Jingjing Liu. 2020

206 citations recorded by the source corpus.

Cross-domain alignment between two sets of entities (e.g., objects in an image, words in a sentence) is fundamental to both computer vision and natural language processing. Existing methods mainly focus on designing advanced attention mechanisms to simulate soft alignment, with no training signals to explicitly encourage alignment. The learned attention matrices are also dense and lacks interpretability. We propose Graph Optimal Transport (GOT), a principled framework that germinates from recent advances in Optimal Transport (OT). In GOT, cross-domain alignment is formulated as a graph matching problem, by representing entities into a dynamically-constructed graph. Two types of OT distances are considered: (i) Wasserstein distance (WD) for node (entity) matching; and (ii) Gromov-Wasserstein distance (GWD) for edge (structure) matching. Both WD and GWD can be incorporated into existing neural network models, effectively acting as a dropin regularizer. The inferred transport plan also yields sparse and self-normalized alignment, enhancing the interpretability of the learned model. Experiments show consistent outperformance of GOT over baselines across a wide range of tasks, including image-text retrieval, visual question answering, image captioning, machine translation, and text summarization.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Zhe Gan
2. Linjie Li
3. Xu Han
4. Yen-Chun Chen
5. Yu Cheng
6. Mohit Bansal
7. Xipeng Qiu
8. Jianfeng Gao

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, methodology): “To learn the fine-grained alignment between image regions and words, UNITER further proposes a word-region alignment task in the way of Optimal Transport (Chen et al., 2020c), which first finds a sparse matching between image regions and words, and then minimizes the alignment distance.”
• MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding (2021, unknown): “Following recent progress in building pre-trained multimedia models (Tan and Bansal 2019; Chen et al. 2020b; Li et al. 2020c), we experiment with finetuning an end-to-end multimedia QA model for our task.”
• UNITER: UNiversal Image-TExt Representation Learning (2019, background): “(OT) [37,7] to explicitly encourage fine-grained alignment between words and image regions.”
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions (2021, methodology): “Representation Learning Self-supervision involves creating supervision objectives from natural data, often by corrupting the input and training the model to reconstruct the input (Kolesnikov et al., 2019) or contrastive learning (Chen et al., 2020b).”
• Graph Information Aggregation Cross-Domain Few-Shot Learning for Hyperspectral Image Classification (2022, methodology): “A transport plan shared by WD and GWD is employed to integrate the WD and GWD [39] and obtain the GOT distance.”