Compact Trilinear Interaction for Visual Question Answering

By Tuong Khanh Long Do, Thanh-Toan Do, Huy Tran, Erman Tjiputra, Quang D. Tran. 2019

67 citations recorded by the source corpus.

In Visual Question Answering (VQA), answers have a great correlation with question meaning and visual contents. Thus, to selectively utilize image, question and answer information, we propose a novel trilinear interaction model which simultaneously learns high level associations between these three inputs. In addition, to overcome the interaction complexity, we introduce a multimodal tensor-based PARALIND decomposition which efficiently parameterizes trilinear teraction between the three inputs. Moreover, knowledge distillation is first time applied in Free-form Opened-ended VQA. It is not only for reducing the computational cost and required memory but also for transferring knowledge from trilinear interaction model to bilinear interaction model. The extensive experiments on benchmarking datasets TDIUC, VQA-2.0, and Visual7W show that the proposed compact trilinear interaction model achieves state-of-the-art results when using a single model on all three datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “llation is shown in Fig. 10. Student Distillation Modality1 Teacher Data Modality2 Fig. 10 The generic framework for cross-modal distillation. For simplicity, only two modalities are shown. Moreover, Do et al. (2019) proposed a knowledge distillation-based visual question answering method, Knowledge”
VQA With No Questions-Answers Training (2018, background): “Many recent works focused on improving the image-question fused features [17, 9, 84, 10, 16], attention mechanisms for selecting important features [80, 54, 47, 4, 12, 59, 50, 27], including self and guided attention [83, 21, 82], applying pre-trained networks [65, 46, 86], and incorporating outputs”
Coarse-to-Fine Reasoning for Visual Question Answering (2021, methodology): “Regarding the multiple-choice Visual7W dataset, our method outperforms the work CTI [11] by 2.”
ConceptBert: Concept-Aware Representation for Visual Question Answering (2020, methodology): “We design the aggregator by applying the Compact Trilinear Interaction (CTI) (Do et al., 2019) to question, answer, and image features and generate a vector to jointly represent the three features.”
Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals (2021, unknown): “al studies also apply the teacher-student framework in question answering task. Yang et al. [35] designed a multi-teacher knowledge distillation paradigm in a Web Question Answering system. Do et al. [4] and Hu et al. [11] applied the teacher-student framework to visual question answering task and r”