Pythia v0.1: the Winning Entry to the VQA Challenge 2018

By Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, Devi Parikh. 2018

210 citations recorded by the source corpus.

This document describes Pythia v0.1, the winning entry from Facebook AI Research (FAIR)'s A-STAR team to the VQA Challenge 2018. Our starting point is a modular re-implementation of the bottom-up top-down (up-down) model. We demonstrate that by making subtle but important changes to the model architecture and the learning rate schedule, fine-tuning image features, and adding data augmentation, we can significantly improve the performance of the up-down model on VQA v2.0 dataset -- from 65.67% to 70.22%. Furthermore, by using a diverse ensemble of models trained with different features and on different datasets, we are able to significantly improve over the 'standard' way of ensembling (i.e. same model with different random seeds) by 1.31%. Overall, we achieve 72.27% on the test-std split of the VQA v2.0 dataset. Our code in its entirety (training, evaluation, data-augmentation, ensembling) and pre-trained models are publicly available at: this https URL

Subfields: Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Linjie Li
4. Mohit Bansal
5. Jingjing Liu
6. Jianfeng Wang
7. Jianfeng Gao
8. Devi Parikh

Citations in the corpus

• VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “For instance, attribute annotations in Visual Genome (Krishna et al., 2017) are used to enhance the object detector in VQA systems (Anderson et al., 2018).”
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “Among them, LoRRA [47] approached Text-VQA by extending a VQA model Pythia [24] with an OCR attention branch.”
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “VQA The SotA result is BAN+Counter in Kim et al. (2018), which achieves the best accuracy among other recent works: MFH (Yu et al., 2018), Pythia (Jiang et al., 2018), DFAF (Gao et al., 2019a), and Cycle-Consistency (Shah et al., 2019).”
• How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “For training VQA models, we use hyperparameters of the open-source implementation9 from (Jiang et al., 2020) for the large version of the MCAN and base version of Pythia.”
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “0 [23], CLEVR [32], Visual7w [81], Visual Genome [41], COCO-QA [57], and models [33, 60, 29, 31, 1, 67, 17, 37] have been used to measure the progress in understanding the interaction between vision and language modalities.”
• All You May Need for VQA are Image Captions (2022, methodology): “0 train/dev splits of *train2014 and minival2014 (Jiang et al., 2018) 2 .”
• Visual Reasoning by Progressive Module Networks (2018, background): “Also, as Jiang et al. (2018) showed, the performance depends strongly on engineering choices such as learning rate scheduling, data augmentation, and ensembling models with different architectures.”
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, background): “For thorough evaluation, we compare model performance on the following set of competing methods: • SOTA (task-specific state of the art): Cycle Consis- tency+BAN [56] for VQA-Rephrasings, LOL [18] for VQA-LOL Compose and Supplement, Pythia [54, 28] for VQA-Introspect, NSM [26] for GQA, SAAA [2, 30] ”
• Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering (2019, background): “1 (Jiang et al., 2018) to preprocess VizWiz dataset and retain 3135 top answers.”