Pythia-A platform for vision & language research

By Amanpreet Singh, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, Devi Parikh. 2018

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “We conduct experiments based on the M4C network architecture [20, 48, 47].”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “The Faster-RCNN is trained on Visual Genome [43] with attribute loss following [69] and features from fc6 layer are fine-tuned using weights of the fc7 layer (ImageRegion).”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “A Faster-RCNN model is used to extract features from o RoIs, denoted by { r 0 , · · · , r o − 1 } , from the image to represent its visual content.”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “For each input image, we ﬁrst use Faster R-CNN (weights are initialized from (Singh et al. 2018)) to extract the visual features (pooled ROI features) for each region.”
Towards VQA Models That Can Read (2019, methodology): “Pythia [17, 40] is inspired from the detector-based bounding box prediction of the bottom-up top-down attention network [1] (VQA winner 2017), which in turn has a multi-modal attention mechanism similar to the VQA 2016 winner [7], which relied on grid-based features.”
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, methodology): “In model (6), we follow [27,29] to replace the visual backbone with ResNext-152, and the results show that our model with a powerful visual backbone will further accelerate the performance by a large margin.”
Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning (2021, background): “The BUTD features are adopted by many vision-language works [2, 20, 33, 35] and pre-training works [7, 19, 37].”
TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, methodology): “For BUTD [4], we use the implementation and hyper-parameters from MMF [29,28].”
Scene Text Visual Question Answering (2019, methodology): “Additionally, [50] also introduces a novel architecture that combines a standard VQA model [49] and an independently trained OCR module [7] with a “copy” mechanism, inspired by pointer networks [54, 17], which allows to use OCR recognized words as predicted answers if needed.”
Are we pretraining it right? Digging deeper into visio-linguistic pretraining (2020, methodology): “We use the Pythia framework [45,46] for our experiments which is based on PyTorch 5 .”
BERT Representations for Video Question Answering (2020, background): “Besides these improvements, many other related work, such as [33, 25, 30, 12, 3, 46, 11, 1, 16, 2, 34], has contributed to the advancement of the ﬁeld.”
PlotQA: Reasoning over Scientific Plots (2019, methodology): “Note that we have used the original code [32] released by the authors of these models.”
On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering (2020, methodology): “Vocabulary Upper Bound: As both [28] and [29] are classification based, two dictionaries are built under the widely used rules.”
All-in-One Image-Grounded Conversational Agents (2019, methodology): “For VQA we report the number from Pythia3 as a comparison point, as that method uses the train set only without data augmentation 3https://learnpythia.readthedocs.io/en/latest/tutorials/ pretrained models.html#pretrained-models from the Visual Genome, VisDial or other data augmentations (similar to ”
Location-Aware Graph Convolutional Networks for Video Question Answering (2020, methodology): “After that, associating feature vector with visual regions becomes a popular framework in the VQA research (i.e. Pythia (Singh et al. 2018)).”
Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA (2019, methodology): “For visual objects, following Pythia [43] and LoRRA [44], we detect objects with a Faster R-CNN detector [41] pretrained on the Visual Genome dataset [26], and keeps 100 top-scoring objects per image.”