Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books
By Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, Sanja Fidler. 2015
2,762 citations recorded by the source corpus.
Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This paper aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in the current datasets. To align movies and books we propose a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Noah A. Smith
- Graham Neubig
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Multimodal Learning
Citations in the corpus
- Layer Normalization (2016, methodology): “Using the publicly available code of Kiros et al. [2015] 4 , we train two models on the BookCorpus dataset [Zhu et al., 2015]: one with and one without layer normalization.”
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “We first pretrain the language model for two epochs on pairs of two sentences in the Toronto Books dataset (Zhu et al., 2015), and then train on sentence pairs from ActivityNet Captions and LSMDC, validating on held-out perplexity.”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “) relating two sentences on data derived from TBC (Zhu et al., 2015).”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “In Section 3.4.1, we showed that pre-training on the RealNews-like, WebText-like, and Wikipedia + TBC datasets outperformed pre-training on C4 on a few downstream tasks.”
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “• BOOKCORPUS (Zhu et al., 2015) plus English WIKIPEDIA.”
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “For fair comparison, all models are based on a 12-layer architecture with the same model hyper-parameters as BERT-Base and are trained on only Wikipedia and the BooksCorpus.”
- Scaling Laws for Neural Language Models (2020, methodology): “6 × 10 8 of these tokens for use as a test set, and we also test on similarly-prepared samples of Books Corpus [ZKZ + 15], Common Crawl [Fou], English Wikipedia, and a collection of publicly-available Internet Books.”
- StereoSet: Measuring stereotypical bias in pretrained language models (2020, methodology): “For example, both BERT-base and BERT-large are trained on Wikipedia and BookCorpus (Zhu et al., 2015) with 110M and 340M parameters respectively.”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “Since the BookCorpus is no longer publicly available, we follow Devlin et al. (2019) and crawl http://smashwords.com .”
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “The experiments done up to this point use only the Wikipedia and BOOKCORPUS datasets, as in (Devlin et al., 2019).”
- Fine-Tuning Language Models from Human Preferences (2019, methodology): “We first apply our method to stylistic text continuation tasks, where the policy is presented with an excerpt from the Book-Corpus dataset (Zhu et al., 2015) and generates a continuation of the text.”
- HellaSwag: Can a Machine Really Finish Your Sentence? (2019, methodology): “OpenAI GPT (Radford et al., 2018): A fine-tuned 12-layer transformer that was pre-trained on the BookCorpus (Zhu et al., 2015). b .”
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, methodology): “Our OpenAI GPT model fine-tunes the 12 layer 768 dimensional uni-directional transformer from Radford et al. (2018), which has been pretrained as a language model on the Books corpus (Zhu et al., 2015).”
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, methodology): “Based on the improvements seen from applying supplementary training on the fake-sentence-detection task, which is built on the same BooksCorpus dataset that the GPT model was trained on, it is also clear that the benefits from supplementary training do not entirely stem from the trained model being e”