Learning to Describe Video with Weak Supervision by Exploiting Negative Sentential Information

By Haonan Yu, Jeffrey Mark Siskind. 2015

9 citations recorded by the source corpus.

Most previous work on video description trains individualparts of speech independently. It is more appealing from a linguistic point of view, for word models for all parts of speech to be learned simultaneously from whole sentences, a hypothesis suggested by some linguists for child language acquisition. In this paper, we learn to describe video by discriminatively training positive sentential labels against negative ones in a weakly supervised fashion: the meaning representations (i.e., HMMs) of individual words in these labels are learned from whole sentences without any correspondence annotation of what those words denote in the video. Textual descriptions are then generated for new video using trained word models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Yang
2. Haonan Yu
3. Wei Xu
4. Jiang Wang
5. Zhiheng Huang
6. Devi Parikh
7. Luowei Zhou
8. Jianfeng Gao

In Computer Vision and Pattern Recognition

1. Yi Yang
2. Luc Van Gool
3. Trevor Darrell
4. Dacheng Tao
5. Cordelia Schmid
6. Xiaogang Wang
7. Andrew Zisserman
8. Li Fei-Fei

In Natural Language Processing

1. Yi Yang
2. Wei Xu
3. Haonan Yu
4. Luowei Zhou
5. Chenliang Xu
6. Jason J. Corso
7. Zhiheng Huang
8. Jiang Wang

In Natural Language Generation

1. Wei Xu
2. Jiang Wang
3. Haonan Yu
4. Zhiheng Huang
5. Jason J. Corso
6. Yi Yang

Citations in the corpus

• Towards Automatic Learning of Procedures From Web Instructional Videos (2017, background): “2016)—learning the steps of accomplishing these tasks and their dependencies is essential, especially for agents’ automatic acquisition of language or manipulation skills from video (Yu and Siskind 2013; Al-Omari et al. 2017; Yu and Siskind 2015).”
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “To eliminate the tedious effort of rule engineering when the problem scales, some recent methods train statistical models for lexical entries, either in a fully [10, 14, 24, 41] or weakly [36, 35, 57, 54] supervised fashion.”