Multi-task deep visual-semantic embedding for video thumbnail selection

By Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, Jiebo Luo. 2015

255 citations recorded by the source corpus.

Given the tremendous growth of online videos, video thumbnail, as the common visualization form of video content, is becoming increasingly important to influence user's browsing and searching experience. However, conventional methods for video thumbnail selection often fail to produce satisfying results as they ignore the side semantic information (e.g., title, description, and query) associated with the video. As a result, the selected thumbnail cannot always represent video semantics and the click-through rate is adversely affected even when the retrieved videos are relevant. In this paper, we have developed a multi-task deep visual-semantic embedding model, which can automatically select query-dependent video thumbnails according to both visual and side information. Different from most existing methods, the proposed approach employs the deep visual-semantic embedding model to directly compute the similarity between the query and video thumbnails by mapping them into a common latent semantic space, where even unseen query-thumbnail pairs can be correctly matched. In particular, we train the embedding model by exploring the large-scale and freely accessible click-through video and image data, as well as employing a multi-task learning strategy to holistically exploit the query-thumbnail relevance from these two highly related datasets. Finally, a thumbnail is selected by fusing both the representative and query relevance scores. The evaluations on 1,000 query-thumbnail dataset labeled by 191 workers in Amazon Mechanical Turk have demonstrated the effectiveness of our proposed method.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Dense-Captioning Events in Videos (2017, methodology): “Meanwhile, others have utilized text inputs from user studies to guide the selection process [34, 23].”
Localizing Moments in Video with Natural Language (2017, methodology): “Though most summarization algorithms do not include textual input ([3, 9, 10, 52, 53]), some use text in the form of video titles [21, 41] or user queries in the form of category labels to guide content selection [37].”
A Survey on Multi-Task Learning (2017, methodology): “In [97], in order to select thumbnails for videos, a deep model is proposed to utilize the available sematic information such as titles and descriptions to align the embeddings of video thumbnails to those of sematic information and the auxiliary task is the click-through image data.”
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries (2021, methodology): “Similar to [22], we define a clip as positive if it has a score of ‘Very Good’.”
Person Search with Natural Language Description (2017, background): “Visual-semantic embedding methods [6, 18, 30, 25, 32] learned to embed both language and images into a common space for image classiﬁcation and retrieval.”
A Deep Learning-Based Approach to Progressive Vehicle Re-identification for Urban Surveillance (2016, background): “In recent years, deep neural networks have achieved great success in computer vision such as object classification [13], detection [14], image understanding [15], video analysis [16], and multimedia search [17].”
Large-scale vehicle re-identification in urban surveillance videos (2016, background): “Recently, deep Convolutional Neural Network (CNN) has achieved excellent performance in many computer vision tasks [14–17].”
Enhancing Video Summarization via Vision-Language Embedding (2017, background): “Joint modeling of visual content and text is becoming increasingly common for video summarization and retrieval, typically to help identify whether a given shot is relevant to the overall story of a video or a particular user query [23, 32, 34, 42].”
Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (2021, background): “Recently, multitask learning is wildly applied in MSA (Liu et al. 2015; Zhang et al. 2016; Akhtar et al. 2019; Yu et al. 2020b).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Initially, most of the work was concentrated on combining low-level linguistic units such as words with images or videos for building visual-semantic embeddings (Frome et al., 2013; Kiros et al., 2014b; Liu et al., 2015; Cao et al., 2016; Tsai et al., 2017; Guo et al., 2018; Wang et al., 2019) usefu”