Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings

By Michael Wray, Diane Larlus, Gabriella Csurka, Dima Damen. 2019

187 citations recorded by the source corpus.

We address the problem of cross-modal fine-grained action retrieval between text and video. Cross-modal retrieval is commonly achieved through learning a shared embedding space, that can indifferently embed modalities. In this paper, we propose to enrich the embedding by disentangling parts-of-speech (PoS) in the accompanying captions. We build a separate multi-modal embedding space for each PoS tag. The outputs of multiple PoS embeddings are then used as input to an integrated multi-modal space, where we perform action retrieval. All embeddings are trained jointly through a combination of PoS-aware and PoS-agnostic losses. Our proposal enables learning specialised embedding spaces that offer multiple views of the same embedded entities. We report the first retrieval results on fine-grained actions for the large-scale EPIC dataset, in a generalised zero-shot setting. Results show the advantage of our approach for both video-to-text and text-to-video action retrieval. We also demonstrate the benefit of disentangling the PoS for the generic task of cross-modal video retrieval on the MSR-VTT dataset.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [48, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [17, 19, 29, 30, 41, 53, 57, 63, 68, 84, 85, 87, 88, 93].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “We compare the NCE based approach (ours) to the frequently used max margin ranking loss [18, 20, 32, 38, 49, 53, 75, 76, 79, 80] and a binary classification loss (i.”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “Liwei et al. [38] and Michael et al. [40] have leveraged the triplet loss to learn the joint image-text embedding for the image-text retrieval and the video-text representation for the video-text action retrieval task, respectively.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “mAP has been used in other retrieval works [98,100,101,102], yet it only considers relevance between items to be binary.”
Multi-modal Transformer for Video Retrieval (2020, methodology): “sume that the video can be decomposed into clips that align with sentences of the caption. A recent alternative is creating separate embedding spaces for dierent parts of speech (e.g., noun or verb) [26]. In contrast to this method, we do not pre-process the sentences but encode them directly throug”
Support-set bottlenecks for video-text representation learning (2020, neutral): “0 − − − − JPoSE (Wray et al., 2019) 14 .”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “2 13 — — — — — JPoSE [35] MSR-VTT T→A+V 14.”
Open-book Video Captioning with Retrieve-Copy-Generate Network (2021, unknown): “In general, there are two kinds of model architectures, i.e ., Bi-encoders [4, 6, 19, 20] and Cross-encoders [26, 30, 33, 3].”
Audio Retrieval with Natural Language Queries (2021, background): “More closely related to our work, a number of methods showed that embedding video and text jointly into a shared space (such that their similarity can be computed efﬁciently) is an effective approach [1, 3, 2, 6, 35, 7].”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, background): “Another line of work learns ﬁne-grained or hierarchical alignment be-tween videos and texts [56, 49, 6].”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, methodology): “Our base model is a joint video-text embedding, as in [32, 52, 54].”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, unknown): “2 JPoSE (Wray et al., 2019) 14.”
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers (2021, methodology): “A large number of vision and language retrieval models [15, 19, 20, 36, 50, 51, 55, 59, 71, 72, 74, 75, 77] use a dual encoder architecture where the text and vision inputs are separately embedded into a joint space.”
HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, unknown): “[41] disentangled sentences into verbs and nouns for fine-grained video retrieval, and Chen et al.”
Egocentric Video-Language Pretraining (2022, methodology): “6, which is same as the baseline method JPoSE [43].”
Learning to Learn Words from Visual Scenes (2019, background): “Compositional models: Due to the diversity of the visual world, there has been extensive work in computer vision on learning compositional representations for objects and attributes [34,20,33,36] as well as for objects and actions [22,36,59].”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, background): “[39] disentangles action phrases into verbs and nouns for fine-grained action retrieval, which however is hard to apply on sentences with more complicated compositions.”