Video Visual Relation Detection

By Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, Tat-seng Chua. 2017

187 citations recorded by the source corpus.

As a bridge to connect vision and language, visual relations between objects in the form of relation triplet $łangle subject,predicate,object\rangle$, such as "person-touch-dog'' and "cat-above-sofa'', provide a more comprehensive visual content understanding beyond objects. In this paper, we propose a novel vision task named Video Visual Relation Detection (VidVRD) to perform visual relation detection in videos instead of still images (ImgVRD). As compared to still images, videos provide a more natural set of features for detecting visual relations, such as the dynamic relations like "A-follow-B'' and "A-towards-B'', and temporally changing relations like "A-chase-B'' followed by "A-hold-B''. However, VidVRD is technically more challenging than ImgVRD due to the difficulties in accurate object tracking and diverse relation appearances in video domain. To this end, we propose a VidVRD method, which consists of object tracklet proposal, short-term relation prediction and greedy relational association. Moreover, we contribute the first dataset for VidVRD evaluation, which contains 1,000 videos with manually labeled visual relations, to validate our proposed method. On this dataset, our method achieves the best performance in comparison with the state-of-the-art baselines.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “To overcome this issue, other approaches create scene graphs from videos [13, 28, 33, 36, 45, 49].”
Video Relationship Reasoning Using Gated Spatio-Temporal Energy Graph (2019, methodology): “For ImageNet Videos, the relationship tagging task [29] focuses on only relationship prediction.”
Annotating Objects and Relations in User-Generated Videos (2019, methodology): “However, they are still primarily limited by the available datasets [19, 20] that are of small-scale and built upon video source under certain constraints.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “paradigm tends to predict pairwise relationships in an independent way [18], [48], [108], [112], while hub nodes (those that belong to multiple visual relationships at the”
Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context (2020, methodology): “Recently, the ﬁrst dataset on video visual relation detection (VidVRD) is introduced by [34].”
Zero-Shot Visual Recognition Using Semantics-Preserving Adversarial Embedding Networks (2017, unknown): “, visual attributes “striped tail” [13, 27, 67], classeme “fox-like” [57, 30, 65, 52], or “raccoon” word vectors [47, 37].”
Visual Relation Grounding in Videos (2020, methodology): “To achieve understandable visual analytics, many works have showed the importance of comprehending the interactions and relationships between objects [8,16,31].”