A Domain Based Approach to Social Relation Recognition

By Qianru Sun, Bernt Schiele, Mario Fritz. 2017

117 citations recorded by the source corpus.

Social relations are the foundation of human daily life. Developing techniques to analyze such relations from visual data bears great potential to build machines that better understand us and are capable of interacting with us at a social level. Previous investigations have remained partial due to the overwhelming diversity and complexity of the topic and consequently have only focused on a handful of social relations. In this paper, we argue that the domain-based theory from social psychology is a great starting point to systematically approach this problem. The theory provides coverage of all aspects of social relations and equally is concrete and predictive about the visual attributes and behaviors defining the relations included in each domain. We provide the first dataset built on this holistic conceptualization of social life that is composed of a hierarchical label space of social domains and social relations. We also contribute the first models to recognize such domains and relations and find superior performance for attribute based features. Beyond the encouraging performance of the attribute based approach, we also find interpretable features that are in accordance with the predictions from social psychology literature. Beyond our findings, we believe that our contributions more tightly interleave visual recognition and social psychology theory that has the potential to complement the theoretical work in the area with empirical and data-driven models of social life.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ALICE: Active Learning with Contrastive Natural Language Explanations (2020, methodology): “Dataset We also evaluate ALICE on the People in Photo Album Relation dataset (Zhang et al., 2015; Sun et al., 2017).”
GAN Memory with No Forgetting (2020, background): “Transfer learning Being general and effective, transfer learning has attracted increasing attention recently in various research fields, with a focus on discriminative tasks like classification [6, 70, 90, 93, 16, 24, 44, 46, 74, 92] and those in natural language processing [3, 63, 53, 52, 2].”
Meta-Transfer Learning Through Hard Tasks (2019, methodology): “the word embedding models used for zero-shot learning [14], [55] and trained attribute models used for social relationship recognition [50].”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “lel. Our approach makes use of the ensemble of training epochs to obtain different models. Ensembling models in a temporal way [28] and utilizing features extracted by an ensemble of attribute models [56] are also related works. Comparing to them, our difference lies in that our multiple models are ”
Adversarial Image Perturbation for Privacy Protection A Game Theory Perspective (2017, background): “Together with recent developments in computer vision techniques [4, 11, 8, 27, 34], this results in increasing concerns that malicious entities employing computer vision technologies could extract private information from visual data.”
When Machine Learning Meets Privacy (2020, background): “Moreover, a series of research work have demonstrated how the advanced artificial neural networks can be used as an adversarial tool to detect sensitive information in images, including people’s age [63], relationship [150] and vehicle license plates [181] from ordinary or even obfuscated images.”
Deep Reasoning with Knowledge Graph for Social Relationship Understanding (2018, methodology): “3) We conduct extensive experiments on the large-scale People in Social Con-text (PISC) [ Zhang et al. , 2015 ] and the People in Photo Al-bum Relation (PIPA-Relation) [ Sun et al. , 2017 ] datasets and demonstrate the superiority of our methods over the existing state-of-the-art methods.”
Disentangled Person Image Generation (2017, background): “Since images themselves contain abundant context information [32], some works have tried to tackle the problem in an unsupervised way.”
Boosting Image-based Mutual Gaze Detection using Pseudo 3D Gaze (2020, unknown): “With recent advances in the ﬁeld of computer vision, social relation discovery has received signiﬁcant attention from the research community (Ding and Yilmaz 2010, 2011; Liu et al. 2019; Sapru and Bourlard 2015; Sun, Schiele, and Fritz 2017; Wang et al. 2010; Yu et al. 2009).”
Natural and Effective Obfuscation by Head Inpainting (2017, methodology): “Generated image For every training and testing sample, we prepare the input by ﬁrst obtaining the body crop , following the procedure in [16, 25]: extend the head box with ﬁxed ratios ( 3 × width and 6 × height), and then re-size and zero-pad the body crop such that it ﬁts tightly in the square 256…”