From Lifestyle Vlogs to Everyday Interactions

By David F. Fouhey, Weicheng Kuo, Alexei A. Efros, Jitendra Malik. 2017

132 citations recorded by the source corpus.

A major stumbling block to progress in understanding basic human interactions, such as getting out of bed or opening a refrigerator, is lack of good training data. Most past efforts have gathered this data explicitly: starting with a laundry list of action labels, and then querying search engines for videos tagged with each label. In this work, we do the reverse and search implicitly: we start with a large collection of interaction-rich video data and then annotate and analyze it. We use Internet Lifestyle Vlogs as the source of surprisingly large and diverse interaction data. We show that by collecting the data first, we are able to achieve greater scale and far greater diversity in terms of actions and actors. Additionally, our data exposes biases built into common explicitly gathered data. We make sense of our data by analyzing the central component of interaction - hands. We benchmark two tasks: identifying semantic object contact at the video level and non-semantic contact state at the frame level. We additionally demonstrate future prediction of hands.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “For MERLOT’s YT-Temporal-180M, the bulk of the video IDs were identified by applying breadth-first-search on YouTube channels from HowTo100M [84] and VLOG [36].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Our YT-Temporal-180M spans multiple domains, datasets, and topics – including instructional videos from HowTo100M [68], lifestyle vlogs of everyday events from the VLOG dataset [29], and YouTube’s auto-suggested videos for popular topics like ‘science’ or ‘home improvement.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “Rather than searching for predefined labels, which can introduce search engine bias [76, 16, 20], we collect images from movie scenes.”
Adversarial Filters of Dataset Biases (2020, background): “Such biases or artifacts 1 are often introduced during data collection (Fouhey et al., 2018) or during human annotation (Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Geva et al., 2019).”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, methodology): “This has been changing recently, with the release of the action classification benchmarks such as [10], [11], [12], [13], [14], [15].”
Moments in Time Dataset: One Million Videos for Event Understanding (2018, background): “and Charades [38] used crowdsourced workers to collect video datasets while the VLOG dataset [12] collects daily human activities with natural spatio-temporal context.”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, background): “, [8, 30, 31], uses strong supervision in the form of direct labels, including a lot of work that focuses on similar objectives [9, 11, 14].”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “We note that certain video domains such as narrated instructional videos [54, 79] and lifestyle vlogs [21, 34] are available in huge numbers (e.”
Understanding Human Hands in Contact at Internet Scale (2020, methodology): “We use this, plus videos from VLOG [14], to make a 100K image dataset annotated with our rich representation.”
Object Level Visual Reasoning in Videos (2018, background): “A concentrated eﬀort has been made to create large scale datasets to overcome these issues [13,12,23,14].”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, background): “As performances on trimmed video datasets advance [13, 4], recent focus of video understanding has shifted towards longer and untrimmed video data, such as VLOG [10], Charades [35], and EPIC-Kitchens [6].”
LifeQA: A Real-life Dataset for Video Question Answering (2020, background): “This is evident in recent video datasets, such as Charades (Sigurdsson et al., 2016) and VLOG (Fouhey et al., 2018), both of which include indoor scenes of human activities.”
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities (2020, background): “Crowd-sourced from online videos and movie sharing platforms, typical large-scale video datasets [53,29,8,9,15] focus on video-level summarization and classiﬁcation .”
Speech2Action: Cross-Modal Supervision for Action Recognition (2020, background): “Recently, a number of works have creatively identified certain domains of videos, such as narrated instructional videos [27, 38, 51] and lifestyle vlogs [11, 17] that are available in huge numbers (e.”
What Is More Likely to Happen Next? Video-and-Language Future Event Prediction (2020, methodology): “, 2019) or routine videos (Ignat et al., 2019; Fouhey et al., 2018), as they focus more on a single person performing actions, while we desire videos with richer multi-person interactions and dialogues.”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, background): “five third-person activity-recognition datasets [18], [19], [26], [27], [28] that focus on object-interaction activities.”
Condensed Movies: Story Based Retrieval with Contextual Embeddings (2020, background): “A number of different works have recently creatively identified that certain domains of videos, such as narrated instructional videos [6,7,8] and lifestyle vlogs [9,10] are available ar X iv :2 00 5.”
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries (2021, methodology): “Following [6, 19], we start with user-created lifestyle vlog videos on YouTube.”