EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset

By Curtis G. Northcutt, Shengxin Zha, Steven J. Lovegrove, Richard A. Newcombe. 2020

44 citations recorded by the source corpus.

Multi-modal datasets in artificial intelligence (AI) often capture a third-person perspective, but our embodied human intelligence evolved with sensory input from the egocentric, first-person perspective. Towards embodied AI, we introduce the Egocentric Communications (EgoCom) dataset to advance the state-of-the-art in conversational AI, natural language, audio speech analysis, computer vision, and machine learning. EgoCom is a first-of-its-kind natural conversations dataset containing multi-modal human communication data captured simultaneously from the participants’ egocentric perspectives. EgoCom includes 38.5 hours of synchronized embodied stereo audio, egocentric video with 240,000 ground-truth, time-stamped word-level transcriptions and speaker labels from 34 diverse speakers. We study baseline performance on two novel applications that benefit from embodied data: (1) predicting turn-taking in conversations and (2) multi-speaker transcription. For (1), we investigate Bayesian baselines to predict turn-taking within 5 percent of human performance. For (2), we use simultaneous egocentric capture to combine Google speech-to-text outputs, improving global transcription by 79 percent relative to a single perspective. Both applications exploit EgoCom's synchronous multi-perspective data to augment performance of embodied AI tasks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Lu Jiang
2. Hao Jiang
3. Maja Pantic
4. Jacob Donley
5. Vamsi Krishna Ithapu
6. Curtis G. Northcutt
7. Jie Shen
8. Isaac L. Chuang

In Multimodal Learning

1. Maja Pantic
2. Curtis G. Northcutt
3. Lu Jiang

In Evaluation & Benchmarks

1. Curtis G. Northcutt
2. Maja Pantic
3. Vamsi Krishna Ithapu
4. Lu Jiang
5. Isaac L. Chuang
6. D. Malerba
7. Vincenzo Pasquadibisceglie

In Signal Processing

1. Maja Pantic
2. Jacob Donley
3. Vamsi Krishna Ithapu
4. Curtis G. Northcutt
5. Lu Jiang
6. Isaac L. Chuang

Citations in the corpus

• EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments (2021, methodology): “The EgoCom dataset [6], released in 2020, contains egocentric video and two channel audio recordings of conversations from pairs of glasses.”
• Confident Learning: Estimating Uncertainty in Dataset Labels (2019, background): “, 2019), the multi-modal Egocentric Communications (EgoCom) benchmark (Northcutt et al., 2020), and the realistic noisy label benchmark CNWL (Jiang et al.”