MediaPipe: A Framework for Building Perception Pipelines

By Camillo Lugaresi, Jiuqiang Tang, H. Nash, C. McClanahan, Esha Uboweja, Michael P. Hays, Fan Zhang, Chuo-Ling Chang, M. Yong, Juhyun Lee and 4 more. arXiv.org, 2019

2,263 citations recorded by the source corpus.

Building applications that perceive the world around them is challenging. A developer needs to (a) select and develop corresponding machine learning algorithms and models, (b) build a series of prototypes and demos, (c) balance resource consumption against the quality of the solutions, and finally (d) identify and mitigate problematic cases. The MediaPipe framework addresses all of these challenges. A developer can use MediaPipe to build prototypes by combining existing perception components, to advance them to polished cross-platform applications and measure system performance and resource consumption on target platforms. We show that these features enable a developer to focus on the algorithm or model development and use MediaPipe as an environment for iteratively improving their application with results reproducible across different devices and platforms. MediaPipe will be open-sourced at this https URL.

Subfields: Electrical and Electronic Engineering, Hardware and Architecture

View Paper: https://www.semanticscholar.org/paper/1c44a3cde07f3f2ffd8880b441449483ed74fc5f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

1. Jeannette Bohg
2. Ailing Zeng

In Hardware and Architecture

1. Ricardo Martin-Brualla
2. S. M. Seitz
3. Jonathan T. Barron
4. Ailing Zeng

Citations in the corpus

• Effective Whole-body Pose Estimation with Two-stages Distillation (2023, background): “We also compare our state-of-the-art model with two widely used models OpenPose [2] and MediaPipe [29, 62], as presented in Fig.”
• MediaPipe Hands: On-device Real-time Hand Tracking (2020, methodology): “With MediaPipe[12], our hand tracking pipeline can be built as a directed graph of modular components, called Cal-culators.”
• Nerfies: Deformable Neural Radiance Fields (2020, methodology): “approximate metric units using estimated facial landmarks from MediaPipe’s face mesh [31] and assuming an average interpupillary distance of 6cm.”
• Grounding Predicates through Actions (2021, methodology): “YOLOv5 object detector [36] for predicate bounding boxes along with Mediapipe object tracking [37] for smoother tracking between frames.”