HARMONIC: A multimodal dataset of assistive human–robot collaboration

By Benjamin A. Newman, Reuben M. Aronson, Siddhartha S. Srinivasa, Kris M. Kitani, Henny Admoni. 2018

58 citations recorded by the source corpus.

We present the Human And Robot Multimodal Observations of Natural Interactive Collaboration (HARMONIC) dataset. This is a large multimodal dataset of human interactions with a robotic arm in a shared autonomy setting designed to imitate assistive eating. The dataset provides human, robot, and environmental data views of 24 different people engaged in an assistive eating task with a 6-degree-of-freedom (6-DOF) robot arm. From each participant, we recorded video of both eyes, egocentric video from a head-mounted camera, joystick commands, electromyography from the forearm used to operate the joystick, third-person stereo video, and the joint positions of the 6-DOF robot arm. Also included are several features that come as a direct result of these recordings, such as eye gaze projected onto the egocentric video, body pose, hand pose, and facial keypoints. These data streams were collected specifically because they have been shown to be closely related to human mental states and intention. This dataset could be of interest to researchers studying intention prediction, human mental state modeling, and shared autonomy. Data streams are provided in a variety of formats such as video and human-readable CSV and YAML files.

Subfields: Rehabilitation, Social Psychology, Multimodal Learning, Human-Computer Interaction, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Rehabilitation

In Social Psychology

In Multimodal Learning

In Human-Computer Interaction

In Robotics & Control

Citations in the corpus

LILA: Language-Informed Latent Actions (2021, background): “Shared autonomy casts robot control as a collaborative process between humans and robots [2, 4, 16, 22].”
Shared Autonomy with Learned Latent Actions (2020, background): “The robot’s overall behavior combines this input action and ar ∈ A, the robot’s assistive guidance [8, 23]:”
Controlling Assistive Robots with Learned Latent Actions (2019, methodology): “with two 2-DoF joysticks, and were given a button to toggle between linear and angular motion [3], [7], [25].”
Learning latent actions to control assistive robots (2021, unknown): “Similar to recent work on shared autonomy [24,16,37], we deﬁne the robot’s overall action as the linear combination of a h (the human’s commanded action) and a r (the robot’s autonomous guidance): In the above, α ∈ [0 , 1] parameterizes the trade-oﬀ be-tween direct human teleoperation ( α = 0) and…”