HOTR: End-to-End Human-Object Interaction Detection with Transformers

By Bumsoo Kim, Junhyun Lee, Eun-Sol Kim, Hyunwoo J. Kim. 2021

345 citations recorded by the source corpus.

Human-Object Interaction (HOI) detection is a task of identifying "a set of interactions" in an image, which involves the i) localization of the subject (i.e., humans) and target (i.e., objects) of interaction, and ii) the classification of the interaction labels. Most existing methods have indirectly addressed this task by detecting human and object instances and individually inferring every pair of the detected instances. In this paper, we present a novel framework, referred by HOTR, which directly predicts a set of 〈human, object, interaction〉 triplets from an image based on a transformer encoder-decoder architecture. Through the set prediction, our method effectively exploits the inherent semantic relationships in an image and does not require time-consuming post-processing which is the main bottleneck of existing methods. Our proposed algorithm achieves the state-of-the-art performance in two HOI detection benchmarks with an inference time under 1 ms after object detection.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

EDTER: Edge Detection with Transformer (2022, methodology): “Recently, it is applied in conjunction with CNN in DETR [7] and the other variants [10, 27, 31, 58, 73].”
Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding (2021, background): “2021), segmentation and detection (Wang et al. 2021a; Liang et al. 2020; Kim et al. 2021), interpretability (Chefer, Gur, and Wolf 2021; Abnar and Zuidema 2020), a deeper understanding of the data efficiency and training difficulties from the architectural perspective is of significant impact.”
Webly Supervised Concept Expansion for General Purpose Vision Models (2022, background): “Although recent models [36, 89, 94] show results up to 32.”
Aggregating Nested Transformers (2021, background): “ViT is given high expectation to improve vision tasks beyond image classiﬁcation, with existing studies on generative modeling [11, 27, 28, 39], video understanding [2, 38], segmentation and detection [30, 34, 53], interpretability [1, 6, 7].”