Audio Visual Attribute Discovery for Fine-Grained Object Recognition
By Hua Zhang, Xiaochun Cao, Rui Wang. 2018
26 citations recorded by the source corpus.
Current progresses on fine-grained recognition are mainly focus on learning the discriminative feature representation via introducing the visual supervisions e.g. part labels. However, it is time-consuming and needs the professional knowledge to obtain the accuracy annotations. Different from these existing methods based on the visual supervisions, in this paper, we introduce a novel feature named audio visual attributes via discovering the correlations between the visual and audio representations. Specifically, our unified framework is training with video-level category label, which consists of two important modules, the encoder module and the attribute discovery module, to encode the image and audio into vectors and learn the correlations between audio and images, respectively. On the encoder module, we present two types of feed forward convolutional neural network for the image and audio modalities. While an attention driven framework based on recurrent neural network is developed to generate the audio visual attribute representation. Thus, our proposed architecture can be implemented end-to-end in the step of inference. We exploit our models for the problem of fine-grained bird recognition on the CUB200-211 benchmark. The experimental results demonstrate that with the help of audio visual attribute, we achieve the superior or comparable performance to that of strongly supervised approaches on the bird recognition.
Subfields: Computer Vision and Pattern Recognition, Developmental Biology, Multimodal Learning, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Kristen Grauman
- Luc Van Gool
- Jitendra Malik
- Trevor Darrell
- Andrew Zisserman
- Cordelia Schmid
- Dacheng Tao
- Ming-Hsuan Yang
In Developmental Biology
- Lorenzo Torresani
- Kristen Grauman
- Antonio Torralba
- Christoph Feichtenhofer
- Vamsi Krishna Ithapu
- Bernard Ghanem
- Anurag Kumar
In Multimodal Learning
- Kristen Grauman
- Lorenzo Torresani
- Dhruv Batra
- Antonio Torralba
- Rohit Girdhar
- Christoph Feichtenhofer
- Satwik Kottur
- Devi Parikh
In Signal Processing
- Kristen Grauman
- Jitendra Malik
- Lorenzo Torresani
- Antonio Torralba
- David J. Crandall
- Christoph Feichtenhofer
- Tushar Nagarajan
- Rohit Girdhar
Citations in the corpus
- Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Relation to existing tasks The past few years have seen audio studied in computer vision tasks [245] for action classification [110,226], object categorization [125,234], source localization and tracking [14, 197, 212] and embodied navigation [33].”
- Fine-Grained Image Analysis With Deep Learning: A Survey (2021, unknown): “[157] AAAI 2018 BBox BBox Audio VGG-16 227× 227 86.”