Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification

By Xiang Bai, Mingkun Yang, Pengyuan Lyu, Yongchao Xu, Jiebo Luo. 2017

79 citations recorded by the source corpus.

Text in natural images contains rich semantics that is often highly relevant to objects or scene. In this paper, we focus on the problem of fully exploiting scene text for visual understanding. The main idea is combining word representations and deep visual features in a globally trainable deep convolutional neural network. First, the recognized words are obtained by a scene text reading system. Next, we combine the word embedding of the recognized words and the deep visual features into a single representation that is optimized by a convolutional neural network for fine-grained image classification. In our framework, the attention mechanism is adopted to compute the relevance between each recognized word and the given image, which further enhances the recognition performance. We have performed experiments on two datasets: con-text dataset and drink bottle dataset, which are proposed for fine-grained classification of business places and drink bottles, respectively. The experimental results consistently demonstrate that the proposed method of combining textual and visual cues significantly outperforms classification with only visual representation. Moreover, we have shown that the learned representation improves the retrieval performance on the drink bottle images by a large margin, making it potentially powerful in product search.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Cong Yao
2. Xiang Bai
3. Minghui Liao
4. Pengyuan Lyu
5. Mingkun Yang
6. Xue Yang
7. Yongchao Xu
8. Junchi Yan

Citations in the corpus

• All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting (2019, background): “Automatic reading text from natural images has attracted great attention due to its wide practical applications such as office automation, network content security, intelligent transportation system (Zhu et al. 2018; Rong, Yi, and Tian 2016), geo-location, and visual search (Bai et al. 2018).”
• Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, background): “R EADING text from images/videos is of great values for plentiful real-world applications such as image recognition/retrieval [4], geo-location, office automation, and assistance for the blind, as scene text contains quite useful semantics for understanding the world.”
• Arbitrary-Oriented Scene Text Detection via Rotation Proposals (2017, background): “dia tasks, such as visual classification [1], [2], video analysis [3], [4] and mobile applications [5].”