Image Reconstruction from Bag-of-Visual-Words

By Hiroharu Kato, Tatsuya Harada. 2014

86 citations recorded by the source corpus.

The objective of this study is to reconstruct images from Bag-of-Visual-Words (BoVW), which is the de facto standard feature for image retrieval and recognition. BoVW is defined here as a histogram of quantized descriptors extracted densely on a regular grid at a single scale. Despite its wide use, no report describes reconstruction of the original image of a BoVW. This task is challenging for two reasons: 1) BoVW includes quantization errors when local descriptors are assigned to visual words. 2) BoVW lacks spatial information of local descriptors when we count the occurrence of visual words. To tackle this difficult task, we use a large-scale image database to estimate the spatial arrangement of local descriptors. Then this task creates a jigsaw puzzle problem with adjacency and global location costs of visual words. Solving this optimization problem is also challenging because it is known as an NP-Hard problem. We propose a heuristic but efficient method to optimize it. To underscore the effectiveness of our method, we apply it to BoVWs extracted from about 100 different categories and demonstrate that it can reconstruct the original images, although the image features lack spatial information and include quantization errors.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Methods for Explaining Black Box Models (2018, background): “elated to producing a certain prediction by realizing a sort of articial image prototype. There is a signicant amount of work in understanding the representation of DNN by means of artifact images, [41,100,105,111]. We conclude this section presenting how [25] deals with artifacts in DNNs. Finding a”
Visualizing Deep Convolutional Neural Networks Using Natural Pre-images (2015, methodology): “[30] looked at the inversion problem for HOG, SIFT, and recent CNNs; our method differs significantly from the ones above as it addresses many different representations using the same energy minimization framework and optimization algorithm.”
Inverting Visual Representations with Convolutional Networks (2015, background): “There has been research on inverting various traditional computer vision representations: HOG and dense SIFT [24], keypoint-based SIFT [26], Local Binary De-scriptors [4], Bag-of-Visual-Words [11].”
Learning visual biases from human imagination (2014, methodology): “m, but in the same feature spaces as computer vision systems. Our new method, which we refer to as imaginary classiﬁers, uses two key ideas. Firstly, we captialize on recent work in feature inversion [38,39,8,21]. Rather than generating noise in pixel space, we generate noise in feature space. We th”
Privacy Preserving Image Queries for Camera Localization (2019, methodology): “Subse-quently, others tried inverting and interpreting HOG features [70], bag of visual words features [29], and CNN features [41, 77, 78].”
Understanding deep image representations by inverting them (2014, background): “[13] and Local Binary Descriptors by d’Angelo et al.”
Inverting Convolutional Networks with Convolutional Networks (2015, methodology): “There has been research on inverting various traditional computer vision representations: HOG [8], DSIFT [9], Local Binary Descriptors [10], Bag-of-Visual-Words [11].”
Salient Deconvolutional Networks (2016, methodology): “Some recent ones such as [24] build on the idea of natural (regularized) pre-images introduced in [13], which in turn are based on prior contributions that applied pre-images to representations such as HOG [21], SIFT [22], BoVW [2,7], as well as early neural networks [23,10,9,12,6,20].”
A Methodology for Formalizing Model-Inversion Attacks (2016, background): “Interestingly, we note that these attacks are reminiscent of privacy attacks discussed in the context of inverting highly compressed image features, which were explored previously [8], [9], [10].”