Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature

By Babak Saleh, Ahmed M. Elgammal. 2015

382 citations recorded by the source corpus.

In the past few years, the number of fine-art collections that are digitized and publicly available has been growing rapidly. With the availability of such large collections of digitized artworks comes the need to develop multimedia systems to archive and retrieve this pool of data. Measuring the visual similarity between artistic items is an essential step for such multimedia systems, which can benefit more high-level multimedia tasks. In order to model this similarity between paintings, we should extract the appropriate visual features for paintings and find out the best approach to learn the similarity metric based on these features. We investigate a comprehensive list of visual features and metric learning approaches to learn an optimized similarity measure between paintings. We develop a machine that is able to make aesthetic-related semantic-level judgments, such as predicting a painting's style, genre, and artist, as well as providing similarity measures optimized based on the knowledge available in the domain of art historical interpretation. Our experiments show the value of using this similarity measure for the aforementioned prediction tasks.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Meta-attention for ViT-backed Continual Learning (2022, methodology): “To validate the superiority of the proposed method, extensive experiments, including benchmark comparison and ablation study, are conducted on a diverse set of image classification benchmarks (including ImageNet [8], CUB [47], Stanford Cars [24], FGVC-Aircraft [29], CIFAR-100 [25], Sketches [11], Wi”
Compacting, Picking and Growing for Unforgetting Continual Learning (2019, methodology): “The statistics are summarized in Table 4, where ImageNet [17] is the ﬁrst task, following by ﬁne-grained classiﬁcation tasks, CUBS [44], Stanford Cars [15] and Flowers [26], and ﬁnally WikiArt [39] and Sketch [8] that are artiﬁcial images drawing in various styles and objects.”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “To better understand the fine-tuning policies learned by the policy network, we visualize them on CUBS, Flowers, WikiArt, Sketches, and Stanford Cars in Figure 3.”
Adding New Tasks to a Single Network with Weight Trasformations using Binary Masks (2018, methodology): “WikiArt [48] contains painting from 195 diﬀerent artists.”
Piggyback: Adding Multiple Tasks to a Single, Fixed Network by Learning to Mask (2018, methodology): “Stanford cars [17], Oxford ﬂowers [25], as well datasets with a signiﬁcant departure from the natural image domain of the ImageNet dataset such as WikiArt paintings [31] and human sketches [4].”
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, neutral): “The templates for tasks VQAv2, S-VQA, WikiART, and RefCOCO are presented in Table 13.”
Provable and Efficient Continual Representation Learning (2022, methodology): “2012)) is the ﬁrst task, followed by CUBS (Wah et al. (2011)), Stanford Cars (Krause et al. (2013)), Flowers (Nilsback and Zisserman (2008)), WikiArt (Saleh and Elgammal (2015)) and Sketch (Eitz et al. (2012)).”
Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights (2018, methodology): “WikiArt [11] 42,129 10,628 195 Sketch [12] 16,000 4,000 250”
Transfer Learning with Style Transfer between the Photorealistic and Artistic Domain (2021, methodology): “They outperformed most of the hand-crafted features or were used in combination with them [32, 33, 34, 35].”
Sound-Guided Semantic Image Manipulation (2021, methodology): “We employ StyleGAN2 [23] generator which is pre-trained with the fine-art paintings dataset called WikiArt [40].”
NetTailor: Tuning the Architecture, Not Just the Weights (2019, methodology): “The first protocol described in [38] consists of five datasets: CUB [67], Stanford Cars [25], Oxford Flowers [46], WikiArt [56] and Sketch [10].”
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions (2023, background): “‘LCS’ abbreviates the LAION [48], CC [50], and SBU [47] datasets.”
Machine Learning for Cultural Heritage: A Survey (2020, positive): “Recent contributions to digital work analysis are the study of imilarity metric learning methods for making aesthetic-related emantic-level judgements, such as predicting the painting’s style, enre, and artist [37] ; the detection of fake artworks by stroke nalysis [17] and the artistic style transf”
GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models (2023, background): “, van Gogh, Monet) from the WikiArt dataset [75].”
Helpful or Harmful: Inter-task Association in Continual Learning (2022, methodology): “The datasets used in the task-incremental scenarios include ImageNet, diverse fine-grained datasets, CIFAR-10, and MNIST, where the fine-grained datasets are CUBS [39], Stanford Cars [18], Flowers [29], Wikiart [34], and Sketch [11].”