An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges
By Yuxin Peng, Xin Huang, Yunzhen Zhao. IEEE transactions on circuits and systems for video technology (Print), 2017
311 citations recorded by the source corpus.
Multimedia retrieval plays an indispensable role in big data utilization. Past efforts mainly focused on single-media retrieval. However, the requirements of users are highly flexible, such as retrieving the relevant audio clips with one query of image. So challenges stemming from the “media gap,” which means that representations of different media types are inconsistent, have attracted increasing attention. Cross-media retrieval is designed for the scenarios where the queries and retrieval results are of different media types. As a relatively new research topic, its concepts, methodologies, and benchmarks are still not clear in the literature. To address these issues, we review more than 100 references, give an overview including the concepts, methodologies, major challenges, and open issues, as well as build up the benchmarks, including data sets and experimental results. Researchers can directly adopt the benchmarks to promptly evaluate their proposed methods. This will help them to focus on algorithm design, rather than the time-consuming compared methods and results. It is noted that we have constructed a new data set XMedia, which is the first publicly available data set with up to five media types (text, image, video, audio, and 3-D model). We believe this overview will attract more researchers to focus on cross-media retrieval and be helpful to them.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/50233ac68c2aa139fabe71e2844c2734adf39a3d
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- Cross-Modality Person Re-Identification with Generative Adversarial Training (2018, methodology): “[Peng et al. , 2017b] proposed a method to effectively correlate large-scale heterogeneous data by GAN.”
- COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval (2018, background): “While much progress has been made for image auto-tagging [23]–[25] and caption retrieval [26]–[28], whether these techniques can be used to assist dataset construction is unexplored.”
- Triplet-Based Deep Hashing Network for Cross-Modal Retrieval (2018, background): “Cross-modal retrieval is designed for scenarios where the queries and retrieval results are from different modalities [1], [2].”
- Cross Modal Audio Search and Retrieval with Joint Embeddings Based on Text and Audio (2019, background): “Cross-modal search and retrieval is an approach that enables matching of text-audio or audio-text pairs, which can compensate for missing or ambiguous information in either modality [5].”
- M5Product: A Multi-modal Pretraining Benchmark for E-commercial Product Downstream Tasks (2021, methodology): “0 [29] 204,721 - - 2 image/text no RPC checkout [5] 30,000 200 367,935 2 image/text no Twitter100k [30] 100,000 - - 2 image/text no INRIA-Websearch [31] 71,478 353 - 2 image/text no NUS-WIDE [32] 269,648 81 - 2 image/text no Open Image [33] 1,670,000 - - 2 image/text no CMU-MOSEI [34] 23,500 2 - 3 t”
- Scalable Deep Multimodal Learning for Cross-Modal Retrieval (2019, background): “Over the past decades, a large number of cross-modal retrieval methods [26] have been developed to eliminate the heterogeneity gap and learn the common representation by different learning models.”
- Separate Anything You Describe (2023, background): “audio sources which could be useful in many applications such as automatic audio editing [10] and multimedia content retrieval [11].”