ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

By Le Xue, Mingfei Gao, Chen Xing, Roberto Mart'in-Mart'in, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Silvio Savarese. Computer Vision and Pattern Recognition, 2022

401 citations recorded by the source corpus.

The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of image, text, and 3D point cloud by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models will be released.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ecb00499e4d41861b269337b743d19e3f588ddcf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Dahua Lin
2. Sai Bi
3. Tai Wang
4. Yilun Chen
5. Kalyan Sunkavalli
6. Jiangmiao Pang

In Computer Vision and Pattern Recognition

1. Jiangmiao Pang
2. Dahua Lin
3. Kalyan Sunkavalli
4. Tai Wang
5. Yilun Chen
6. Sai Bi
7. Luc Van Gool
8. Runsen Xu

Citations in the corpus

• Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, positive): “To evaluate shape generative performance, we employ ULIP [105] and Uni3D [122] to compute the similarity between the generated mesh and input images (ULIP-I and Uni3D-I) and the similarity between the generated mesh and images prompt synthesizing by the vision language model [13] (ULIP-T and…”
• PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “ULIP [49] and ULIP-2 [50] take a more direct approach by training a point cloud encoder to align with CLIP representations using point cloud, image, and text triplets.”
• LRM: Large Reconstruction Model for Single Image to 3D (2023, positive): “There are early attempts in this direction that minimize the difference between encoded image and 3D representations (Girdhar et al., 2016; Mandikal et al., 2018), as well as recent research, ULIP (Xue et al., 2023) and CLIP 2 (Zeng et al., 2023), which bridges 3D, language, and images via contrasti”