PointCLIP: Point Cloud Understanding by CLIP

By Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, Hongsheng Li. 2021

671 citations recorded by the source corpus.

Recently, zero-shot and few-shot learning via Contrastive Vision-Language Pre-training (CLIP) have shown inspirational performance on 2D visual recognition, which learns to match images with their corresponding texts in open-vocabulary settings. However, it remains under explored that whether CLIP, pre-trained by large-scale image-text pairs in 2D, can be generalized to 3D recognition. In this paper, we identify such a setting is feasible by proposing PointCLIP, which conducts alignment between CLIP-encoded point clouds and 3D category texts. Specifically, we encode a point cloud by projecting it onto multi-view depth maps and aggregate the view-wise zero-shot prediction in an end-to-end manner, which achieves efficient knowledge transfer from 2D to 3D. We further design an inter-view adapter to better extract the global feature and adaptively fuse the 3D few-shot knowledge into CLIP pre-trained in 2D. By just fine-tuning the adapter under few-shot settings, the performance of PointCLIP could be largely improved. In addition, we observe the knowledge complementary property between PointCLIP and classical 3D-supervised networks. Via simple ensemble during inference, PointCLIP contributes to favorable performance enhancement over state-of-the-art 3D networks. Therefore, PointCLIP is a promising alternative for effective 3D point cloud understanding under low data regime with marginal resource cost. We conduct thorough experiments on Model-NetlO, ModelNet40 and ScanObjectNN to demonstrate the effectiveness of PointCLIP. Code is available at https://github.com/ZrrSkywalker/PointCLIP.

Subfields: Multimodal Learning, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In 3D Vision & Pose

Citations in the corpus

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Adapters [22] insert lightweight adaption modules into each layer of the pre-trained transformer and have been extended across numerous domains [19, 18, 70, 71].”
Conditional Prompt Learning for Vision-Language Models (2022, background): “In computer vision, prompt learning is a nascent research direction that has only been explored very recently [27, 42, 56,58,62].”
Unsupervised Prompt Learning for Vision-Language Models (2022, methodology): “Tip-Adapter (Zhang et al. 2021a) further extends CLIP-Adapter by creating the weights by a key-value cache model constructed from the few-shot training set.”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, methodology): “Our method does not require any labeled 3D data for training, and it handles a broad range of queries supported by a large language model.”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, methodology): “The prior PointCLIP series [30, 90, 100] project 3D point clouds into depth maps, and utilize CLIP [59] for zero-shot recognition.”
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, background): “Learning to represent from unlabeled data without annotations, known as self-supervised learning, has attained great success in natural language processing [11, 38, 39, 5], computer vision [21, 8, 9, 20] and multi-modality learning [37, 61, 23].”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, neutral): “Many efforts also extend LLMs into more modalities, such as ImageBind-LLM (Han et al., 2023a), Point-LLM (Guo et al., 2023), and others (Zhu et al., 2023b; Zhang et al., 2022a; 2023b).”
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (2023, background): “For instance, CLIP4Clip (Luo et al., 2022) aligns video with text, CLAP (Wu* et al., 2023) aligns audio with text, and PointCLIP (Zhang et al., 2022) aligns point clouds with text.”
ImageBind-LLM: Multi-modality Instruction Tuning (2023, background): “AudioCLIP [32] and PointCLIP [33] respectively extend the embedding space of CLIP to other modalities, such as audio and 3D point clouds.”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, methodology): “For example, CoOp [61], CLIP-Adapter [16] and Tip-Adapter [58] use the pretrained CLIP for improving the few-shot transfer, while PointCLIP [59] and DenseCLIP [38,60] transfer the knowledge to point cloud understanding and dense prediction, respectively.”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “Therefore, feature adaptation has clear advantages in adapting VLMs to work on very different and complex downstream tasks [168], [169], [170], [171].”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “…a range of PEFT strategies have been proposed and implemented in VLA models, such as semantic segmentation [193], [205], [196], point cloud understanding [219], [241], [181], [81], video understanding [89], [138], [114], visual reasoning [63], [41], temporal action detection [137], to name a few.”
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification (2022, background): “The accuracy of vision tasks, such as classification [35,22,26,13,42,17,71], detection [51,5,73,65,70,9] and 3D understanding [47,69,64,68] has been boosted significantly thanks to better neural architecture designs [22,59] and delicately designed frameworks [51,37,5,72].”
Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, methodology): “Previous methods have also leveraged text embeddings in 3D learning for zero-shot pointcloud segmentation [34,8] and classification [56].”
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding (2022, methodology): “Following [59], zero-shot 3D classification is conducted by measuring distances between the 3D features of an object and the text features of category candidates.”
Foundation models in robotics: Applications, challenges, and the future (2023, positive): “PointCLIP [54] transfers CLIP’s pre-trained knowledge of 2D images to 3D point cloud understanding by aligning point clouds with text.”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, background): “…al., 2021; Jia et al., 2021) have shown powerful capability for various downstream tasks, including visual understanding (Gu et al., 2021; Wang et al., 2021a; Rao et al., 2022) , image-text generation (Patashnik et al., 2021; Mokady et al., 2021) and so on (Guzhov et al., 2022; Zhang et al., 2022).”
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, background): “Besides, many follow-up works [37, 45, 65, 74, 77, 80, 84] have also been proposed for further adapting CLIP to various vision tasks.”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, methodology): “For instance, PointCLIP [53] leverages depth image projections of point clouds for 3D recognition tasks with pre-trained 2D CLIP models.”