Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

By Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, C. Chen, S. Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik and 5 more. Neural Information Processing Systems, 2023

303 citations recorded by the source corpus.

We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 4.3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/326f6a8011e43322c433751b9cc31fd56564621c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, unknown): “Some approaches incorporate VLMs [3, 44, 58] into end-to-end trainable policies by using them as pretrained backbone for better visuo-language representation [10, 26, 38, 43, 53].”
π0.5: a Vision-Language-Action Model with Open-World Generalization (2025, unknown): “Prior methods have explored initializing vision encoders from computer vision datasets [85, 58, 57, 18], or leveraging off-the-shelf task planners [38, 48, 73, 81].”
Octo: An Open-Source Generalist Robot Policy (2024, methodology): “However, pretrained visual representations such as VC-1 [56] have been used in this way, and therefore we use these methods as another point of comparison.”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Most works focus on pre-trained visual representations that can be used to initialize the encoder of the robot’s camera observations, either via supervised ImageNet classification [53], data augmentation [54, 55, 56, 57] or objectives that are tailored towards robotic control [58, 59, 60, 61, 62].”
HumanPlus: Humanoid Shadowing and Imitation from Humans (2024, methodology): “Human data has been used extensively for robot learning, including for pre-training visual or intermediate representations or tasks [8, 28, 50, 53, 66, 79, 86, 99] leveraging Internet-scale data [18, 26, 27], and for imitation learning on in-domain human data [3, 12, 43, 47”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, unknown): “…and vision foundation models to enhance robotic capabilities, utilizing them as pretrained visual representations that accelerate robotic policy learning [29, 32, 30, 19, 31], for object localization in robotics tasks [9, 45], and for high-level planning and reasoning [1, 17, 42, 16, 43, 18, 6].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “While their own experiments demonstrate superior performance compared to R3M in specific tasks, subsequent research presents conflicting findings through more comprehensive evaluations [47].”