A measure of intelligence

By A. Tate. 2012

1,044 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “Or, as, Francois Chollet (Chollet, 2019) recently put it,”
Large Language Models are Zero-Shot Reasoners (2022, background): “In contrast, the versatility of this single prompt across diverse reasoning tasks hints at untapped and understudied zero-shot fundamental capabilities of LLMs, such as higher-level broad cognitive capabilities like generic logical reasoning [Chollet, 2019].”
Imitating Interactive Intelligence (2020, background): “As the capability to interact is, largely, commensurate with psychological notions of intelligence (Duncan, 2010), evaluating interactions is perhaps as hard as evaluating intelligence (Turing, 1950; Chollet, 2019).”
The MAGICAL Benchmark for Robust Imitation (2020, methodology): “Although there is no sequential decision-making aspect to ARC, Chollet [11] claims that solving the corpus may still require priors for “objectness”, goal-directedness, and various geometric concepts, which means that methods suitable for solving MAGICAL may also be useful on ARC, and vice versa.”
Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “One encouraging prior work is Acquaviva et al. (2022), showing that humans, when asked to explain how to solve problems in the Abstract Reasoning Corpus (Chollet, 2019), tended to provide step-by-step hierarchical descriptions with many veriﬁ-cation steps.”
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024, neutral): “Logical reasoning is a critical trait of intelligent systems, and building intelligent systems capable of such reasoning has long been an important goal in artificial intelligence (Newell & Simon, 1956; Marcus, 2003; Brachman & Levesque, 2004; Legg & Hutter, 2007; Pearl, 2014; Chollet, 2019).”
A large-scale benchmark for few-shot program induction and synthesis (2021, methodology): “…few-shot program induction datasets, such as those used in FlashFill (Gulwani, 2011; Gulwani et al., 2015) and Dream-Coder (Ellis et al., 2020), as well as the Abstract Reasoning Challenge(ARC) (Chollet, 2019), a list functions benchmark (Rule, 2020), or the SyGus competition (Alur et al., 2017).”
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2022, background): “An important topic on benchmarking in the era of large pre-trained language models is to quantify the level of prior knowledge the LLM has gained about the end task being evaluated, which is crucial for assessing how well can the model truly extrapolate from pretraining and acquire new skills (Choll”
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback (2022, background): “Humanmodel conversations – which are never seen at training time – can have a quite different distribution to the original human-human training data used, and our current techniques can lose performance due to lack of robustness to such deviations (Chollet, 2019; Bengio, 2019).”
Programming Puzzles (2021, methodology): “A recent ARC dataset [14] adopts PbE for evaluating abstraction and reasoning in AI, but as in all PbE applications, there can be ambiguity.”
mGPT: Few-Shot Learners Go Multilingual (2022, background): “GPT-based Models In spite of the fact that the “pre-train & fine-tune” paradigm remains the dominant approach in the field, novel learning concepts and methods such as “zero-shot”, “one-shot” and “few-shot” techniques have been designed to balance the cost of pre-training extensive LMs and accelerat”
Position: Levels of AGI for Operationalizing Progress on the Path to AGI (2023, neutral): “…(per Principle 3), measuring diverse properties including (but not limited to) linguistic intelligence, mathematical and logical reasoning (Webb et al., 2023), spatial reasoning, interpersonal and intra-personal social intelligences, the ability to learn new skills (Chollet, 2019), and creativity.”
Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (2025, neutral): “Additionally, the scope of our dataset could be expanded to encompass domains beyond mathematics, such as programming or more complex tasks like ARC-AGI (Chollet et al., 2025; Chollet, 2019).”
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (2024, neutral): “Recently, Greenblatt [23] shows that repeated sampling is effective when solving puzzles from the ARC challenge [16], observing log-linear scaling as the number of samples increases.”
Neurosymbolic Program Synthesis (neutral): “ARC [49], the AI Math Olympiad, and Natural Plan [50] are three examples of such challenges.”
On the link between conscious function and general intelligence in humans and machines (2022, methodology): “Examining the agents behind these successes more closely, we ﬁnd that they do indeed often fail to live up to the deﬁnition of intelligence provided by Chollet (2019).”