From machine learning to machine reasoning

By L. Eon Bottou. 2011

297 citations recorded by the source corpus.

A plausible definition of “reasoning” could be “algebraically manipulating previously acquired knowledge in order to answer a new question”. This definition covers first-order logical inference or probabilistic inference. It also includes much simpler manipulations commonly used to build large learning systems. For instance, we can build an optical character recognition system by first training a character segmenter, an isolated character recognizer, and a language model, using appropriate labelled training sets. Adequately concatenating these modules and fine tuning the resulting system can be viewed as an algebraic operation in a space of models. The resulting model answers a new question, that is, converting the image of a text page into a computer readable text. This observation suggests a conceptual continuity between algebraically rich inference systems, such as logical or probabilistic inference, and simple manipulations, such as the mere concatenation of trainable learning systems. Therefore, instead of trying to bridge the gap between machine learning systems and sophisticated “all-purpose” inference mechanisms, we can instead algebraically enrich the set of manipulations applicable to training systems, and build reasoning capabilities from the ground up.

Subfields: Artificial Intelligence, Prompting & Reasoning, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

In Knowledge Graphs & Representation

Citations in the corpus

What can the brain teach us about building artificial intelligence? (2016, background): “to-learn can occur through the sharing of features between the models learned for old objects (or old tasks) and the models learned for new objects (or new tasks) (Anselmi et al., 2016; Baxter, 2000; Bottou, 2014; Lopez-Oaz, Bottou, Scholkopf, & Vapnik, 2016; Salakhutdinov, Torralba, & Tenen”
Gated Graph Sequence Neural Networks (2015, background): “RNN and LSTM fail on both tasks, but GGS-NNs learns to make perfect predictions using only 50 training examples.”
Representation Learning: A Review and New Perspectives (2012, background): “amples) (Martens, 2010; Le et al., 2011a), with a variant of stochastic gradient descent recently winning an optimization challenge 14. Recursive networks (Pollack, 1990; Frasconi et al., 1997, 1998; Bottou, 2011; Socher et al., 2011) generalize recurrent networks: instead of the unfolded computatio”
Learning to Compose Neural Networks for Question Answering (2016, neutral): “Other approaches in this general family include the “universal parser” sketched by Bottou (2014), the graph transformer networks of Bottou et al. (1997), the knowledge-based neural networks of Towell and Shavlik (1994) and the recursive neural networks of Socher et al. (2013), which use a ﬁxed tree…”
Compositional Attention Networks for Machine Reasoning (2018, background): “rm a chain of operations that build upon each other, allowing MAC to develop reasoning skills from the ground up, realizing the vision of an algebraic, compositional model of inference as proposed by Bottou (2014). Although each cell’s functionality has only a limited range of possible continuous be”
What do we need to build explainable AI systems for the medical domain? (2017, background): “This brings us to enormously important fundamental research opportunities in causality (Krynski and Tenenbaum, 2007; Pearl, 2009; Bottou, 2014) - which is novel in the context of personalized (P4) medicine.”
Counterfactual reasoning and learning systems: the example of computational advertising (2012, background): “We can approach the manipulability puzzle in the same spirit by viewing causation as a reasoning model (Bottou, 2011) rather than a property of the world.”
A semantic matching energy function for learning with multi-relational data (2013, methodology): “Inspired by the framework introduced by Bordes et al (2011) as well as by recent work of Bottou (2011), the main idea behind our semantic matching energy function is the following.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, methodology): “This idea has been generalized from sequences and 2-D images to arbitrary graphs with recursive neural networks or recursive graphical models (Pollack, 1990; Frasconi et al., 1998; Bottou, 2011; Socher et al., 2011), Markov Logic Networks (Richardson and Domingos, 2006) and relational learning (Geto”
Unifying distillation and privileged information (2015, neutral): “In a wider context, the machines-teaching-machines paradigm is one step toward the deﬁnition of machine reasoning of Bottou (2014), “the algebraic manipulation of previously acquired knowledge to answer a new question”.”
Learning by Abstraction: The Neural State Machine (2019, methodology): “g skills such as multi-step inference, relational and spatial reasoning, logic and comparisons. 2 Related work Our model connects to multiple lines of research, including works about compositionality [14, 38], concept acquisition [36, 81], and neural computation [28, 62, 7]. Several works have explo”
Strongly-Typed Recurrent Neural Networks (2016, background): “A definition of machine reasoning, adapted from (Bottou, 2014), is “algebraically manipulating features to answer a question”.”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, background): “Recursive networks (Pollack, 1990; Frasconi et al. , 1997, 1998; Bottou, 2011; Socher et al. , 2011) generalize recurrent networks: instead of the unfolded computation being represented by a chain (with one element per time step), it is represented by a directed acyclic graph, where the same…”
Large Language Models as Analogical Reasoners (2023, background): “Teaching language models to reason has been a long-standing area of research (Bottou, 2014; Zhao et al., 2023; Wei et al., 2022b).”
Learning to Discover Efficient Mathematical Identities (2014, background): “[22, 23], as well as generic knowledge representation Bottou [2].”