Learning and development in neural networks: the importance of starting small.

By Jeffrey L. Elman. 1993

Subfields: Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Science & Psychology

Citations in the corpus

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, neutral): “Elman (1991) tested an SRN on a miniature language that simulated English relative clauses, and found that the network was only able to learn the language under highly specific circumstances (Elman, 1993), though later work has called some of his conclusions into question (Rohde and Plaut, 1999; Car”
Colorless Green Recurrent Networks Dream Hierarchically (2018, background): “Early work showed that RNNs can, to a certain degree, handle data generated by context-free and even context-sensitive grammars (e.g., Elman, 1991, 1993; Rohde and Plaut, 1997; Christiansen and Chater, 1999; Gers and Schmidhuber, 2001; Cartling, 2008).”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “Learning and development interact in a very intricate way (Elman, 1993). Humans show an exceptional capacity to learn throughout their lifespan and,with respect to other species, exhibit the lengthiest developmental process for reaching maturity. There is a limited time window in development in whic”
Automated Curriculum Learning for Neural Networks (2017, background): “Over two decades ago, in The importance of starting small , Elman put forward the idea that a curriculum of progressively harder tasks could signiﬁcantly accelerate a neural network’s training (Elman, 1993).”
Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, neutral): “…such constraints in synergy with goal babbling and intrinsic motivation, such as explored in [129], would potentially allow to constrain and simplify further learning since the first actions of the robot [130,131], and could be crucial when considering life-long learning in unbounded task spaces.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “+ forward transfer + backward transfer + knowledge retention – task boundaries – off-line learning (Elman, 1993) (Bengio et al.”
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks (2016, background): “The idea dates back at least to Elman [11], who showed that both the evolution of tasks and the network architectures can be beneficial in the language processing scenario.”
Hindsight Experience Replay (2017, positive): “Our approach may be seen as a form of implicit curriculum learning (Elman, 1993; Bengio et al., 2009).”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “There are also results showing that specific ways of setting the initial distribution and ordering of examples (“curriculum learning”) can yield better solutions (Elman, 1993; Bengio et al., 2009; Krueger and Dayan, 2009).”
A Roadmap Towards Machine Intelligence (2015, background): “This is motivated by the observation that humans learn incrementally when developing complex skills, an idea that has also previously been studied in the context of recurrent neural network training by Elman (1993).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Since the early 1990s, when interest in neural approaches waned in the nlp and ai communities, cognitive science research has continued to explore their application to syntax and language production (e.g., Elman, 1990, 1993; Chang et al., 2006).”
Meta-Learning in Neural Networks: A Survey (2020, background): “More generally, the curriculum [149] refers to sequences of”
Rethinking eliminative connectionism. (1998, background): “A third example is the simple recurrent network (henceforth, SRN) account of learning aspects of language by Elman (1990, 1991, 1993).”
A Model of Grounded Language Acquisition: Sensorimotor Features Improve Lexical and Grammatical Learning. (2005, methodology): “For the simulations reported here, we have adopted and extended the Simple-Recurrent Network architecture that has been shown many times to be capable of learning simple aspects of grammar, namely basic syntax (e.g., Elman, 1990, 1993; Howell & Becker, 2001).”
The Problem of Rapid Variable Creation (2009, unknown): “In particular, with regard to possible difﬁculties with the prediction task performed in various SRN applications (see Elman, 1990, 1993), he objects to the use of distributed representations on the grounds that they fail “to unambiguously represent all and only the possible continuations to a…”
Algorithmic and Human Teaching of Sequential Decision Tasks (2012, background): “For instance, curriculum design suggests that the teacher should select examples by starting with easy examples and gradually increasing the difficulty (Elman 1993; Bengio et al. 2009; Zang et al. 2010; Taylor 2009).”