Credit Assignment in Multiple Goal Embodied Visuomotor Behavior
By Constantin A. Rothkopf, Dana H. Ballard. 2010
39 citations recorded by the source corpus.
The intrinsic complexity of the brain can lead one to set aside issues related to its relationships with the body, but the field of embodied cognition emphasizes that understanding brain function at the system level requires one to address the role of the brain-body interface. It has only recently been appreciated that this interface performs huge amounts of computation that does not have to be repeated by the brain, and thus affords the brain great simplifications in its representations. In effect the brain's abstract states can refer to coded representations of the world created by the body. But even if the brain can communicate with the world through abstractions, the severe speed limitations in its neural circuitry mean that vast amounts of indexing must be performed during development so that appropriate behavioral responses can be rapidly accessed. One way this could happen would be if the brain used a decomposition whereby behavioral primitives could be quickly accessed and combined. This realization motivates our study of independent sensorimotor task solvers, which we call modules, in directing behavior. The issue we focus on herein is how an embodied agent can learn to calibrate such individual visuomotor modules while pursuing multiple goals. The biologically plausible standard for module programming is that of reinforcement given during exploration of the environment. However this formulation contains a substantial issue when sensorimotor modules are used in combination: The credit for their overall performance must be divided amongst them. We show that this problem can be solved and that diverse task combinations are beneficial in learning and not a complication, as usually assumed. Our simulations show that fast algorithms are available that allot credit correctly and are insensitive to measurement noise.
Subfields: Artificial Intelligence, Cognitive Neuroscience
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Cognitive Neuroscience
Citations in the corpus
- Information seeking, curiosity and attention: computational and neural mechanisms (2013, neutral): “Computational models have also incorporated uncertainty-based strategies, generating biases toward actions or states that have high variance or entropy (Cohn, Ghahramani et al. 1996; Rothkopf and Ballard 2010).”
- The two-dimensional organization of behavior (2011, unknown): “Other modular reinforcement-learning methods [5]–[8], [15]–[23] that break the task down in other ways on the basis of other constraints might also be adaptable to a two-dimensional map, yielding systems with different properties.”
- Attention, learning and the value of information (2012, methodology): “By including an estimate of the uncertainty that arises at each step, one can further calculate the costs of this uncertainty and the benefits of reducing it by obtaining information (Dayan and Daw, 2008; Rothkopf and Ballard, 2010; Sprague and Ballard, 2005; Tatler et al., 2011).”
- Eye guidance in natural vision: Reinterpreting salience (2011, unknown): “This assumption of behavioral modules is critical to make the problem computationally tractable (Rothkopf & Ballard, 2010), but it is not known whether it is a good model of sensorymotor behavior.”