Learning Recursive Distributed Representations for Holistic Computation
By Lonnie Chrisman. 1991
A number of connectionist models capable of representing data with compositional structure have recently appeared. These new models suggest the intriguing possibility of performing holistic structure-sensitive computations with distributed representations. Two possible forms of holistic inference, transformational inference and confluent inference, are identified and compared. Transformational inference was successfully demonstrated by Chalmers; however, the pure transformational approach does not consider the eventual inference tasks during the process of learning its representations. Confluent inference is introduced as a method for achieving a tight coupling between the distributed representations of a problem and the solution for the given inference task while the net is still learning its representations. A dual-ported RAAM architecture based on Pollack's Recursive Auto-Associative Memory is implemented and demonstrated in the domain of Natural Language translation.
Subfields: Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Felix Stahlberg
- Quoc V. Le
- Graham Neubig
- Kyunghyun Cho
- Yonatan Belinkov
- Orhan Firat
- Mohit Bansal
- Richard Socher
In Representation & Self-Supervised Learning
- Quoc V. Le
- Yonatan Belinkov
- Oriol Vinyals
- Ruslan Salakhutdinov
- Yoav Goldberg
- Yoshua Bengio
- Mohammad Norouzi
- Yonghui Wu
Citations in the corpus
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “C L ] 8 responsible for this gap: its slower training and inference speed, ineffectiveness in dealing with rare words, and sometimes failure to translate all words in the source sentence.”
- Asynchronous translations with recurrent neural nets (1997, methodology): “It should be mentioned that an alternative neural approach to asynchronous translation exists [5, 6] which uses RAAMs (recursive auto-associative memories, [16]); there, a RAAM, which may be seen as a recurrent neural net, is specially trained to build unique, reversible state-space representations ”
- The University of Cambridge’s Machine Translation Systems for WMT18 (2018, background): “Encoder-decoder networks (Pollack, 1990; Chrisman, 1991; Forcada and Ñeco, 1997; Kalchbrenner and Blunsom, 2013) are the current prevailing architecture for neural machine translation”
- Recursive Hetero-associative Memories for Translation (1997, methodology): “In particular, the tasks used by Chrisman (1991) are limited to one-to-one translations; that is, no two di erent sentences in one language can have the same translation in the other language.”
- On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, methodology): “2 Neural Machine Translation Early work involving neural networks in machine translation includes [331], where a neural parser was integrated in a speech translation system, as well as more independent neural approaches for machine translation [54, 72, 109, 159, 231, 334].”
- Inductive Learning in Symbolic Domains Using Structure-Driven Recurrent Neural Networks (1996, methodology): “[3, 4, 2]) linguistic applications have been chosen for tests and evaluation.”