A unified architecture for natural language processing: deep neural networks with multitask learning

By Ronan Collobert, Jason Aaron Edward Weston. 2008

6,023 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “We experiment with two ways of combining the varying number of {ht}t to form a fixed-size vector, either by selecting the maximum value over each dimension of the hidden units (max pooling) (Collobert and Weston, 2008) or by considering the average of the representations (mean pooling).”
GloVe: Global Vectors for Word Representation (2014, methodology): “Collobert and Weston (2008) decoupled the word vector training from the downstream training objectives, which paved the way for Collobert et al. (2011) to use the full context of a word for learning the word representations, rather than just the preceding context as is the case with language models.”
Distributed Representations of Sentences and Documents (2014, methodology): “…for many natural language processing tasks such as language modeling (Bengio et al., 2006; Mikolov, 2012), natural language understanding (Collobert & Weston, 2008; Zhila et al., 2013), statistical machine translation (Mikolov et al., 2013b; Zou et al., 2013), image understanding (Frome…”
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank (2013, background): “These vectors can be trained in an unsupervised fashion to capture distributional similarities (Collobert and Weston, 2008; Huang et al., 2012) but then also be ﬁne-tuned and trained to speciﬁc tasks such as sentiment detection (Socher et al., 2011b).”
Better Mixing via Deep Representations (2012, background): “, as already exploited in multi-task learning (Caruana, 1995; Baxter, 1997; Collobert and Weston, 2008) and learning algorithms involving parameter sharing (Lang and Hinton, 1988; LeCun, 1989).”
Experience Grounds Language (2020, background): “learned from text capture a great deal of information about meaning in and out of context (Collobert and Weston, 2008; Turian et al., 2010; Mikolov et al., 2013; McCann et al., 2017).”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, background): “…and features to the downstream tasks, which includes word-level representations (Brown et al., 1992; Ando & Zhang, 2005; Blitzer et al., 2006; Collobert & Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) and sentence-level representations (Kiros et al., 2015; Logeswaran & Lee,…”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “…neural models with large amounts of available supervision can produce representations that effectively transfer to a broad range of NLP tasks (Collobert and Weston, 2008; Dai and Le, 2015; Kiros et al., 2015; Hill et al., 2016; Conneau and Kiela, 2018; McCann et al., 2017; Peters et al.,…”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “The embeddings (Collobert and Weston, 2008; Collobert et al., 2011), intermediate representations (Peters et al.”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “Estimation of the word vectors itself was performed using different model architectures and trained on various corpora [4, 26, 21, 18, 9], and some of the resulting word vectors were made available for future research and comparison2.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “Extensions to other areas in NLP.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “Multitask learning (Caruana, 1997) has been successfully applied to multiple domains, including NLP, speech processing, drug discovery and many others (Collobert and Weston, 2008; Deng et al., 2013; Ramsundar et al., 2015; Maurer et al., 2016; Ruder, 2017; Kaiser et al., 2017).”
Unifying Question Answering and Text Classification via Span Extraction (2019, background): “The goal of unifying NLP is not new (Collobert and Weston, 2008; Collobert et al., 2011).”