Natural Language Processing (Almost) from Scratch
By Ronan Collobert, Jason Aaron Edward Weston, L. Eon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel P. Kuksa. 2011
7,931 citations recorded by the source corpus.
We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Graham Neubig
- Kyunghyun Cho
- Quoc V. Le
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Richard Socher
- Percy Liang
- Yejin Choi
Citations in the corpus
- Structured Attention Networks (2017, background): “Modeling structural dependencies at the final, output layer has been shown to be important in many deep learning applications, most notably in seminal work on graph transformers (LeCun et al., 1998), key work on NLP (Collobert et al., 2011), and in many other areas (Peng et al., 2009; Do & Arti´eres,”
- Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Particularly noteworthy for NLP are algorithms for learning effective word embeddings (Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014) and language model pretraining (Dai and Le, 2015; Ramachandran et al.”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “2 Related Work Collobert et al. (2011), one of the earliest works exploring deep learning for NLP, used a multi-task model with a shared sentence understanding component to jointly learn POS tagging, chunking, named entity recognition, and semantic role labeling.”
- Deep Contextualized Word Representations (2018, methodology): “Following recent state-of-the-art systems (Lample et al., 2016; Peters et al., 2017), the base-line model uses pre-trained word embeddings, a character-based CNN representation, two biLSTM layers and a conditional random field (CRF) loss (Lafferty et al., 2001), similar to Collobert et al. (2011).”
- Learned in Translation: Contextualized Word Vectors (2017, background): “part-of-speech tagging [Collobert et al., 2011], named entity recognition [Pennington et al.”
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, background): “Distributed representations of words (or word embeddings) (Bengio et al., 2003; Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014) have shown to provide useful features for various tasks in natural language processing and computer vision.”
- Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “The token embed-dings, w k , are obtained as a lookup E ( · , θ w ) , initialized using pre-trained word embeddings, and fine tuned during training (Collobert et al., 2011).”
- context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, methodology): “With the exception of the sentence completion task (MSCC), which comes with its own learning corpus, we used the two billion word ukWaC (Ferraresi et al., 2008) as our learning corpus.”
- Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “(Collobert et al., 2011; Mikolov et al., 2013a; Kiros et al., 2015) - or by comparing with human relatedness judgements -unspervised evaluation (Hill et al.”
- GloVe: Global Vectors for Word Representation (2014, background): “Collobert and Weston (2008) decoupled the word vector training from the downstream training objectives, which paved the way for Collobert et al. (2011) to use the full context of a word for learning the word representations, rather than just the preceding context as is the case with language models.”
- Distributed Representations of Sentences and Documents (2014, neutral): “…NLP applications such as word representation, named entity recognition, word sense disam-biguation, parsing, tagging and machine translation (Col-lobert & Weston, 2008; Turney & Pantel, 2010; Turian et al., 2010; Collobert et al., 2011; Socher et al., 2011b; Huang et al., 2012; Zou et al., 2013).”
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, methodology): “We extend the knowledge distillation method (Hinton et al., 2015) to the multi-task learning setting (Caruana, 1997; Xu et al., 2018; Collobert et al., 2011; Zhang and Yang, 2017; Liu et al., 2015).”
- Multi-Task Deep Neural Networks for Natural Language Understanding (2019, background): “Recently, there is a growing interest in applying MTL to representation learning using deep neural networks (DNNs) (Collobert et al., 2011; Liu et al., 2015; Luong et al., 2015; Xu et al., 2018) for two reasons.”
- The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “Finally, as a check to our hypothesis, we also tried a curriculum schedule that used SST, QA-SRL, QA-ZRE, WOZ, WikiSQL and MWSC in the initial curriculum.”
- Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “For example, the study reported in (Collobert et al., 2011) demonstrated significant accuracy gains in tagging, named entity recognition, and semantic role labeling when using vector space word”
- Efficient Estimation of Word Representations in Vector Space (2013, background): “It was later shown that the word vectors can be used to significantly improve and simplify many NLP applications [4, 5, 26].”
- CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “Language models might also offer a foundation to extend proposals of unified, multi-task systems for all of NLP (Collobert & Weston, 2008; Collobert et al., 2011), parsing and tagging (Hashimoto et al., 2016), multiple languages (Wu et al., 2016; Johnson et al., 2017), and multiple modalities (Luong…”
- Unifying Question Answering and Text Classification via Span Extraction (2019, background): “The use of pre-trained encoders for transfer learning in NLP dates back to (Collobert and Weston, 2008; Collobert et al., 2011) but has had a resurgence in the recent past.”
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “Multitask representation learning in NLP is well studied, and again can be traced back at least as far as Collobert et al. (2011).”
- Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (2016, background): “With the emergence of deep learning as a viable alternative for many NLP tasks (Collobert et al., 2011), researchers have started considering this framework as an attractive, fully data-driven alternative to abstractive summarization.”