Text Understanding from Scratch

By Yann LeCun. 2015

575 citations recorded by the source corpus.

This article demontrates that we can apply deep learning to text understanding from character-level inputs all the way up to abstract text concepts, using temporal convolutional networks (ConvNets). We apply ConvNets to various large-scale datasets, including ontology classification, sentiment analysis, and text categorization. We show that temporal ConvNets can achieve astonishing performance without the knowledge of words, phrases, sentences and any other syntactic or semantic structures with regards to a human language. Evidence shows that our models can work for both English and Chinese.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “Deep learning models perform extremely well with correlated data, which contributed to substantial improvements in computer vision [45], image processing, video processing, face recognition [80], speech recognition [33], text-to-speech systems [62] and natural language processing [2, 14, 88].”
Opening the Black Box of Deep Neural Networks via Information (2017, background): “In the last decade, deep learning algorithms have made remarkable progress on numerous machine learning tasks and dramatically improved the state-of-the-art in many practical areas [Graves et al. (2013); Zhang & LeCun (2015); Hinton et al. (2012); He et al. (2015); LeCun et al. (2015)].”
Very Deep Convolutional Networks for Text Classification (2016, methodology): “We present our results on eight freely available large-scale data sets introduced by (Zhang et al., 2015) which cover several classiﬁcation tasks such as sentiment analysis, topic classiﬁcation or news categorization (see (Zhang et al., 2015) for a detailed description. and 14.”
Multilingual Language Processing From Bytes (2015, background): “This is truly language annotation from scratch (see Collobert et al. (2011) and Zhang and LeCun (2015)).”
End-to-end learning of semantic role labeling using recurrent neural networks (2015, methodology): “Inspired by the work in computer vision, people hierarchically organized a window of words through convolution layers in deep form to account for the higher level of organization to solve the document classiﬁcation task (Kim, 2014; Zhang and LeCun, 2015).”
A Bayesian Data Augmentation Approach for Learning Deep Models (2017, background): “Deep learning has become the “backbone” of several state-of-the-art visual object classiﬁcation [19, 14, 25, 27], speech recognition [17, 12, 6], and natural language processing [4, 5, 31] systems.”
Long Text Generation via Adversarial Training with Leaked Information (2017, methodology): “For the discriminator, we choose the CNN architecture (Zhang and LeCun 2015) as the feature extractor and the binary classiﬁer.”
Adversarial Ranking for Language Generation (2017, background): “Recent studies show that the convolutional neural network can achieve high performance for machine translation [7, 34] and text classification [36].”
StarSpace: Embed All The Things! (2017, background): “Several more recent works including (Tang, Qin, and Liu 2015), (Zhang and LeCun 2015), (Conneau et al. 2016), TagSpace (Weston, Chopra, and Adams 2014) and fastText (Joulin et al. 2016) have yielded good results on classiﬁcation tasks such as sentiment analysis or hashtag prediction.”
Globally Normalized Reader (2017, background): “Zhang and LeCun (2015) use a thesaurus to generate new training examples based on synonyms.”
Molding CNNs for text: non-linear, non-consecutive convolutions (2015, background): “, 2010; Kalchbrenner and Blunsom, 2013) to recursive models (Pollack, 1990; Küchler and Goller, 1996), and including convolutional neural nets (Collobert and Weston, 2008; Collobert et al., 2011; Yih et al., 2014; Shen et al., 2014; Kalchbrenner et al., 2014; Zhang and LeCun, 2015).”
On the influence of momentum acceleration on online learning (2016, methodology): “1 When β1 = 0 and β2 = β we recover the heavy-ball algorithm (Polyak, 1964, 1987), and when β2 = 0 and β1 = β, we recover Nesterov’s algorithm (Nesterov, 2004).”
$\mathtt {Deepr}$: A Convolutional Net for Medical Records (2016, background): “The architecture of Deepr is multilayered and is inspired by recent convolutional neural nets (CNNs) in natural languages [9], [11]–[14].”
Weakly-Supervised Neural Text Classification (2018, background): “3-6014-2/18/10...$15.00 https://doi.org/10.1145/3269206.3271737 categorization [32] and query intent classification [29]. Recently, deep neural models — including convolutional neural networks (CNNs) [11, 12, 35, 36] and recurrent neural networks (RNNs) [22, 23, 32] — have demonstrated superiority f”
hep-th (2018, unknown): “Answers [37] is about “science & math” or “business & ﬁnance” (c.f. the discussion in Zhang et al. [38]).”
Contextual LSTM (CLSTM) models for Large scale NLP tasks (2016, methodology): “Convolution networks have been used for multi-level text understanding, starting from character-level inputs all the way to abstract text concepts [47].”
Attention-Based Prototypical Learning Towards Interpretable, Confident and Robust Deep Neural Networks (2019, methodology): “A sequence of raw characters (of 69 different values (Zhang & LeCun, 2015)) is used as the input.”
Denoising Bodies to Titles: Retrieving Similar Questions with Recurrent Convolutional Models (2015, methodology): “, 1998) have also been successfully applied to various NLP tasks (Kalchbrenner et al., 2014; Kim, 2014; Kim et al., 2015; Zhang and LeCun, 2015; Gao et al., 2014).”