Sequential neural text compression

By J. Urgen Schmidhuber, Stefan Heil. 1996

Subfields: Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “Neural Language Models An alternative to n-gram models, first proposed by Miikkulainen and Dyer (1991) and Schmidhuber and Heil (1996) and later popularized by Bengio et al. (2000), is to use a neural network to estimate the probability of the next token given prior tokens.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “, through predictive coding [228, 248]) videos of people placing toys in boxes, or to summarize such videos through textual outputs.”
One Big Net For Everything (2018, methodology): “For example, suppose that ONE has learned to represent (e.g., through predictive coding [61, 78]) videos of people placing toys in boxes, or to summarize such videos through textual outputs.”
Auto-Sizing Neural Networks: With Applications to n-gram Language Models (2015, methodology): “Schmidhuber and Heil (1996) proposed a character n -gram model using neural networks which they used for text compression.”
Adaptive weighing of context models for lossless data compression (2005, methodology): “P12 is based on an earlier 3-layer neural network model which was trained off line by back propagation to predict text (Schmidhuber and Heil 1996).”
A Neural Probabilistic Language Model (2003, methodology): “of character-based text compression using neural networks [11].”
Maximizing Fun by Creating Data with Easily Reducible Subjective Complexity (2013, background): “3; (Huffman 1952; Schmidhuber and Heil 1996)), thus achieving a compressed history representation.”
Optimal Artiﬁcial Curiosity, Creativity, Music, and the Fine Arts (2005, background): “Similarly, a p that learns a probability distribution of the possible next events, given previous events, can be used to efﬁciently encode observations with high (low) predicted probability by few (many) bits [8, 42], thus achieving a compressed history representation.”
Neural Probabilistic Language Models (2006, methodology): “The approach proposed here is also related to previous proposals of characterbased text compression using neural networks to predict the probability of the next character [ Schmidhuber, 1996 ].”
Online Learning with Gated Linear Networks (2017, methodology): “Building on the work of Schmidhuber and Heil (1996), Mahoney (2000) introduced a number of key algorithmic and architectural changes tailored towards computationally eﬃ-cient online density estimation.”
Continuous space language models (2007, unknown): “Similar ideas were also formulated in the context of character-based text compression (Schmidhuber, 1996).”
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes (2008, methodology): “Even when there is limited predictability, efﬁcient compression can still be achieved by assigning short codes to events that are predictable with high probability [28, 95].”
Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) (2010, background): “3) Let some (possibly application-dependent) compressor improvement algorithm (such as a learning algorithm for an adaptive neural network predictor, possibly triggered by a controller action) use to obtain a hopefully better compressor (such as a neural net with the same size and the same constant ”
Hierarchical Probabilistic Neural Network Language Model (2005, background): “The idea of using neural networks for language modeling is not new either (?; ?), and is similar to proposals of character-based text compression using neural networks to predict the probability of the next character (Schmidhuber, 1996).”
Image compression with neural networks - A survey (1999, methodology): “By considering each pixel as an individual symbol in a pre-de"ned scanned order, other general lossless data compression neural networks can also be applied for image compression [28,59].”