Universal coding, information, prediction, and estimation

By Jorma Rissanen. 1984

Subfields: Information Retrieval & Search, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

In Information Theory

Citations in the corpus

True Few-Shot Learning with Language Models (2021, background): “MDL can be estimated by evaluating the average loss on a fold F ( D ) k when training on the previous folds This procedure is referred to as “online coding” [29, 30], as it evaluates the generalization loss of the algorithm as it learns “online” from more and more data.”
Information-Theoretic Probing with Minimum Description Length (2020, background): “The online (or prequential) code (Rissanen, 1984) is a way to encode both the model and the labels without directly encoding the model weights.”
Estimating means of bounded random variables by betting (2020, background): “(4)], Dawid [1984], and Lorden and Pollak [2005] as well as in the information theory literature [Rissanen, 1984].”
Visual SLAM and Structure from Motion in Dynamic Environments (2018, methodology): “On the other hand, Rissanen [129] developed Minimum Description Length (MDL) by using a minimum-bit representation to minimize the coding length of the data.”
Statistical Inference Under Multiterminal Data Compression (1998, background): “The true significance of the multiterminal statistical inference system of Berger, as well as the single-user universal coding system of Rissanen [2], is that those systems enabled us to incorporate problems of statistical inference into information theory.”
On Prediction Using Variable Order Markov Models (2004, background): “…on the redundancy of any universal prediction (and compression) algorithm is Ω(K( log T2T )), where K is (roughly) the number of parameters of the model encoding the distribution P̂ (Rissanen, 1984).4 Some of the algorithms we discuss below are universal with respect to some classes of sources.”
Combining Linear Regression Models (2005, background): “…and/or specific theoretical/empirical considerations, including the Akaike information criterion (AIC) (Akaike 1973), the Bayes information criterion (BIC) (Schwarz 1978), cross-validation (Allen 1974; Stone 1974), and Minimum Description Length (e.g., Rissanen 1984; Barron, Rissanen, and Yu 1998).”
Context-based adaptive binary arithmetic coding in the H.264/AVC video compression standard (2003, background): “2A more rigorous treatment of that problem can be found in [23] and [24] Fig.”
Prediction, learning, and games (2006, background): “The work of Weinberger, Merhav, and Feder was inspired by a similar result of Rissanen [238] in the model of probabilistic prediction.”
The Minimum Description Length Principle in Coding and Modeling (1998, background): “To bolster the connection given here between the individual risk of efficient estimators of order and the optimal cumulative risk or redundancy of order , we mention here that classic results on negligibility of the set of parameter values for which an estimator is superefficient (LeCam [35] assumin”
Identification of humans using gait (2004, methodology): “We propose two different approaches to model the structure and dynamics of gait.”
Predictability , Complexity , and Learning (2002, background): “He has emphasized strongly (Rissanen, 1984, 1986, 1987, 1989) that tting a model to data represents an encoding of those data, or predicting future data, and that in searching for an ef cient code, we need to measure not only the number of bits required to describe the deviations of the data from…”
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, methodology): “These methods are all compatible with RDA, and here, we use online (or prequential) coding (Rissanen, 1984; Dawid, 1984), an effective method for estimating MDL when used with deep learning (Blier and Ollivier, 2018).”