Elements of Information Theory
By Giorgio Matteo Vitetta, Desmond P. Taylor, Giulio Colavolpe, Fabrizio Pancaldi, Philippa A. Martin. 2013
900 citations recorded by the source corpus.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/8ffba4d9486c66b4d0a63ef95b41aaabded3c844
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Log-odds sequence logos (2015, neutral): “From the information theory (Cover and Thomas, 1991), the entropy (in bits) of a frequency distribution ~f is defined asHð~fÞ PL j=1 fjlog2 fj. Schneider and Stephens (1990) use entropy to define the ‘information content’ of a given position as the difference between the maximum possible entropy…”
- Implicit Causal Models for Genome-wide Association Studies (2017, positive): “Moreover, under the data processing inequality (Cover & Thomas, 1991), h n preserves more information relevant to y n than subsequent layers of the neural network (making it statistically efficient).”
- Non-commercial Research and Educational Use including without Limitation Use in Instruction at Your Institution, Sending It to Specific Colleagues That You Know, and Providing a Copy to Your Institution's Administrator. All Other Uses, Reproduction and Distribution, including without Limitation Comm (2005, neutral): “All these methods require considerable A u t h o r ' s p e r s o n a l”
- Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE (2017, neutral): “CD k is always non-negative because running Markov chain forward can only decrease the KL divergence (Cover & Thomas, 2012), and equals zero only if q 0 matches p θ .”
- Angular Synchronization by Eigenvectors and Semidefinite Programming (2009, positive): “In this section we provide a positive answer to this question using the information theoretic Shannon bound [8].”
- Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach (2021, positive): “Specifically, the quantity 𝐼 ( ˆ 𝑍, 𝑋 ) is comprehended as the preserved information in ˆ 𝑍 given 𝑋 and measured by the minimum description length [37] (or rate).”
- Mixtures of Conditional Gaussian Scale Mixtures Applied to Multiscale Image Representations (2011, positive): “For a strictly stationary Markov process, one can show that these quantities reduce to [6, 11]”
- Measuring Semantic Progress in Multi-turn Dialogue via Information Gain (2026, unknown): “Its differential entropy h ( (cid:98) p z ) cannot be larger than the entropy of a Gaussian with zero mean and covariance (cid:98) Σ z , see [40], Theorem 8.6.5.”
- Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs (2026, unknown): “We compare the two partitions with mutual information [52], written MI l ( τ ) = I ( C l ( τ ); C L ( τ )) .”