Nonnegative Decomposition of Multivariate Information

By Paul L. Williams, Randall D. Beer. arXiv.org, 2010

748 citations recorded by the source corpus.

Of the various attempts to generalize information theory to multiple variables, the most widely utilized, interaction information, suffers from the problem that it is sometimes negative. Here we reconsider from first principles the general structure of the information that a set of sources provides about a given variable. We begin with a new definition of redundancy as the minimum information that any source provides about each possible outcome of the variable, averaged over all possible outcomes. We then show how this measure of redundancy induces a lattice over sets of sources that clarifies the general structure of multivariate information. Finally, we use this redundancy lattice to propose a definition of partial information atoms that exhaustively decompose the Shannon information in a multivariate system in terms of the redundancy between synergies of subsets of the sources. Unlike interaction information, the atoms of our partial information decomposition are never negative and always support a clear interpretation as informational quantities. Our analysis also demonstrates how the negativity of interaction information can be explained by its confounding of redundancy and synergy.

Subfields: Artificial Intelligence, Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/42b3626ec3baa1f5f06bc1773db96fcbd419db48

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

Citations in the corpus

Gene Regulatory Network Inference from Single-Cell Data Using Multivariate Information Measures (2016, methodology): “Also, unlike CLR it was designed to capture higher order dependencies and to distinguish between direct and indirect interactions (Williams & Beer 2010).”
Maximally Informative Hierarchical Representations of High-Dimensional Data (2014, background): “This quantity has appeared as a measure of the redundant information that the Xi’s carry about Y [12] and this interpretation has been explored in depth [13, 14].”
Controllable Guarantees for Fair Outcomes via Contrastive Information Estimation (2021, background): “The triple overlap region of the information Venn diagram, sometimes called the “interaction information”, can be negative (Williams and Beer 2010).”
Quantifying unique information (2013, methodology): “We call these axioms the PI axioms after the partial information decomposition framework derived from these axioms in [4].”
Inferring spatial and signaling relationships between cells from single cell transcriptomic data (2020, methodology): “To interrogate whether two genes affect each other across cells through space, we utilize partial information decomposition 28,29,34 to compute the intercellular gene – gene regulatory information ﬂ ow (Fig.”
A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula (2016, background): “While it is not clear to what degree such cancellation might occur in neuroimaging recordings this is a question that has recently received much interest [Bertschinger et al., 2014; Griffith and Koch, 2012; Harder et al., 2012; Olbrich et al., 2015; Timme et al., 2013; Williams and Beer, 2010].”
Neural interactions in the human frontal cortex dissociate reward and punishment learning (2024, methodology): “To address this question, we performed model-based network-level analyses based on the Partial Information Decomposition (PID) framework (Williams and Beer, 2010; Wibral et al., 2017; Lizier et al., 2018).”