Mutual Information, Fisher Information, and Population Coding

By Nicolas J.-B. Brunel, Jean-Pierre Nadal. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Where is the Information in a Deep Neural Network? (2019, methodology): “As a side note, we may wonder what distribution of the inputs would maximize the effective mutual information Ieff,β(x; z), that is, what distribution the network is maximally adapted to represent [5].”
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015, background): “The bound (4) is highly convenient (especially when compared to other bounds) since the transition probability p ( s | a , s ) appears linearly in the expectation and we never need to evaluate its probability — we can thus evaluate the expectation directly by Monte Carlo using data obtained by…”
Inference and computation with population codes. (2003, background): “The Fisher information is related to the Shannon information I(s;r ), which quantifies the deviation from independence of the stimulussand the noisy activitiesr (see Brunel & Nadal 1998).”
Optimal neural population coding of an auditory spatial cue (2004, result): “Previous general theoretical studies using relatively narrow tuning curves also demonstrate that the optimal distribution of preferred tuning follows the probability distribution of the stimulu”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, methodology): “, extracting local patterns with high-order directional derivatives [72], training GANs with gradient penalty [40], or optimizing the Fisher information [5].”
Information Maximization in Noisy Channels : A Variational Approach (2003, methodology): “In a neural context, maximal information transmission has been extensively studied and proposed as a principal goal of sensory processing[3, 9, 7, 11, 12].”
Predictability , Complexity , and Learning (2002, background): “In certain limits, decoding a signal from a population of N neurons can be thought of as “learning” a parameter from N observations, with a parallel notion of information transmission (Brunel & Nadal, 1998; Kang & Sompolinsky, 2001).”
Predictability, Complexity, and Learning (2000, background): “In certain limits, decoding a signal from a population of N neurons can be thought of as “learning” a parameter from N observations, with a parallel notion of information transmission ( Brunel & Nadal, 1998 ; Kang & Sompolinsky, 2001).”
The IM algorithm: a variational approach to Information Maximization (2003, methodology): “In a neural context, maximal information transmission has been extensively studied and proposed as a principal goal of sensory processing[2, 5, 7].”
Degeneration in VAE: in the Light of Fisher Information Loss (2018, methodology): “When the divergence reach its optimal, the first order derivative is zero and the secondary derivative at θ0 is defined as Fisher Information (Brunel and Nadal, 1998): Ip(θ) = ∂2 ∂θ2 Dx(pθ0 , pθ) = −EX [ ∂2 log pθ(x) ∂θ2 ] (4) Fisher Information is thus not a function w.r.t. the stochastic variable…”
Synergy, Redundancy, and Independence in Population Codes (2003, background): “The Fisher information can be used to place a lower bound on the encoded information, but this bound is tight only under specialized conditions (Brunel and Nadal, 1998; Kang and Sompolinsky, 2001).”