Mutual information and minimum mean-square error in Gaussian channels

By Dongning Guo, Shlomo Shamai, Sergio Verdú. IEEE Transactions on Information Theory, 2004

1,269 citations recorded by the source corpus.

This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise ratio (SNR) is equal to half the MMSE, regardless of the input statistics. This relationship holds for both scalar and vector signals, as well as for discrete-time and continuous-time noncausal MMSE estimation. This fundamental information-theoretic result has an unexpected consequence in continuous-time nonlinear estimation: For any input signal with finite power, the causal filtering MMSE achieved at SNR is equal to the average value of the noncausal smoothing MMSE achieved with a channel whose SNR is chosen uniformly distributed between 0 and SNR.

Subfields: Computer Networks and Communications, Electrical and Electronic Engineering, Signal Processing

View Paper: https://www.semanticscholar.org/paper/4e9e2469a549897f9885e63e7659b746e9e07f4f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Signal Processing

Citations in the corpus

Continuous trajectory planning of mobile sensors for informative forecasting (2010, methodology): “Similar quantification is performed for non-Gaussian signals [14, 19] and fractional Gaussian channel [9].”
Cooperative Algorithms for MIMO Interference Channels (2010, methodology): “Inspired by the connection between mutual information and mean-square error [16], we derive an explicit joint MMSE pre-coder/receiver design for the interference channel.”
Weighted sum-rate maximization using weighted MMSE for MIMO-BC beamforming design (2008, background): “…results highlighting the relationship between information theoretic quantities (mutual information) and MMSE in single user Multiple Input Multiple Output (MIMO) channels [13], [14], we have established a relationship between WSR and Weighted sum-Minimum Mean Square Error (WMMSE) in the MIMO-BC.”
Optimal errors and phase transitions in high-dimensional generalized linear models (2017, methodology): “1 The additive Gaussian scalar channel We recall some properties (see [54] and [97] for proofs) of the free entropy of the rst scalar channel (36).”
Optimum power allocation for parallel Gaussian channels with arbitrary input distributions (2006, methodology): “Thus, the solution in this paper provides another application for the nexus between mutual information and MMSE of Gaussian channels found in [32].”
Integrated Sensing and Communications: Recent Advances and Ten Open Challenges (2023, methodology): “As for parameter estimation, maximizing the MI between the target impulse response and received echo signals usually leads to minimizing the minimum mean-square error (MMSE) [73], [76].”
Framework for Link-Level Energy Efficiency Optimization with Informed Transmitter (2011, methodology): “The rate functions ri are computed through integration of the MMSE over ρ [26],”
Shredder: Learning Noise Distributions to Protect Inference Privacy (2019, methodology): “y, i.e, higher 1=MI. To this end, we use the reverse of signal to noise ratio (1=SNR) as proxy for our ex vivo notion of privacy. Mutual information is shown to be a function of SNR in noisy channels [32, 33]. In addition, in this paper, we empirically investigate the relation between the two and sh”
0-1 Phase Transitions in Sparse Spiked Matrix Estimation (2019, background): “These are mostly known [19,20], but we adapt them to our setting and provide detailed proofs for the convenience of the reader.”
Shredder: Learning Noise to Protect Privacy with Partial DNN Inference on the Edge (2019, methodology): “Mutual information is shown to be a function of SNR in noisy channels [32, 33].”
Integrated Sensing and Communications: Toward Dual-Functional Wireless Networks for 6G and Beyond (2021, background): “[103], which connected the input-output mutual information, a com-”
A simple proof of the entropy-power inequality (2006, methodology): “Following a simple “noise-incremental” argument, [3] shows that regardless of the distribution of X we can write d d I(X; p X +N) = 1 2 (X; ) (3)”
Impact of antenna correlation on the capacity of multiantenna channels (2005, background): “Far from coincidental, these various dependences are manifestations of a fundamental relationship between the mutual information and the [56].”
Near-Field Communications: A Tutorial Review (2023, methodology): “Moreover, we observe that in the low-SNR regime, the EMI achieved by finite-alphabet inputs is close to that achieved by Gaussian inputs, which is consistent with the results in [115].”
Randomly spread CDMA: asymptotics via statistical physics (2005, background): “In the special case of individually optimal detection, for whi ch the postulated system is identical to the actual one, the rig ht 5In fact, the proof of Theorem 1 led us to the discovery of the ge neral I-MMSE relationship in [41].”