Statistical mechanics of optimal convex inference in high dimensions

Physical Review X

Volumn 6, Issue 3, 2016, Pages

  Advani, Madhu ^a   Ganguli, Surya ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BAYESIAN NETWORKS; CLUSTERING ALGORITHMS; COMPUTATION THEORY; CONVEX OPTIMIZATION; CRAMER-RAO BOUNDS; INFERENCE ENGINES; MAXIMUM LIKELIHOOD ESTIMATION; PROBLEM SOLVING; RANDOM VARIABLES; SIGNAL TO NOISE RATIO; SITE SELECTION; STATISTICAL MECHANICS;

COMPUTATIONAL TRACTABILITY; CONVEX OPTIMIZATION PROBLEMS; CONVEX OPTIMIZATION THEORY; HIGH DIMENSIONAL INFERENCE; HIGH-DIMENSIONAL DATA ANALYSIS; INFORMATION-THEORETIC LIMITS; OPTIMIZATION PROBLEMS; RANDOM MATRIX THEORY;

PARAMETER ESTIMATION;

EID: 84992602703     PISSN: None     EISSN: 21603308     Source Type: Journal    
DOI: 10.1103/PhysRevX.6.031034     Document Type: Article

References (46)

1. T. J. Sejnowski, P. S. Churchland, and J. A. Movshon, Putting Big Data to Good Use in Neuroscience, Nat. Neurosci. 17, 1440 (2014).
2. S. Ganguli and H. Sompolinsky, Compressed Sensing, Sparsity, and Dimensionality in Neuronal Information Processing and Data Analysis, Annu. Rev. Neurosci. 35, 485 (2012).
3. R. Clarke, H.W. Ressom, A. Wang, J. Xuan, M. C. Liu, E. A. Gehan, and Y. Wang, The Properties of High-Dimensional Data Spaces: Implications for Exploring Gene and Protein Expression Data, Nat. Rev. Cancer 8, 37 (2008).
4. W. Raghupathi and V. Raghupathi, Big Data Analytics in Healthcare: Promise and Potential, Health Inf. Sci. Syst. 2, 3 (2014).
5. J. Fan, J. Lv, and L. Qi, Sparse High Dimensional Models in Economics, Ann. Rev. Econ. 3, 291 (2011).
6. J. Leskovec, K. J. Lang, A. Dasgupta, and M.W. Mahoney, Community Structure in Large Networks: Natural Cluster Sizes and the Absence of Large Well-Defined Clusters, Internet Math. 6, 29 (2009).
7. M. L. Jockers, Macroanalysis: Digital Methods and Literary History (University of Illinois Press, Champaign, IL, 2013).
8. D. L. Donoho, High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality, AMS Conference on Math Challenges of the 21st Century (2000), pp. 1-32, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.329.3392.
9. V. I. Serdobolskii, Multivariate Statistical Analysis: A High-Dimensional Approach (Springer Science & Business Media, Dordrecht, 2013), Vol. 41.
10. M. Advani, S. Lahiri, and S. Ganguli, Statistical Mechanics of Complex Neural Systems and High Dimensional Data, J. Stat. Mech. (2013) P03014.
11. M. Mezard, G. Parisi, and M. A. Virasoro, Spin Glass Theory and Beyond (World Scientific, Singapore, 1987).
12. D. J. Amit, H. Gutfreund, and H. Sompolinsky, Statistical Mechanics of Neural Networks Near Saturation, Ann. Phys. (N.Y.) 173, 30 (1987).
13. E. Gardner and B. Derrida, Optimal Storage Properties of Neural Network Models, J. Phys. A 21, 271 (1988).
14. E. Gardner, The Space of Interactions in Neural Network Models, J. Phys. A 21, 257 (1988).
15. D. Guo and S. Verdú, Randomly Spread CDMA: Asymptotics via Statistical Physics, IEEE Trans. Inf. Theory 51, 1983 (2005).
16. D. Gou, D. Baron, and S. Shamai, A Single-Letter Characterization of Optimal Noisy Compressed Sensing, in Proceedings of the 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton) (IEEE, 2009), pp. 52-59, https://www.scholars.northwestern.edu/en/publications/a-single-letter-characterization-of-optimalnoisy-compressed-sens.
17. S. Rangan, V. Goyal, and A. K. Fletcher, Asymptotic Analysis of MAP Estimation via the Replica Method and Compressed Sensing, in Advances in Neural Information Processing Systems 22 (Curran Associates, Inc., Red Hook, 2009), pp. 1545-1553.
18. S. Ganguli and H. Sompolinsky, Statistical Mechanics of Compressed Sensing, Phys. Rev. Lett. 104, 188701 (2010).
19. F. Krzakala, M. Mézard, F. Sausset, Y. Sun, and L. Zdeborová, Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices, J. Stat. Mech. (2012) P08009.
20. Y. Kabashima, F. Krzakala, M. Mézard, A. Sakata, and L. Zdeborová, Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization, IEEE Trans. Inf. Theory 62, 4228 (2016).
21. A. Engel and C. V. den Broeck, Statistical Mechanics of Learning (Cambridge University Press, Cambridge, England, 2001).
22. H. Cramér, Mathematical Methods of Statistics, Princeton Mathematical Series (Princeton University Press, Princeton, 1946), Vol. 9.
23. P. J. Huber, Robust Regression: Asymptotics, Conjectures and Monte Carlo, Ann. Stat. 1, 821 (1973).
24. A.W. van der Vaart, Asymptotic Statistics (Cambridge University Press, Cambridge, England, 2000), Vol. 3.
25. P. Huber and E. Ronchetti, Robust Statistics (Wiley, New York, 2009).
26. D. L. Donoho and M. Elad, Optimally Sparse Representation in General (Non-orthogonal) Dictionaries via L1 Minimization, Proc. Natl. Acad. Sci. U.S.A. 100, 2197 (2003).
27. E. Candes, J. Romberg, and T. Tao, Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information., IEEE Trans. Inf. Theory 52, 489 (2006).
28. E. Candes and T. Tao, Decoding by Linear Programming, IEEE Trans. Inf. Theory 51, 4203 (2005).
29. R. Tibshirani, Regression Shrinkage and Selection via the Lasso, J. R. Stat. Soc. Ser. B 58, 267 (1996).
30. D. Bean, P. J. Bickel, N. El Karoui, and B. Yu, Optimal M-estimation in High-Dimensional Regression, Proc. Natl. Acad. Sci. U.S.A. 110, 14563 (2013).
31. D. Donoho and A. Montanari, High Dimensional Robust M-estimation: Asymptotic Variance via Approximate Message Passing, Probab. Theory Relat. Fields, doi:10.1007/s00440-015-0675-z (2013).
32. D. Koller and N. Friedman, Probabilistic Graphical Models: Principles and Techniques (MIT Press, Cambridge, MA, 2009).
33. J. S. Yedidia, W. T. Freeman, and Y. Weiss, Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms, IEEE Trans. Inf. Theory 51, 2282 (2005).
34. M. Mezard and A. Montanari, Information, Physics, and Computation (Oxford University Press, New York, 2009).
35. D. L. Donoho, A. Maleki, and A. Montanari, Message-Passing Algorithms for Compressed Sensing, Proc. Natl. Acad. Sci. U.S.A. 106, 18914 (2009).
36. M. Bayati and A. Montanari, The Dynamics of Message Passing on Dense Graphs, with Applications to Compressed Sensing, IEEE Trans. Inf. Theory 57, 764 (2011).
37. S. Rangan, Generalized Approximate Message Passing for Estimation with Random Linear Mixing, in IEEE International Symposium on Information Theory Proceedings (ISIT), St. Petersburg, 2011 (IEEE, New York, 2011), pp. 2168-2172.
38. N. El Karoui, Asymptotic Behavior of Unregularized and Ridge-Regularized High-Dimensional Robust Regression Estimators: Rigorous Results, arXiv:1311.2445.
39. http://link.aps.org/supplemental/10.1103/PhysRevX.6.031034
40. E. J. G. Pitman, The Estimation of the Location and Scale Parameters of a Continuous Population of Any Given Form, Biometrika 30, 391 (1939).
41. R. D. Gill and B. Y. Levit, Applications of the van Trees Inequality: A Bayesian Cramér-Rao Bound, Bernoulli 1, 59 (1995).
42. M. Mezard, G. Parisi, and M. A. Virasoro, Spin Glass Theory and Beyond (World Scientific, Singapore, 1987).
43. N. Parikh and S. Boyd, Proximal Algorithms, Found. Trends. Optimization 1, 123 (2013).
44. V. A. Marchenko and L. A. Pastur, Distribution of Eigenvalues for Some Sets of Random Matrices, Mat. Sb. 114, 507 (1967).
45. E. L. Lehmann and H. Scheffé, Completeness, Similar Regions, and Unbiased Estimation: Part I, Ind. J. Stat. 10, 305 (1950).
46. C. Thrampoulidis, E. Abbasi, and B. Hassibi, Precise Error Analysis of Regularized M-estimators in High-Dimensions, arXiv:1601.06233.