Fast sparse superposition codes have near exponential error probability for R< C

IEEE Transactions on Information Theory

Volumn 60, Issue 2, 2014, Pages 919-942

  Joseph, Antony ^a   Barron, Andrew R ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b YALE UNIVERSITY   (United States)

Author keywords

achieving channel capacity; compressed sensing; error exponents; Gaussian channel; greedy algorithms; multiuser detection; orthogonal matching pursuit; subset selection; successive cancelation decoding

Indexed keywords

ALGORITHMS; COMPRESSED SENSING; MULTIUSER DETECTION; TELECOMMUNICATION;

ERROR EXPONENT; GAUSSIAN CHANNELS; GREEDY ALGORITHMS; ORTHOGONAL MATCHING PURSUIT; SUBSET SELECTION;

DECODING;

EID: 84893235559     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2013.2289865     Document Type: Article

References (47)

1. A. Abbe and A. R. Barron, "Polar codes for the AWGN, " in Proc. IEEE ISIT, Aug. 2011, pp. 194-198.
2. M. Akçakaya and V. Tarokh, "Shannon-theoretic limits on noisy com-pressive sampling, " IEEE Trans. Inf. Theory, vol. 56, no. 1, pp. 492-504, Jan. 2010.
3. E. Arikan, "Channel polarization: A method for constructing capacity-achieving codes for symmetric binary-input memoryless channels, " IEEE Trans. Inf. Theory, vol. 55, no. 7, pp. 3051-3073, Jul. 2009.
4. E. Arikan and E. Telatar, "On the rate of channel polarization, " in Proc. IEEE ISIT, Apr. 2009, pp. 1493-1495.
5. A. R. Barron, "Universal approximation bounds for superpositions of a sigmoidal function, " IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 930-945, May 1993.
6. A. R. Barron and S. Cho, "High-rate sparse superposition codes with iteratively optimal estimates, " in Proc. IEEE ISIT, Jul. 2012, pp. 120-124.
7. A. R. Barron, A. Cohen, W. Dahmen, and R. A. DeVore, "Approximation and learning by greedy algorithms, " Ann. Statist., vol. 36, no. 1, pp. 64-94, 2008.
8. A. R. Barron and A. Joseph, "Toward fast reliable communication at rates near capacity with Gaussian noise, " in Proc. IEEE ISIT, Jun. 2010, pp. 315-319.
9. A. R. Barron and A. Joseph, "Analysis of fast sparse superposition codes, " in Proc. IEEE ISIT, Aug. 2011, pp. 1772-1776.
10. A. R. Barron and A. Joseph, "Sparse superposition codes are fast and reliable at rates approaching capacity with Gaussian noise, " Dept. Statist., Yale Univ., New Haven, CT, USA, Tech. Rep., 2011.
11. M. Bayati and A. Montanari, "The dynamics of message passing on dense graphs, with applications to compressed sensing, " IEEE Trans. Inf. Theory, vol. 57, no. 2, pp. 764-785, Feb. 2011.
12. M. Bayati and A. Montanari, "The LASSO risk for Gaussian matrices, " IEEE Trans. Inf. Theory, vol. 58, no. 4, pp. 1997-2017, Apr. 2012.
13. C. Berrou, A. Glavieux, and P. Thitimajshima, "Near Shannon limit error-correcting coding and decoding: Turbo-codes. 1, " in Proc. IEEE ICC, vol. 2. May 1993, pp. 1064-1070. (Pubitemid 23674800)
14. 1 minimization, " Ann. Statist., vol. 37, no. 5A, pp. 2145-2177, 2009.
15. S. S. Chen, D. L. Donoho, and M. A. Saunders, "Atomic decomposition by basis pursuit, " SIAM Rev., vol. 43, no. 1, pp. 129-159, 2001. (Pubitemid 32406896)
16. T. Cover, "Broadcast channels, " IEEE Trans. Inf. Theory, vol. 18, no. 1, pp. 2-14, Jan. 1972.
17. T. M. Cover and J. A. Thomas, Elements of Information Theory, vol. 6. New York, NY, USA: Wiley, 1991.
18. I. Csiszár, "Sanov property, generalized I-projection and a conditional limit theorem, " Ann. Probab., vol. 12, no. 3, pp. 768-793, 1984.
19. A. K. Fletcher, V.K. Goyal, and S. Rangan, "A sparsity detection framework for on-off random access channels, " in Proc. IEEE ISIT, Jul. 2009, pp. 169-173.
20. A. K. Fletcher and S. Rangan, "Orthogonal matching pursuit: A Brownian motion analysis, " IEEE Trans. Signal Process., vol. 60, no. 3, pp. 1010-1021, Mar. 2012.
21. G. David Forney, "Concatenated codes, " DTIC, Fort Belvoir, VA, USA, Tech. Rep., 1965.
22. R. Gallager, "Low-density parity-check codes, " IRE Trans. Inf. Theory, vol. 8, no. 1, pp. 21-28, Jan. 1962.
23. 1 penalization for flexible function libraries, " Ph.D. dissertation, Dept. Statist., Yale Univ., New Haven, CT, USA, Nov. 2008.
24. L. Jones, "A simple lemma for optimization in a Hilbert space, with application to projection pursuit and neural net training, " Ann. Statist., vol. 20, no. 1, pp. 608-613, Mar. 1992.
25. A. Joseph, "Achieving information-theoretic limits with high-dimensional regression, " Ph.D. dissertation, Yale Univ., New Haven, CT, USA, Jun. 2012.
26. A. Joseph, "Variable selection in high dimensions with random designs and orthogonal matching pursuit, " J. Mach. Learn. Res., pp. 1771-1800, 2013.
27. A. Joseph and A. R. Barron, "Least squares superposition coding of moderate dictionary size are reliable at rates up to channel capacity, " IEEE Trans. Inf. Theory, vol. 58, no. 5, pp. 2541-2557, May 2012.
28. I. Kontoyiannis, S. Gitzenis, and K. R. Rad, "Superposition codes for Gaussian vector quantization, " in Proc. IEEE Inf. Theory Workshop, Jan. 2010, pp. 368-372.
29. W. S. Lee, P. L. Bartlett, and R. C. Williamson, "Efficient agnostic learning of neural networks with bounded fan-in, " IEEE Trans. Inf. Theory, vol. 42, no. 6, pp. 2118-2132, Nov. 1996. (Pubitemid 126771540)
30. S. G. Mallat and Z. Zhang, "Matching pursuits with time-frequency dictionaries, " IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, Dec. 1993.
31. Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, "Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition, " in Proc. Conf. Rec. 27th Asilomar Conf. Signals, Syst. Comput., Nov. 1993, pp. 40-44.
32. Y. Polyanskiy, H. V. Poor, and S. Verdú, "Channel coding rate in the finite blocklength regime, " IEEE Trans. Inf. Theory, vol. 56, no. 5, pp. 2307-2359, May 2010.
33. G. Reeves and M. Gastpar, "Approximate sparsity pattern recovery: Information-theoretic lower bounds, " IEEE Trans. Inf. Theory, vol. 59, no. 6, pp. 3451-3465, Jun. 2013.
34. G. Reeves and M. Gastpar, "Fundamental tradeoffs for sparsity pattern recovery, " Inf. Transf. Manag., Jun. 2010.
35. G. Reeves and M. Gastpar, "The sampling rate-distortion tradeoff for sparsity pattern recovery in compressed sensing, " IEEE Trans. Inf. Theory, vol. 58, no. 5, pp. 3065-3092, May 2012.
36. C. E. Shannon, "A mathematical theory of communication, " ACM SIGMOBILE Mobile Comput. Commun. Rev., vol. 5, no. 1, pp. 3-55, 2001.
37. R. Tibshirani, "Regression shrinkage and selection via the lasso, " J. R. Statist. Soc. Ser. B, Statist. Methodol., vol. 58, no. 1, pp. 267-288, Jan. 1996.
38. J. A. Tropp, "Greed is good: Algorithmic results for sparse approximation, " IEEE Trans. Inf. Theory, vol. 50, no. 10, pp. 2231-2242, Oct. 2004.
39. J. A. Tropp, "Just relax: Convex programming methods for identifying sparse signals in noise, " IEEE Trans. Inf. Theory, vol. 52, no. 3, pp. 1030-1051, Mar. 2006. (Pubitemid 46444890)
40. J. A. Tropp and A. C. Gilbert, "Signal recovery from random measurements via orthogonal matching pursuit, " IEEE Trans. Inf. Theory, vol. 53, no. 12, pp. 4655-4666, Dec. 2007.
41. R. Venkataramanan, A. Joseph, and S. Tatikonda, "Gaussian rate-distortion via sparse linear regression over compact dictionaries, " in Proc. IEEE ISIT, Jul. 2012, pp. 368-372.
42. R. Venkataramanan, T. Sarkar, and S. Tatikonda, "Lossy compression via sparse linear regression: Computationally efficient encoding and decoding, " Dec. 2012.
43. M. J. Wainwright, "Information-theoretic limits on sparsity recovery in the high-dimensional and noisy setting, " IEEE Trans. Inf. Theory, vol. 55, no. 12, pp. 5728-5741, Dec. 2009.
44. 1-constrained quadratic programming (Lasso), " IEEE Trans. Inf. Theory, vol. 55, no. 5, pp. 2183-2202, May 2009.
45. C. H. Zhang, "Nearly unbiased variable selection under minimax concave penalty, " Ann. Statist, vol. 38, no. 2, pp. 894-942, 2010.
46. T. Zhang, Adaptive Forward-Backward Greedy Algorithm for Sparse Learning with Linear Models. Lake Tahoe, NV, USA: NIPS, 2008.
47. P. Zhao and B. Yu, "On model selection consistency of lasso, " J. Mach. Learn. Res., vol. 7, pp. 2541-2563, Nov. 2006. (Pubitemid 44866738)