Least squares superposition codes of moderate dictionary size are reliable at rates up to capacity

IEEE Transactions on Information Theory

Volumn 58, Issue 5, 2012, Pages 2541-2557

  Joseph, Antony ^a   Barron, Andrew R ^a  

^a YALE UNIVERSITY   (United States)

Author keywords

Achieving capacity; compressed sensing; exponential error bounds; Gaussian channel; maximum likelihood estimation; subset selection

Indexed keywords

ACHIEVING CAPACITY; COMPRESSED SENSING; EXPONENTIAL ERROR BOUNDS; GAUSSIAN CHANNELS; SUBSET SELECTION;

DECODING; TELECOMMUNICATION;

MAXIMUM LIKELIHOOD ESTIMATION;

EID: 84860253232     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2012.2184847     Document Type: Conference Paper

References (72)

1. A. Abbe and A. R. Barron, "Polar coding schemes for the AWGN channel," in Proc. IEEE Int. Symp. Inf. Theory, St. Petersburg, Russia, Aug. 2011, pp. 194-198.
2. M. Akcakaya and V. Tarokh, "Shannon-theoretic limits on noisy compressive sampling," IEEE Trans. Inf. Theory, vol. 56, no. 1, pp. 492-504, Jan. 2010.
3. E. Arikan, "Channel polarization," IEEE Trans. Inf. Theory, vol. 55, no. 7, pp. 3051-3073, Jul. 2009.
4. E. Arikan and E. Telatar, "On the rate of channel polarization," in Proc. IEEE Int. Symp. Inf. Theory, Seoul, Korea, Jul. 2009, pp. 1493-1495.
5. A. Barg and G. Zémor, "Error exponents of expander codes under linear-complexity decoding," SIAM J. Discrete Math., vol. 17, no. 3, pp. 426-445, 2004.
6. A. R. Barron, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 930-944, May 1993.
7. A. Barron, A. Cohen, W. Dahmen, and R. Devore, "Approximation and learning by greedy algorithms," Ann. Statist., vol. 36, no. 1, pp. 64-94, 2007.
8. A. R. Barron and A. Joseph, Sparse superposition codes: Fast and reliable at rates approaching capacity with Gaussian noise [Online]. Available: http://www.stat.yale.edu/arb4
9. A. R. Barron and A. Joseph, "Towards fast reliable communication at rates near capacity with Gaussian noise," in Proc. IEEE. Int. Symp. Inf. Theory, Austin, TX, Jun. 13-18, 2010, pp. 315-319.
10. A. R. Barron and A. Joseph, "Least squares superposition codes of moderate dictionary size, reliable at rates up to capacity," in Proc. IEEE. Int. Symp. Inf. Theory, Austin, TX, Jun. 13-18, 2010, pp. 275-279.
11. A. R. Barron and A. Joseph, Least squares superposition codes of moderate dictionary size, reliable at rates up to capacity [Online].Available: http://arxiv.org/abs/1006.3780
12. G. Berrou, A. Glavieux, and P. Thitimajshima, "Near Shannon limit error-correcting coding: turbo codes," in Proc. Int. Conf. Commun., Geneva, Switzerland, May 1993, pp. 1064-1070.
13. T. Blumensath and M. E. Davies, "Iterative thresholding for sparse approximations," J. Fourier Anal. Appl., vol. 14, pp. 629-654, 2008.
14. L. Applebaum, W. U. Bajwa, M. F. Duarte, and R. Calderbank, "Multiuser detection in asynchronous on-off random access channels using lasso," in Proc. 48th Annu. Allerton Conf. Commun., Control, Comput., Sep. 2010, pp. 130-137.
15. E. Candes and Y. Plan, "Near-ideal model selection by g minimization," Ann. Statist., vol. 37, no. 5A, pp. 2145-2177, 2009.
16. E. Candes and T. Tao, "Decoding by linear programming," IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005.
17. E. Candes and T. Tao, "Near-optimal signal recovery from random projections: Universal encoding strategies?," IEEE Trans. Inf. Theory, vol. 52, no. 12, pp. 5406-5425, Dec. 2006.
18. J. Cao and E. M. Yeh, "Asymptotically optimal multiple-access communication via distributed rate splitting," IEEE Trans. Inf. Theory, vol. 53, no. 1, pp. 304-319, Jan. 2007.
19. T. M. Cover, "Broadcast channels," IEEE Trans. Inf. Theory, vol. 18, no. 1, pp. 2-14, Jan. 1972.
20. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley-Interscience, 2006.
21. I. Daubechies, M. Defrise, and C. De Mol, "An iterative thresholding algorithm for linear inverse problems with a sparsity constraint," Commun. Pure Appl. Math., vol. 57, no. 11, pp. 1413-1457, 2004.
22. D. Donoho, "For most large underdetermined systems of linear equations the minimal α-norm near solution approximates the sparest solution," Commun. Pure Appl. Math., vol. 59, no. 10, pp. 907-934, 2006.
23. D. L. Donoho, M. Elad, and V. M. Temlyakov, "Stable recovery of sparse overcomplete representations in the presence of noise," IEEE Trans. Inf. Theory, vol. 52, no. 1, pp. 6-18, Jan. 2006.
24. D. Donoho and X. Huo, "Uncertainty principles and ideal atomic decomposition," IEEE Trans. Inf. Theory, vol. 47, no. 7, pp. 2845-2862, Nov. 2001.
25. D. L. Donoho and J. Tanner, "Exponential bounds implying construction of compressed sensing matrices, error-correcting codes, and neighborly polytopes by random sampling," IEEE Trans. Inf. Theory, vol. 56, no. 4, pp. 2002-2016, Apr. 2010.
26. P. Elias, "Coding for noisy channels," in IRE Conv. Rec., Mar. 1955, pp. 37-46.
27. A. K. Fletcher, S. Rangan, and V. K. Goyal, On-off random access channels: A compressed sensing framework 2009 [Online]. Available: arXiv:0903.1022v2
28. A. K. Fletcher, S. Rangan, V. K. Goyal, and K. Ramchandran, "Denoising by sparse approximation: Error bounds based on rate-distortion theory," J. Appl. Signal Process., vol. 10, pp. 1-19, 2006.
29. A. K. Fletcher, S. Rangan, and V. K. Goyal, "Necessary and sufficient conditions on sparsity pattern recovery," IEEE Trans. Inf. Theory, vol. 55, no. 12, pp. 5758-5772, Dec. 2009.
30. G. D. Forney Jr., Concatenated Codes. Cambridge, MA: MIT Press, 1966, vol. 37, Research Monograph.
31. G. D. Forney Jr. and G. Ungerboeck, "Modulation and coding for linear Gaussian channels," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2384-2415, Oct. 1998.
32. J. J. Fuchs, "On sparse representations in arbitrary redundant bases," IEEE Trans. Inf. Theory, vol. 50, no. 6, pp. 1341-1344, Jun. 2004.
33. J. J. Fuchs, "Recovery of exact sparse representations in the presence of noise," in Proc. Int. Conf. Acoust., Speech Signal Process., Montreal, QC, Canada, 2004, vol. 2, pp. 533-536.
34. R. G. Gallager, Low Density Parity-Check Codes. Cambridge, MA: MIT Press, 1963.
35. R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968.
36. A. C. Gilbert and J. A. Tropp, "Applications of sparse approximation in communications," in Proc. IEEE. Int. Symp. Inf. Theory, Adelaide, Australia, Sep. 2005, pp. 1000-1004.
37. C. He, M. Lentmaier, D. J. Costello, and K. S. Zigangirov, "Joint permutor analysis and design for multiple turbo codes," IEEE Trans. Inf. Theory, vol. 52, no. 9, pp. 4068-4083, Sep. 2006.
38. C. Huang, A. R. Barron, and G. H. L. Cheang, Risk of penalized least squares, greedy selection and L1 penalization for flexible function libraries 2008 [Online]. Available: http://www.stat.yale.edu/arb4
39. W. Hoeffding, "Probability inequalities for sums of bounded random variables," J. Amer. Statist. Assoc., vol. 58, pp. 13-30, Mar. 1963.
40. J. X. Hu, H. Zhao, and H. H. Zhou, "False discovery rate control with groups," J. Amer. Statist. Assoc., vol. 105, pp. 1215-1227, Sep. 2010.
41. L. Jones, "A simple lemma for optimization in a Hilbert space, with application to projection pursuit and neural net training," Ann. Statist., vol. 20, pp. 608-613, 1992.
42. I. Kontoyiannis, S. Gitzenis, and K. R. Rad, "Superposition codes for Gaussian vector quantization," in IEEE Inf. Theory Workshop, Cairo, Egypt, Jan. 2010.
43. A. M. Kakhaki, H. K. Abadi, P. Pad, H. Saeedi, K. Alishahi, and F. Marvasti, Capacity achieving random sparse linear codes 2011 [Online]. Available: arXiv:1102.4099v1
44. W. S. Lee, P. Bartlett, and B. Williamson, "Efficient agnostic learning of neural networks with bounded fan-in," IEEE Trans. Inf. Theory, vol. 42, no. 6, pp. 2118-2132, Nov. 1996.
45. S. Lin and D. Costello, Error Control Coding: Fundamentals and Applications. Englewood Cliffs, NJ: Prentice-Hall, 2004.
46. M. G. Luby,M.Mitzenmacher,M. A. Shokrollahi, and D. A. Spielman, "Efficient erasure correcting codes," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 569-584, Feb. 2001.
47. M. G. Luby,M.Mitzenmacher,M. A. Shokrollahi, and D. A. Spielman, "Improved low-density parity-check codes using irregular graphs," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 585-598, Feb. 2001.
48. F. J. MacWilliams and N. J. A. Sloane, The Theory of Error-Correcting Codes. Amsterdam, The Netherlands: North Holland, 1981.
49. S. Mallat and Z. Zhang, "Matching pursuit with time-frequency dictionaries," IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, Dec. 1993.
50. R. J. McEliece, D. J. C. MacKay, and J. F. Cheng, "Turbo decoding as an instance of Pearl's belief propagation algorithm," IEEE J. Sel. Areas Commun., vol. 16, no. 2, pp. 140-152, Feb. 1998.
51. D. Needell and J. Tropp, "CoSaMP: Iterative signal recovery from incomplete and inaccurate samples," Appl. Comput. Harmon. Anal., vol. 26, no. 3, pp. 301-321, 2009.
52. Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, "Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition," in Proc. 27th Annu. Asilomar Conf. Signals, Syst., Comput., Pacific Grove, CA, Nov. 1993, pp. 40-44.
53. Y. Polyanskiy, H. V. Poor, and S. Verdu, "Channel coding rate in the finite block length regime," IEEE Trans. Inf. Theory, vol. 56, no. 5, pp. 2307-2359, May 2010.
54. L. Perez, J. Seghers, and D. J. Costello Jr., "A distance spectrum interpretation of turbo codes," IEEE Trans. Inf. Theory, vol. 42, no. 6, pp. 1698-1709, Nov. 1996.
55. I. S. Reed and G. Solomon, "Polynomial codes over certain finite fields," J. SIAM, vol. 8, pp. 300-304, Jun. 1960.
56. G. Reeves and M. Gastpar, "Sampling bounds for sparse support recovery in the presence of noise," in Proc. IEEE Int. Symp. Inf. Theory, Toronto, ON, Canada, Jul. 2008, pp. 2187-2191.
57. T. Richardson, A. Shokrollahi, and R. Urbanke, "Design of capacityapproaching irregular low-density parity-check codes," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 619-637, Feb. 2001.
58. T. Richardson and R. Urbanke, "The capacity of low-density parity check codes under message-passing decoding," IEEE Trans. Inform. Theory, vol. 47, no. 2, pp. 599-618, Feb. 2001.
59. B. Rimoldi and R. Urbanke, "A rate-splitting approach to the Gaussian multiple-access channel capacity," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 364-375, Mar. 2001.
60. C. E. Shannon, "A mathematical theory of communication," Bell Syst. Tech. J., vol. 27, pp. 379-423, 1948.
61. S. J. Szarek, "Condition numbers of random matrices," J. Complexity, vol. 7, pp. 131-149, 1991.
62. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Roy. Statist. Soc., ser. B, vol. 58, no. 1, pp. 267-288, 1996.
63. J. A. Tropp, "Greed is good: Algorithmic results for sparse approximation," IEEE Trans Inf. Theory, vol. 50, no. 10, pp. 2231-2242, Oct. 2004.
64. J. Tropp, "Just relax: Convex programming methods for identifying sparse signals in noise," IEEE Trans. Inf. Theory, vol. 52, no. 3, pp. 1030-1051, Mar. 2006.
65. J. A. Tropp, "On the conditioning of random subdictionaries," Appl. Comput. Harmon. Anal., vol. 25, pp. 1-24, 2008.
66. J. A. Tropp, "Norms of random submatrices and sparse approximation," C. R. Math. Acad. Sci. Paris, vol. 346, no. 23, pp. 1271-1274, 2008.
67. M. J.Wainwright, "Information-theoretic limits on sparsity recovery in the high-dimensional and noisy setting," IEEE Trans. Inf. Theory, vol. 55, no. 12, pp. 5728-5741, Dec. 2009.
68. M. J. Wainwright, "Sharp thresholds for high-dimensional and noisy sparsity recovery using α-constrained quadratic programming (lasso)," IEEE Trans. Inf. Theory, vol. 55, no. 5, pp. 2183-2202, May 2009.
69. W. Wang, M. J. Wainwright, and K. Ramchandran, "Information- theoretic limits on sparse signal recovery: Dense versus sparse measurement matrices," IEEE Trans. Inf. Theory, vol. 56, no. 6, pp. 2967-2979, Jun. 2010.
70. T. Zhang, "On the consistency of feature selection using greedy least squares regression," J. Mach. Learn. Res., vol. 10, pp. 555-568, 2009.
71. T. Zhang, "Adaptive forward-backward greedy algorithm for learning sparse representations," IEEE Trans. Inf. Theory, vol. 57, no. 7, pp. 4689-4708, Jul. 2011.
72. P. Zhao and B. Yu, "On model selection consistency of lasso," J. Mach. Learn. Res., no. 7, pp. 2541-2567, 2006.