Binary Linear Classification and Feature Selection via Generalized Approximate Message Passing

IEEE Transactions on Signal Processing

Volumn 63, Issue 8, 2015, Pages 2020-2032

  Ziniel, Justin ^a   Schniter, Philip ^a   Sederberg, Per ^b  

^a Ohio State University   (United States)

^b OHIO STATE UNIVERSITY   (United States)

Author keywords

Belief propagation; classification; feature selection; message passing; one bit compressed sensing

Indexed keywords

ALGORITHMS; CHANNEL ESTIMATION; COMPRESSED SENSING; FEATURE EXTRACTION; IMAGE SEGMENTATION; MAXIMUM PRINCIPLE; MESSAGE PASSING; SIGNAL RECONSTRUCTION;

ALGORITHMIC APPROACH; BELIEF PROPAGATION; CLASSIFICATION ERROR RATE; COMPRESSIVE SENSING; EXPECTATION MAXIMIZATION; LINEAR CLASSIFICATION; MISCLASSIFICATION RATES; ONE BIT COMPRESSED SENSING;

CLASSIFICATION (OF INFORMATION);

EID: 84925193449     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2015.2407311     Document Type: Article

References (51)

1. C. M. Bishop, Pattern Recognition and Machine Learning. New York, NY, USA: Springer-Verlag, 2006.
2. G. Forman, "An extensive empirical study of feature selection metrics for text classification," J. Mach. Learn. Res., vol. 3, pp. 1289-1305, 2003.
3. J. V. Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pietrini, "Distributed and overlapping representations of faces and objects in ventral temporal cortex," Sci., vol. 293, pp. 2425-2430, Sep. 2001.
4. K. A. Norman, S. M. Polyn, G. J. Detre, and J. V. Haxby, "Beyond mind-reading: Multi-voxel pattern analysis of fMRI data," Trends Cognit. Sci., vol. 10, pp. 424-430, Sep. 2006.
5. F. Pereira, T. Mitchell, and M. Botvinick, "Machine learning classifiers and fMRI: A tutorial overview," NeuroImage, vol. 45, pp. S199-S209, Mar. 2009.
6. A. Gustafsson, A. Hermann, and F. Huber, Conjoint Measurement: Methods and Applications. Berlin, Germany: Springer-Verlag, 2007.
7. E. P. Xing, M. I. Jordan, and R. M. Karp, "Feature selection for high-dimensional genomic microarray data," in Proc. Int. Workshop Mach. Learn., 2001, pp. 601-608.
8. P. T. Boufounos and R. G. Baraniuk, "1-bit compressive sensing," presented at the Conf. Inf. Sci. Sys., Princeton, NJ, USA, Mar. 2008.
9. Y. Plan and R.Vershynin, "Robust 1-bit compressed sensing and sparse logistic regression: A convex programming approach," IEEE Trans. Inf. Theory, vol. 59, no. 1, pp. 482-494, 2013.
10. th Int. Conf.Mach. Learn. (ICML), Bari, Italy, 1996, pp. 284-292.
11. R. Kohavi and G. John, "Wrapper for feature subset selection," Artif. Intell., vol. 97, pp. 273-324, 1997.
12. M. E. Tipping, "Sparse Bayesian learning and the relevance vector machine," J. Mach. Learn. Res., vol. 1, pp. 211-244, 2001.
13. th Conf. Adv. Neural Inf. Process. Syst., Cambridge, MA, USA, 2001, pp. 697-704.
14. M. Figueiredo, "Adaptive sparseness for supervised learning," IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), vol. 25, no. 9, pp. 1150-1159, 2003.
15. A. Kabán, "On Bayesian classification with Laplace priors," Pattern Recognit. Lett., vol. 28, no. 10, pp. 1271-1282, 2007.
16. H. Chen, P. Tino, and X. Yao, "Probabilistic classification vector machines," IEEE Trans. Neural Net., vol. 20, no. 6, pp. 901-914, 2009.
17. G.-X. Yuan, K.-W. Chang, C.-J. Hsieh, and C.-J. Lin, "A comparison of optimization methods and software for large-scale L1-regularized linear classification," J. Mach. Learn. Res., vol. 11, pp. 3183-3234, 2010.
18. A. Gupta, R. Nowak, and B. Recht, "Sample complexity for 1-bit compressed sensing and sparse classification," presented at the Int. Symp. Inf. Theory (ISIT), Austin, TX, 2010.
19. J. N. Laska, Z. Wen, W. Yin, and R. G. Baraniuk, "Trust, but verify: Fast and accurate signal recovery from 1-bit compressive measurements," IEEE Trans. Signal Process., vol. 59, no. 11, pp. 5289-5301, 2011.
20. U. S. Kamilov, A. Bourquard, A. Amini, and M. Unser, "One-bit measurements with adaptive thresholds," IEEE Signal Process. Lett., vol. 19, pp. 607-610, 2012.
21. U. S. Kamilov, V. K. Goyal, and S. Rangan, "Message-passing de-quantization with applications to compressed sensing," IEEE Trans. Signal Process., vol. 60, pp. 6270-6281, Dec. 2012.
22. S. Rangan, "Generalized approximate message passing for estimation with random linear mixing," in Proc. IEEE Int. Symp. Inf. Theory, St. Petersburg, Russia, Aug. 2011, pp. 2168-2172.
23. D. L. Donoho, A. Maleki, and A. Montanari, "Message passing algorithms for compressed sensing," in Proc. Nat. Acad. Sci., Nov. 2009, vol. 106, pp. 18914-18919.
24. D. L. Donoho, A. Maleki, and A. Montanari, "Message passing algorithms for compressed sensing: I. Motivation and construction," in Proc. Inf. Theory Workshop, Jan. 2010, pp. 1-5.
25. A. Javanmard and A. Montanari, "State evolution for general approximate message passing algorithms, with applications to spatial coupling," Inf. Inference, vol. 2, no. 2, pp. 115-144, 2013.
26. J. Ziniel, S. Rangan, and P. Schniter, "A generalized framework for learning and recovery of structured sparse signals," presented at the IEEE Statist. Signal Process. Workshop, Ann Arbor, MI, USA, Aug. 2012.
27. J. P. Vila and P. Schniter, "Expectation-Maximization Gaussian-mixture approximate message passing," IEEE Trans. Signal Process., vol. 61, pp. 4658-4672, Oct. 2013.
28. P. Schniter, "Turbo reconstruction of structured sparse signals," in Proc. Conf. Inf. Sci. Syst. (CISS), Princeton, NJ, USA, Mar. 2010, pp. 1-6.
29. B. J. Frey and D. J. C. MacKay, "A revolution: Belief propagation in graphs with cycles," Adv. Neural Inf. Process. Syst., pp. 479-485, 1998.
30. F. R. Kschischang, B. J. Frey, and H. A. Loeliger, "Factor graphs and the sum-product algorithm," IEEE Trans. Inf. Theory, vol. 47, pp. 498-519, Feb. 2001.
31. R. J. McEliece, D. J. C. MacKay, and J. Cheng, "Turbo decoding as an instance of Pearl's belief propagation algorithm," IEEE J. Sel. Areas Commun., vol. 16, pp. 140-152, Feb. 1998.
32. W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, "Learning low-level vision," Int. . J. Comput. Vis., vol. 40, pp. 25-47, Oct. 2000.
33. S. Rangan, P. Schniter, E. Riegler, A. K. Fletcher, and V. Cevher, "Fixed points of generalized approximate message passing with arbitrary matrices," in Proc. IEEE Int. Symp. Inf. Theory, Istanbul, Jul. 2013, pp. 664-668.
34. S. Rangan, P. Schniter, and A. Fletcher, "On the convergence of generalized approximate message passing with arbitrary matrices," in Proc. IEEE Int. Symp. Inf. Theory, Honolulu, HI, USA, Jul. 2014, pp. 236-240.
35. H.-A. Loeliger, "An introduction to factor graphs," IEEE Signal Process. Mag., vol. 21, pp. 28-41, Jan. 2004.
36. J. Ziniel, "Message passing approaches to compressive inference under structured signal priors," Ph.D. dissertation, The Ohio State Univ., Columbus, OH, USA, 2014.
37. M. I. Jordan, "Why the logistic function? A tutorial discussion on probabilities and neural networks," MIT Comput. Congnit. Sci., Cambridge, MA, USA, Tech. Rep. 9503, Aug. 13, 1995.
38. C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. Cambridge, MA, ISA: MIT Press, 2006.
39. J. Bi, K. Bennett, M. Embrechts, C. Breneman, and M. Song, "Dimensionality reduction via sparse support vector machines," J. Mach. Learn. Res., vol. 3, pp. 1229-1243, 2003.
40. M. Opper and O. Winther, Gaussian Processes and SVM: Mean Field Results and Leave-One-Out Estimator . Cambridge, MA, USA: MIT Press, 2000, ch. 17, pp. 311-326.
41. H. Zou and T. Hastie, "Regularization and variable selection via the elastic net," J. Roy. Statist. Soc., B, vol. 67, no. 2, pp. 301-320, 2005.
42. A. K. Nigam, K. McCallum, S. Thrun, and T.Mitchell, "Text classification from labeled and unlabeled documents using EM," Mach. Learn., vol. 39, pp. 103-134, 2000.
43. J. P. Vila and P. Schniter, "An empirical-Bayes approach to recovering linearly constrained non-negative sparse signals," IEEE Trans. Signal Process., vol. 62, pp. 4689-4703, Sep. 2014.
44. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. Roy. Statist. Soc., B, vol. 39, pp. 1-38, 1977.
45. U. S. Kamilov, S. Rangan, A. K. Fletcher, and M. Unser, "Approximate message passing with consistent parameter estimation and applications to sparse learning," in Proc. Neural Inf. Process. Syst. Conf., Lake Tahoe, NV, USA, Dec. 2012, pp. 2447-2455.
46. D. D. Lewis,Y.Yang, T.G. Rose, and F. Li, "RCV1: A new benchmark collection for text categorization research," J. Mach. Learn. Res., vol. 5, pp. 361-397, 2004.
47. th Int. Conf. Mach. Learn., Corvallis, OR, USA, 2007, pp. 561-568.
48. S. R. Becker, E. J. Candès, and M. C. Grant, "Templates for convex cone problems with applications to sparse signal recovery," Math. Prog. Comput., vol. 3, no. 3, pp. 165-218, 2011.
49. C. J. Lin and J. J. Moré, "Newton's method for large-scale bound constrained problems," SIAM J. Optim., vol. 9, pp. 1100-1127, 1999.
50. P. Schniter and S. Rangan, "Compressive phase retrieval via generalized approximate message passing," presented at the Allerton Conf. Commun., Control, Comput., Monticello, IL, USA, Oct. 2012.
51. D. R. Barr and E. T. Sherrill, "Mean and variance of truncated normal distributions," Amer. Statist., vol. 53, Nov. 1999.