A collaborative training algorithm for distributed learning

IEEE Transactions on Information Theory

Volumn 55, Issue 4, 2009, Pages 1856-1871

  Predd, Joel B ^a   Kulkarni, Sanjeev R ^b   Poor, H Vincent ^b  

^a RAND CORPORATION   (United States)

^b Princeton University   (United States)

Author keywords

Collaboration; Distributed learning; Empirical risk minimization; Kernel methods; Learning; Nonparametric; Sensor networks

Indexed keywords

COLLABORATION; DISTRIBUTED LEARNING; EMPIRICAL RISK MINIMIZATION; KERNEL METHODS; LEARNING; NONPARAMETRIC;

COMPUTATIONAL FLUID DYNAMICS; DISTRIBUTED COMPUTER SYSTEMS; EDUCATION; ELECTRIC NETWORK TOPOLOGY; SENSOR NETWORKS; WIRELESS SENSOR NETWORKS;

LEARNING ALGORITHMS;

EID: 64249136579     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2009.2012992     Document Type: Article

References (52)

1. T. Davis, "Sparse Matrix. From MathWorld - A Wolfram Web Resources," Aug. 16, 2008 [Online]. Available: http://mathworld.wolfram. com/SparseMatrix.html, created by Eric W. Weisstien
2. M. G. Rabbat, R. D. Nowak, and J. A. Bucklew, "Generalized consensus computation in networked systems with erasure links," in Proc. IEEE Workshop on Signal Processing Advances in Wireless Communications, New York, Jun. 2005, pp. 1088-1092.
3. T. Kailath, "An RKHS approach to detection and estimation problems. Pt. I: Deterministic signals in Gaussian noice," IEEE Trans. Inf. Theory vol. IT-17, no. 5, pp. 530-549, Sep. 1971.
4. N. Aronszajn, "Theory of reproducing kernels," Trans. Amer. Math. Soc., vol. 68, no. 3, pp. 337-404, 1950.
5. S. M. Aji and R. J. McEliece, "The generalized distributive law," IEEE Trans. Inf. Theory, vol. 46, no. 2, pp. 325-343, Mar. 2000.
6. S. Bandyopadhyay and E. Coyle, "An energy efficient hierarchical clustering algorithm for wireless sensor networks," in Proc. 22nd Annu. Joint Conf. IEEE Computer and Communications Societies (Infocom San Francisco, CA, Mar./Apr. 2003, vol. 3, pp. 1713-1723.
7. H. H. Bauschke and J. M. Borwein, "On projection algorithms for solving convex feasibility problems," SIAM Rev., vol. 38, no. 3, pp. 367-426, Sep. 1996.
8. D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods. Belmont, MA: Athena Scientific, 1997.
9. O. Bousquet and A. Elisseeff, "Stability and generalization," J. Mach. Learning Res., vol. 2, pp. 499-526, 2002.
10. L. M. Bregman, "The relaxation method of finding a common point of convex sets andits application to the solution of problems in convex programming," U. S. S. R. Comput. Math. Math. Phys., vol. 78, no. 384, pp. 200-217, 1967.
11. Y. Censor and S. A. Zenios, Parallel Optimization: Theory, Algorithms, and Applications. New York: Oxford Univ. Press, 1997.
12. M. Cetin, L. Chen, J. W. Fisher III, A. T. Ihler, R. L. Moses, M. J. Wainwright, and A. S.Willsky, "Distributed fusion in sensor networks: A graphical models perspective," IEEE Signal Process. Mag. (Special Issue on Distributed Signal Processing in Wireless Sensor Networks), vol. 23, no. 4, pp. 42-55, Jul. 2006.
13. V. Delouille, R. Neelamani, and R. Baraniuk, "Robust distributed estimation in sensor networks using the embedded polygons algorithm," in Proc. 3rd Int. Symp. Information Processing in Sensor Networks, Berkeley, CA, Apr. 2004, pp. 405-413.
14. L. Devroye, L. Györfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition. New York: Springer-Verlag, 1996.
15. Y. Freund and R. E. Schapire, "A decision-theoretic generalization of on-line learning and an application to boosting," Comput. Syst. Scie., vol. 55, no. 1, pp. 119-139, 1997.
16. S. Gambs, B. Kégl, and E. Aïmeur, "Privacy-Preserving Boosting," 2005, preprint.
17. S. Gezici, Z. Tian, G. B. Giannakis, H. Kobayashi, A. F. Molisch, H. V. Poor, and Z. Sahinoglu, "Localization via ultra-wideband radios," IEEE Signal Process. Mag., vol. 22, no. 4, pp. 70-84, Jul. 2005.
18. G. Golub and C. Van Loan, Matrix Computations. Baltimore, MD: Johns Hopkins Univ. Press, 1989.
19. C. Guestrin, P. Bodi, R. Thibau, M. Paskin, and S. Madde, "Distributed regression: An efficient framework for modeling sensor network data," in Proc. 3rd Int. Symp. Information Processing in Sensor Networks, Berkeley, CA, Apr. 2004, pp. 1-10.
20. P. Gupta and P. R. Kumar, "Capacity of wireless networks," IEEE Trans. Inf. Theory, vol. 46, no. 2, pp. 388-401, Mar. 2000.
21. L. Györfi, M. Kohler, A. Krzyzak, and H. Walk, A Distribution-Free Theory of Nonparametric Regression. New York: Springer-Verlag, 2002.
22. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer-Verlag, 2001.
23. M. Jordan, Ed., Learning in Graphical Models. Cambridge, MA: MIT Press, 1999.
24. G. Kimeldorf and G. Wahba, "Some results on Tchebycheffian spline functions," J. Math. Anal. Applic., vol. 33, no. 1, pp. 82-95, 1971.
25. J. Kivinen and M. K. Warmuth, "Additive versus exponentiated gradient updates for linear prediction," Inf. Comput., vol. 132, no. 1, pp. 1-64, 1997.
26. F. R. Kschischang, B. J. Frey, and H.-A. Loeliger, "Factor graphs and the sum-product algorithm," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 498-519, Feb. 2001.
27. S. R. Kulkarni and P. Viswanath, "A deterministic approach to throughput scaling in wireless networks," IEEE Trans. Inf. Theory vol. 50, no. 6, pp. 1041-1049, Jun. 2004.
28. A. Lazarevic and Z. Obradovic, "The distributed boosting algorithm," in Proc. 7th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, San Francisco, CA, 2001, pp. 311-316, published by ACM Press.
29. H. A. Loeliger, "An introduction to factor graphs," IEEE Signal Process. Mag., vol. 21, no. 1, pp. 28-41, Jan. 2004.
30. C. C. Moallemi and B. Van Roy, "Distributed optimization in adaptive networks," in Advances in Neural Information Processing Systems 16 S. Thrun, L. Saul, and B. Schölkopf, Eds. Cambridge, MA: MIT Press, 2004.
31. A. Nedic and D. Bertsekas, Incremental Subgradient Methods for Nondifferentiable Optimization MIT, Cambridge, MA, Tech. Rep. LIDS-P-2460, 1999.
32. A. Nedic and D. Bertsekas, "Convergence rate of incremental subgradient algorithms," in Stochastic Optimization: Algorithms and Applications S. Uryasev and P. M. Pardalos, Eds. Dordrecht, The Netherlands: Kluwer, 2000, pp. 263-304.
33. X. Nguyen, M. I. Jordan, and B. Sinopoli, "A kernel-based learning approach to ad hoc sensor network localization," ACM Trans. Sensor Networks, vol. 1, no. 1, pp. 134-152, 2005.
34. X. Nguyen, M. J.Wainwright, and M. I. Jordan, "Nonparametric decentralied detection using kernel methods," IEEE Trans. Signal Process., vol. 53, no. 11, pp. 4053-4066, Nov. 2005.
35. R. Nowak and U. Mitra, "Boundary estimation in sensor networks: Theory and methods," in Proc. 2nd Int. Workshop on Information Processing in Sensor Networks, Palo Alto, CA, Apr. 22-23, 2003, pp. 80-95.
36. R. D. Nowak, "Distributed EM algorithms for density estimation and clustering in sensor networks," IEEE Trans. Signal Process., vol. 51, no. 8, pp. 2245-2253, Aug. 2003.
37. M. A. Paskin and G. D. Lawrence, Junction Tree Algorithms for Solving Sparse Linear Systems EECS Dep., Univ.California, Berkeley, Tech. Rep. UCB/CSD-03-1271, 2003.
38. M. A. Paskin, C. E. Guestrin, and J. McFadden, "A robust architecture for inference in sensor networks," in Proc. 4rth Int. Symp. Information Processing in Sensor Networks, UCLA, Los Angeles, CA, Apr. 2005, pp. 55-62.
39. N. Patwari, J. N. Ash, S. Kyperountas, A. O. Hero, R. L. Moses, and N. S. Correal, "Locating the nodes: Cooperative localization in wireless sensor networks," IEEE Signal Process. Mag., vol. 22, no. 4, pp. 54-69, Jul. 2005.
40. J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Francisco, CA: Morgan Kaufmann, 1988.
41. K. Plarre and P. R. Kumar, "Extended message passing algorithm for inference in loopy Gaussian graphical models," Ad Hoc Networks, vol. 2, no. 2, pp. 153-169, 2004.
42. H. V. Poor, An Introduction to Signal Detection and Estimation, 2nd ed. New York: Springer-Verlag, 1994.
43. J. B. Predd, S. R. Kulkarni, and H. V. Poor, "Regression in sensor networks: Training distributively with alternating projections," in Proc. SPIE Conf. Advanced Signal Processing Algorithms, Architectures, and Implementations XV, San Diego, CA, Jul./Aug. 2005, pp. 591006-1-591006-15, Invited Paper.
44. J. B. Predd, S. R. Kulkarni, and H. V. Poor, "Distributed kernel regression: An algorithm for training collaboratively," in Proc. 2006 IEEE Information Theory Workshop, Punta del Este, Uruguay, Mar. 2006.
45. M. Rabbat and R. Nowak, "Distributed optimization in sensor networks," in Proc. 3rd Int. Symp. Information Processing in Sensor Networks, Berkeley, CA, Apr. 2004, pp. 20-27.
46. M. G. Rabbat and R. D. Nowak, "Quantized incremental algorithms for distributed optimization," IEEE J. Sel. Areas Commun., vol. 23, no. 4, pp. 798-808, Apr. 2005.
47. J. B. Predd, S. R. Kulkarni, and H. V. Poor, "Consistency in models for distributed learning under communication constraints," IEEE Trans. Inf. Theory, vol. 52, no. 1, pp. 52-53, Jan. 2006.
48. C. Rudin, Stability Analysis for Regularized Least Squares Regression 2005, arXiv: Cs.LG/0502016.
49. B. Schölkopf and A. Smola, Learning with Kernels, 1st ed. Cambridge, MA: MIT Press, 2002.
50. S.-H. Son, M. Chiang, S. R. Kulkarni, and S. C. Schwartz, "The value of clustering in distributed estimation for sensor networks," in Proc. IEEE Int. Conf. Wireless Networks, Communications, and Mobile Computing Maui, HI, Jun. 2005, vol. 2, pp. 969-974.
51. J. von Neumann, Function Operators II. Princeton, NJ: Princeton Univ. Press, 1950.
52. G. Wahba, Spline Models for Observational Data. Philadelphia, PA: SIAM, 1990.