An overview of advances in reliability estimation of individual predictions in machine learning

Intelligent Data Analysis

Volumn 13, Issue 2, 2009, Pages 385-401

  Bosnić, Zoran ^a   Kononenko, Igor ^a  

^a UNIVERSITY OF LJUBLJANA   (Slovenia)

Author keywords

Data perturbation; Prediction accuracy; Predictions; Reliability; Supervised learning; Unlabeled examples

Indexed keywords

EID: 62449194481     PISSN: 1088467X     EISSN: 15714128     Source Type: Journal    
DOI: 10.3233/IDA-2009-0371     Document Type: Review

References (69)

1. E. Alpaydin, Introduction to Machine Learning, The MIT Press, Cambridge, Massachussetts, 2004.
2. S. Baluja, Probabilistic modeling for face orientation discrimination: Learning from labeled and unlabeled data, in: Proceedings of the 1998 conference on Advances in neural information processing systems II, M.J. Kearns, ed., The MIT Press, 1998, pp. 854-860.
3. M. Birattari, H. Bontempi and H. Bersini, Local Learning for Data Analysis, in: Proceedings of the 8th Belgian-Dutch Conference on Machine Learning, 1998, pp. 55-61.
4. A. Blum and T. Mitchell, Combining labeled and unlabeled data with co-training, in: Proceedings of the 11th Annual Conference on Computational Learning Theory, 1998, pp. 92-100.
5. Z. Bosnić and I. Kononenko, Estimation of individual prediction reliability using the local sensitivity analysis, Applied Intelligence 29(3) (2007), 187-203.
6. Z. Bosnić and I. Kononenko, Estimation of regressor reliability, Journal of intelligent systems 17(1/3) (2008), 297-311.
7. Z. Bosnić and I. Kononenko, Automatic Selection of Reliability Estimates for Individual Regression Predictions Using Meta-Learning and Internal Cross-Validation, Knowledge Engineering Review (2008), in press.
8. Z. Bosnić and I. Kononenko, Comparison of approaches for estimating reliability of individual regression predictions, Data & Knowledge Engineering 67(3) (2008), 504-516.
9. Z. Bosnić, I. Kononenko, M. Robnik-Šikonja and M. Kukar, Evaluation of prediction reliability in regression using the transduction principle, in: Proceedings of Eurocon 2003, B. Zajc and M. Tkalčič, eds, 2003, pp. 99-103.
10. O. Bousquet and A. Elisseeff, Algorithmic Stability and Generalization Performance, in: Neural Information Procesing Systems, 2001, pp. 196-202.
11. O. Bousquet and A. Elisseeff, Stability and generalization, Journal of Machine Learning Research 2 (2002), 499-526.
12. O. Bousquet and M. Pontil, Leave-one-out error and stability of learning algorithms with applications, in: Advances in Learning Theory: Methods, Models and Applications, J.A.K. Suykens, ed., IOS Press, 2003.
13. L. Breierova and M. Choudhari, An Introduction to Sensitivity Analysis, repared for the MIT System Dynamics in Education Project, MIT, 1996.
14. L. Breiman, Bagging Predictors, Machine Learning 24 (1996), 123-140.
15. L. Breiman, Pasting bites together for prediction in large data sets and on-line, Technical Report, University of California, 1997.
16. J. Carney and P. Cunningham, Confidence and prediction intervals for neural network ensembles, in: Proceedings of The International Joint Conference on Neural Networks, Washington, USA, 1999, pp. 1215-1218.
17. D.A. Cohn, Z. Ghahramam and M. I. Jordan, Active Learning with Statistical Models, Advances in Neural Information Processing Systems 7 (1995), 705-712.
18. D.A. Cohn, L. Atlas and R. Ladner, Training Connectionist Networks with Queries and Selective Sampling, Advances in Neural Information Processing Systems 2 (1990), 566-573.
19. M.J. Crowder, A.C. Kimber, R.L. Smith and T. J. Sweeting, Statistical concepts in reliability, in: Statistical Analysis of Reliability Data Chapman & Hall, London, UK, 1991, pp. 1-11.
20. A.P. Dempster, N.M. Laird and D.B. Rubin, Maximum likelihood from incomplete data via the EM algorithm, Journal of the Royal Statistical Society, Series B 39(1) (1977), 1-38.
21. V. de Sa, Learning Classification with Unlabeled Data, in: Proceedings of Neural Information Processing Systems, J.D. Cowan, ed., , Morgan Kaufmann Publishers, San Francisco, CA, 1993, pp. 112-119.
22. H. Drucker, Improving regressors using boosting techniques, in: Machine Learning: Proceedings of the Fourteenth International Conference 1997, pp. 107-115.
23. A. Dudek, Cluster Quality Indexes for Symbolic Classification ?An Examination, in: Advances in Data Analysis, Springer Berlin Heidelberg, 2007, pp. 31-38.
24. G. Elidan, M. Ninio, N. Friedman and D. Shuurmans, Data Perturbation for Escaping Local Maxima in Learning, in: Proceedings AAAI/IAAI, 2002, pp. 13-139.
25. Y. Freund and R. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, Journal of Computer and System Sciences 55(1) (1997), 119-139.
26. J. Gama and P.P. Rodrigues, Stream-Based Electricity Load Forecast, in: Proceedings of PKDD 2007, Lecture Notes in Artificial Intelligence Vol. 4702, J.N. Kok, ed., Warsaw, Poland, 2007, pp. 446-453.
27. A. Gammerman, V. Vovk and V. Vapnik, Learning by Transduction, in: Proceedings of the 14 th Conference on Uncertainty in Artificial Intelligence, Madison, Wisconsin, 1998, pp. 148-155.
28. A. Gelman and J. Hill, Data Analysis Using Regression and Multilevel/ hierarchical Models, Cambridge University Press, 2006.
29. P. Geurts, Dual perturb and combine algorithm, in: Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics 2001, pp. 196-201.
30. Z. Ghahramani and M.I. Jordan, Supervised learning from incomplete data via an EM approach, Advances in Neural Information Processing Systems 6(1994), 120-127.
31. G. Giacinto and F. Roli, Dynamic classifier selection based on multiple classifier behaviour, Pattern Recognition 34(9) (2001), 1879-1881.
32. S. Goldman and Y. Zhou, Enhancing Supervised Learning with Unlabeled Data, in: Proceedings of 17th International Conference on Machine Learning, Morgan Kaufmann, San Francisco, CA, 2000, pp. 327-334.
33. H. Hashem, Sensitivity Analysis for Feedforward Artificial Neural Networks with Differentiable Activation Functions, in: Proceedings of 1992 International Joint Conference on Neural Networks IJCNN92 I 1992, pp. 419-424.
34. T. Heskes, Practical Confidence and Prediction Intervals, Advances in Neural Information Processing Systems 9 (1997), 176-182.
35. M.J. Kearns and D. Ron, Algorithmic Stability and Sanity-Check Bounds for Leave-one-Out Cross-Validation, in: Computational Learing Theory 1997, pp. 152-162.
36. M. Kerr and G. Churchill, Bootstrapping cluster analysis: Assessing the reliability of conclusions from microarray experiments, Proc Natl Acad Sci USA 96 (2001), 8961-8965.
37. U. Kjaerulff and L.C. van der Gaag, Making sensitivity analysis computationally efficient, submitted to UAI 2000, 2000.
38. J. Kleijnen, Experimental Designs for Sensitivity Analysis of Simulation Models, tutorial at the Eurosim 2001 Conference, 2001.
39. I. Kononenko and M. Kukar, Machine Learning and Data Mining: Introduction to Principles and Algorithms, Horwood Publishing Limited, UK, 2007.
40. M. Kukar and I. Kononenko, Reliable Classifications with Machine Learning, in: Proceedings of Machine Learning: ECML-2002, Springer Verlag, Helsinki, Finland, 2002, pp. 219-231.
41. F. Li and H. Wechsler, Open Set Face Recognition Using Transduction, IEEE transactions on pattern analysis and machine intelligence 27 11) (2005), 1686-1697.
42. M. Li and P. Vitányi, An Introduction to Kolmogorov Complexity and its Applications, Springer-Verlag, New York, 1993.
43. A. Linden and F. Weber, Implementing inner drive by competence reflection, in: Proceedings of the 2nd International Conference on Simulation of Adaptive Behavior, Hawaii, 1992, pp. 321-326.
44. T. Mitchell, The role of unlabelled data in supervised learning, in: Proceedings of the 6th International Colloquium of Cognitive Science, San Sebastian, Spain, 1999.
45. I. Nouretdinov, T. Melluish and V. Vovk, Ridge Regressioon Confidence Machine, in: Proceedings of the 18th International Conf. on Machine Learning, Morgan Kaufmann, San Francisco, CA, 2001, pp. 385-392.
46. K. Osei-Bryson, Assessing Cluster Quality Using Multiple Measures - A Decision Tree Based Approach, in: The Next Wave in Computing, Optimization, and Decision Technologies, Springer US, 2005, pp. 371-384.
47. K. Patan and T. Parisini, Stochastic learning methods- for dynamic neural networks: Simulated and real-data comparisons, in: Proceedings of the American Control Conference, Inst. of Control & Comput. Eng., Poland, 2002, pp. 2577-2582.
48. G. Ridgeway, The State of Boosting, Technical Report, University of Washington Seattle, 1998.
49. G. Ridgeway, D. Madigan and T. Richardson, Boosting methodology for regression problems, in: Proc. Artificial Intelligence and Statistics 1999, pp. 152-161.
50. R. Rosipal, M. Girolami and L. Trejo, On Kernel Principal Component Regression with Covariance Inflation Criterion or Model Selection, Technical Report, University of Paisley, 2000.
51. A. Saltelli, M. Ratto, S. Tarantola and F. Campolongo, Sensitivity Analysis for Chemical Models, Chemical Reviews 105(7) (2005), 2811-2828.
52. A. Saltelli, S. Tarantola, F. Campolongo and M. Ratto, Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models, John Wiley & Sons Ltd, England, 2003.
53. C. Saunders, A. Gammerman and V. Vovk, Transduction with Confidence and Credibility, in: Proceedings of IJCAI 2, 1999, pp. 722-726.
54. S. Schaal and C.G. Atkeson, Assessing the Quality of Learned Local Models, Advances in Neural Information Processing Systems 6, (1994), 160-167.
55. S. Schaal and C.G. Atkeson, Constructive Incremental Learning from Only Local Information, Neural Computation 10(8) (1998), 2047-2084.
56. R.E. Schapire, A Brief Introduction to Boosting, in: Proceedings of IJCAI, 1999, pp. 1401-1406.
57. J. Schmidhuber and J. Storck, Reinforcement driven information acquisition in nondeterministic environments, Technical Report, Technische Universitat Munchen, 1993.
58. M. Seeger, Learning with labeled and unlabeled data, Technical Report, http://www.dai.ed.ac.uk/seeger/papers.html, 2000.
59. S.B. Thrun and K. Möller, Active Exploration in Dynamic Environments, Advances in Neural Information Processing Systems 4 (1992), 531-538.
60. R. Tibshirani and K. Knight, The covariance inflation criterion for adaptive model selection, Journal of the Royal Statistical Society B 61 (1999), 529-546.
61. R. Tibshirani and K. Knight, Model Search and Inference by Bootstrap Bumping, Journal of Computational and Graphical Statistics 8 (1999), 671-686.
62. K. Tsuda, G. Rätsch, S. Mika and K. Müller, Learning to Predict the Leave-One-Out Error of Kernel Based Classifiers, Lecture Notes in Computer Science 2130 (2001), 331.
63. V. Vapnik, The Nature of Statistical Learning Theory, Springer Verlag, 1995.
64. A. Weigend and D. Nix, Predictions with Confidence Intervals (Local Error Bars), in: Proceedings of the International Conference on Neural Information Processing (ICONIP'94), Seoul, Korea, 1994, pp. 847-852.
65. S.D. Whitehead, A Complexity Analysis of Cooperative Mechanisms in Reinforcement Learning, in: Proceedings of AAAI, 1991, pp. 607-613.
66. Wikipedia, the free encyclopedia, Confidence Interval, http://en.wikipedia.org/wiki/Confidence_interval, august 2007.
67. Wikipedia, the free encyclopedia, Reliability, http://en.wikipedia.org/wiki/Reliability, august 2007.
68. D.H. Wolpert, Stacked Generalization, Neural Networks 5 (1992), 241-259.
69. K. Woods, W.P. Kegelmeyer and K. Bowyer, Combination of Multiple Classifiers Using Local Accuracy Estimates, IEEE Transactions on PAMI 19(4) (1997), 405-410.