Comparison of approaches for estimating reliability of individual regression predictions

Data and Knowledge Engineering

Volumn 67, Issue 3, 2008, Pages 504-516

  Bosnić, Zoran ^a   Kononenko, Igor ^a  

^a UNIVERSITY OF LJUBLJANA   (Slovenia)

Author keywords

Prediction accuracy; Prediction error; Regression; Reliability estimate; Sensitivity analysis

Indexed keywords

FORECASTING; LEARNING SYSTEMS; NETWORK PROTOCOLS; NEURAL NETWORKS; REGRESSION ANALYSIS; RELIABILITY; RELIABILITY ANALYSIS; SENSOR NETWORKS; SUPPORT VECTOR MACHINES; TREES (MATHEMATICS);

AVERAGE PERFORMANCES; CROSS-VALIDATION; DENSITY ESTIMATIONS; DO-MAINS; GENERALIZED ADDITIVE MODELS; INDIVIDUAL PREDICTIONS; LOCALLY WEIGHTED REGRESSIONS; MODELING; PREDICTION ACCURACY; PREDICTION ERROR; PREDICTION ERRORS; RANDOM FORESTS; REGRESSION; REGRESSION MODELS; REGRESSION TREES; SUPPORT VECTORS;

SENSITIVITY ANALYSIS;

EID: 54349094489     PISSN: 0169023X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.datak.2008.08.001     Document Type: Article

References (61)

1. Kononenko I., and Kukar M. Machine Learning and Data Mining: Introduction to Principles and Algorithms (2007), Horwood Publishing Limited, UK
2. Crowder M.J., Kimber A.C., Smith R.L., and Sweeting T.J. Statistical concepts in reliability. Statistical Analysis of Reliability Data (1991), Chapman & Hall, London, UK
3. A. Gammerman, V. Vovk, V. Vapnik, Learning by transduction, in: Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, Madison, Wisconsin, 1998, pp. 148-155.
4. C. Saunders, A. Gammerman, V. Vovk, Transduction with confidence and credibility, in: Proceedings of IJCAI'99, vol. 2, 1999, pp. 722-726.
5. Nouretdinov I., Melluish T., and Vovk V. Ridge regression confidence machine. Proc. 18th International Conf. on Machine Learning (2001), Morgan Kaufman, San Francisco, CA 385-392
6. A. Weigend, D. Nix, Predictions with confidence intervals (local error bars), in: Proceedings of the International Conference on Neural Information Processing (ICONIP'94), Seoul, Korea, 1994, pp. 847-852.
7. Heskes T. Practical confidence and prediction intervals. In: Mozer M.C., Jordan M.I., and Petsche T. (Eds). Advances in Neural Information Processing Systems vol. 9 (1997), The MIT Press 176-182
8. J. Carney, P. Cunningham, Confidence and prediction intervals for neural network ensembles, in: Proceedings of IJCNN'99, The International Joint Conference on Neural Networks, Washington, USA, 1999, pp. 1215-1218.
9. M. Birattari, H. Bontempi, H. Bersini, Local learning for data analysis, in: Proceedings of the Eighth Belgian-Dutch Conference on Machine Learning, 1998, pp. 55-61.
10. Giacinto G., and Roli F. Dynamic classifier selection based on multiple classifier behaviour. Pattern Recognition 34 9 (2001) 1879-1881
11. Tsuda K., Rätsch G., Mika S., and Müller K. Learning to predict the leave-one-out error of kernel based classifiers. Lecture Notes in Computer Science (2001) 331
12. Vapnik V. The Nature of Statistical Learning Theory (1995), Springer
13. Kukar M., and Kononenko I. Reliable classifications with machine learning. In: Elomaa T., Manilla H., and Toivonen H. (Eds). Proc. Machine Learning: ECML-2002 (2002), Springer-Verlag, Helsinki Finland 219-231
14. Z. Bosnić, I. Kononenko, M. Robnik-Šikonja, M. Kukar, Evaluation of prediction reliability in regression using the transduction principle, in: B. Zajc, M. Tkalčič (Eds.), Proceedings of Eurocon 2003, Ljubljana, 2003, pp. 99-103.
15. Bosnić Z., and Kononenko I. Estimation of regressor reliability. Journal of Intelligent Systems 17 1/3 (2008) 297-311
16. Z. Bosnić, I. Kononenko, Estimation of individual prediction reliability using the local sensitivity analysis, Applied Intelligence, in press [Online edition] http://www.springerlink.com/content/e27p2584387532g8/.
17. Li M., and Vitányi P. An Introduction to Kolmogorov Complexity and its Applications (1993), Springer-Verlag, New York
18. Breiman L. Bagging predictors. Machine Learning 24 2 (1996) 123-140
19. Wolpert D. Stacked generalization. Neural Networks vol. 5 (1992), Pergamon Press 241-259
20. Tibshirani R., and Knight K. Model search and inference by bootstrap bumping. Journal of Computational and Graphical Statistics 8 (1999) 671-686
21. Freund Y., and Schapire R. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55 1 (1997) 119-139
22. G. Elidan M. Ninio N. Friedman D. Schuurmans, Data perturbation for escaping local maxima in learning, 2002.
23. Cohn D.A., Ghahramani Z., and Jordan M.I. Active learning with statistical models. In: Tesauro G., Touretzky D., and Leen T. (Eds). Advances in Neural Information Processing Systems vol. 7 (1995), The MIT Press 705-712. citeseer.ist.psu.edu/cohn95active.html
24. L.A.D.A. Cohn and, R. Ladner, Training connectionist networks with queries and selective sampling, in: M.K.D. Touretzky, (Ed.), Advances in Neural Information Processing Systems, vol. 2, 1990, pp. 566-573.
25. A. Linden, F. Weber, Implementing inner drive by competence reflection, in: Proceedings of the Second International Conference on Simulation of Adaptive Behavior, Hawaii, 1992, pp. 321-326.
26. J. Schmidhuber, J. Storck, Reinforcement driven information acquisition in nondeterministic environments, Fakultat fur Informatik, Technische Universit at Munchen, Technical Report, 1993.
27. S.D. Whitehead, A complexity analysis of cooperative mechanisms in reinforcement learning, in: AAAI, 1991, pp. 607-613.
28. M. Seeger, Learning with labeled and unlabeled data, Technical Report, http://www.dai.ed.ac.uk/seeger/papers.html, 2000.
29. A. Blum, T. Mitchell, Combining labeled and unlabeled data with co-training, in: Proceedings of the 11th Annual Conference on Computational Learning Theory, 1998, pp. 92-100.
30. T. Mitchell, The role of unlabelled data in supervised learning, in: Proceedings of the Sixth International Colloquium of Cognitive Science, San Sebastian, Spain, 1999.
31. de Sa V. Learning classification with unlabeled data. In: Cowan J.D., Tesauro G., and Alspector J. (Eds). Proc. NIPS'93, Neural Information Processing Systems (1993), Morgan Kaufmann Publishers, San Francisco, CA 112-119
32. Goldman S., and Zhou Y. Enhancing supervised learning with unlabeled data. Proc. 17th International Conf. on Machine Learning (2000), Morgan Kaufman, San Francisco, CA 327-334
33. L. Breierova, M. Choudhari, An introduction to sensitivity analysis, MIT System Dynamics in Education Project, September 1996.
34. J. Kleijnen, Experimental designs for sensitivity analysis of simulation models, in: Tutorial at the Eurosim 2001 Conference.
35. Bousquet O., and Elisseeff A. Stability and generalization. Journal of Machine Learning Research 2 (2002) 499-526
36. Kearns M.J., and Ron D. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Computational Learning Theory (1997) 152-162
37. O. Bousquet, A. Elisseeff, Algorithmic stability and generalization performance, in: NIPS, 2000, pp. 196-202.
38. Bousquet O., and Pontil M. Leave-one-out error and stability of learning algorithms with applications. In: Suykens J.A.K., et al. (Ed). Advances in Learning Theory: Methods, Models and Applications (2003), IOS Press
39. Saltelli A., Tarantola S., Campolongo F., and Ratto M. Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models (2003), John Wiley & Sons Ltd., England
40. Schaal S., and Atkeson C.G. Constructive incremental learning from only local information. Neural Computation 10 8 (1998) 2047-2084
41. Woods K., Kegelmeyer W.P., and Bowyer K. Combination of multiple classifiers using local accuracy estimates. IEEE Transactions on PAMI 19 4 (1997) 405-410
42. Schaal S., and Atkeson C.G. Assessing the quality of learned local models. In: Cowan J.D., Tesauro G., and Alspector J. (Eds). Advances in Neural Information Processing Systems vol. 6 (1994), Morgan Kaufman Publishers, Inc. 160-167
43. Wand M.P., and Jones M.C. Kernel Smoothing (1995), Chapman and Hall, London
44. Silverman B.W. Density Estimation for Statistics and Data Analysis. Monographs on Statistics and Applied Probability (1986), Chapman and Hall, London
45. Jeon B., and Landgrebe D.A. Parzen density estimation using clustering-based branch and bound. Transactions on Pattern Analysis and Machine Intelligence (1994) 950-954
46. R Development Core Team, A Language and Environment for Statistical Computing, R Foundation for Statistical Computing,Vienna, Austria, 2006.
47. Breiman L., Friedman J.H., Olshen R.A., and Stone C.J. Classification and Regression Trees (1984), Wadsworth International Group, Belmont CA
48. McCulloch W.S., and Pitts W. A logical calculus of the ideas imminent in nervous activity. Bulletin of Mathematical Biophysics 5 (1943) 115-133
49. A.J. Smola, B. Schölkopf, A tutorial on support vector regression, NeuroCOLT2 Technical Report NC2-TR-1998-030, 1998.
50. Christiannini N., and Shawe-Taylor J. Support Vector Machines and Other Kernel-Based Learning Methods (2000), Cambridge University Press
51. C. Chang, C. Lin, LIBSVM: a library for support vector machines, software available at http://www.csie.ntu.edu.tw/cjlin/libsvm, 2001.
52. Breiman L. Random forests. Machine Learning 45 1 (2001) 5-32
53. Wood S.N. Generalized Additive Models: An Introduction with R (2006), Chapman & Hall, CRC
54. Hastie T., and Tibshirani R. Generalized Additive Models (1990), Chapman and Hall, London
55. A. Asuncion D.J. Newman, UCI machine learning repository, 2007.
56. Department of Statistics at Carnegie Mellon University, Statlib - data, software and news from the statistics community, http://lib.stat.cmu.edu/, 2005.
57. Mendling J., Verbeek H.M.W., van Dongen B.F., and van der Aalst W.M.P. Detection and prediction of errors in epcs of the sap reference model. Data and Knowledge Engineering 64 1 (2008) 312-329
58. Fan T.H., and Cheng K.F. Tests and variables selection on regression analysis for massive datasets. Data and Knowledge Engineering 63 4 (2007) 811-819
59. Han Y., and Lam W. Utilizing hierarchical feature domain values for prediction. Data and Knowledge Engineering 61 3 (2007) 540-553
60. Lee Y.J., Hsieh W.F., and Huang C.M. epsilon-ssvr: A smooth support vector machine for epsilon-insensitive regression. IEEE Transactions of Knowledge and Data Engineering 17 5 (2005) 678-685
61. Zhou Z.H., and Li M. Semi-supervised regression with co-training style algorithms. IEEE Transactions of Knowledge and Data Engineering 19 11 (2007) 1479-1493