The evaluation of predictive learners: Some theoretical and empirical results

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 2167, Issue , 2001, Pages 276-287

  Korb, Kevin B ^a   Hope, Lucas R ^a   Hughes, Michelle J ^a  

^a MONASH UNIVERSITY   (Australia)

Author keywords

Evaluation; Information reward; Kullback Leibler distance; Machine learning; Predictive accuracy; Scoring rules

Indexed keywords

CALIBRATION; DATA MINING; LEARNING SYSTEMS; MACHINE LEARNING;

EVALUATION; INFORMATION REWARD; KULLBACK-LEIBLER DISTANCE; PREDICTIVE ACCURACY; SCORING RULES;

LEARNING ALGORITHMS;

EID: 84894469885     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/3-540-44795-4_24     Document Type: Conference Paper

References (27)

1. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1-3.
2. Cover, T. M. and Thomas, J. A. (1991). Elements of information theory. New York: Wiley.
3. Domingos, P. and Pazzani, M. (1996). Beyond independence: Conditions for the optimality of the simple bayesian classifier. In Proceedings of the Thirteenth International Conference on Machine Learning, (pp. 105-112), Bari, Italy. Morgan Kaufmann.
4. Dowe, D. L., Farr, G. E., Hurst, A. J. and Lentin, K. L. (1996). Information-theoretic football tipping. Technical Report 96/297, Dept. of Computer Science, Monash University.
5. Dowe, D. L. (2000). Learning and prediction notes. School of Computer Science and Software Engineering, Monash University.
6. Good, I. J. (1952). Rational decisions. Jrn. of the Royal Statistical Society, B, 14, 107-114. Reprinted in Good thinking: The foundations of probability and its applications, Minnesota, 1983.
7. Griffen, D., and Tversky, A. (1992). The weighing of evidence and the determinants of confidence. Cognitive Psychology, 24, 411-435.
8. Holte, R. C., (1993). Very simple classification rules perform well on most commonly used datasets. Machine Learning, 11, 63-91.
9. Jeffrey, R. (1983). The logic of decision, 2nd ed. New York: McGraw-Hill.
10. Kononenko, I., and Bratko, I. (1991). Information-based evaluation criterion for classifier’s performance. Machine Learning, 6, 67-80.
11. Leslie, C. (1998). Lack of confidence. MA Thesis, Department of History and Philosophy of Science, University of Melbourne.
12. Lewis, D. (1980). A subjectivist’s guide to objective chance. In Jeffrey (Ed.) Studies in inductive logic and probability, vol II (pp. 263-293). Univ of California.
13. Lichtenstein, S., Fischhoff, B., and Phillips, L. D. (1977). Calibration of probabilities: The state of the art. In H. Jungermann and G. de Zeeuw (Eds.), Decision making and change in human affairs (pp. 275-324). Dordrecht: Reidel.
14. Matheson, J. E., and Winkler, R. L. (1976). Scoring rules for continuous probability distributions. Management Science, 22.
15. McClelland, A. G. R., and Bolger, F. (1994). The calibration of subjective probabilities: Theories and models, 1980-1994. In G. Wright and P. Ayton (Eds.) Subjective probability, Wiley.
16. Mitchell, T. (1997). Machine learning. McGraw-Hill.
17. Morgan, M. G., and Henrion, M. (1990). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University.
18. Murphy, A. H., and Winkler, R. L. (1984). The probability of precipitation forecasts. Journal of the American Statistical Association, 79, 391-400.
19. Oliver, J. (1993). Decision graphs-an extension of decision trees. Fourth Int. Conf. Artificial Intelligence and Statistics, pp. 343-350
20. Pearl, J. (1978). An economic basis for certain methods of evaluating probabilistic forecasts. International Journal of Man-Machine Studies, 10, 175-183.
21. Provost, F., Fawcett, T. and Kohavi, R. (1998). The case against accuracy estimation for comparing induction algorithms. International Conference on Machine Learning, 1998, Morgan Kaufmann.
22. Quinlan, J. R. (1993). C4. 5: programs for machine learning. Morgan Kaufmann.
23. Ramsey, F. P. (1931). The foundations of mathematics and other logical essays, edited by R. B. Braithwaite. New York: Humanities Press.
24. Savage, L. J. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66.
25. Turney, P. (1995). Cost-sensitive classification: Empirical evaluation of a hybrid genetic decision tree induction algorithm. Journal of Artificial Intelligence Research, 369-409.
26. Wallace, C. S., and Boulton, D. M. (1968). An information measure for classification. The Computer Journal, 11, 185-194.
27. Wallace, C. S., and Patrick, J. D. (1993). Coding decision trees. Machine Learning, 11, 7-22.