Measuring and comparing the accuracy of species distribution models with presence-absence data

Ecography

Volumn 34, Issue 2, 2011, Pages 232-243

  Liu, Canran ^a   White, Matt ^a   Newell, Graeme ^a  

^a ARTHUR RYLAH INSTITUTE FOR ENVIRONMENTAL RESEARCH   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY ASSESSMENT; BOOTSTRAPPING; CLIMATE CHANGE; CLIMATE EFFECT; COMPARATIVE STUDY; CONFIDENCE INTERVAL; CONSERVATION PLANNING; DATA SET; ECOLOGICAL MODELING; POPULATION DISTRIBUTION;

EID: 79953286484     PISSN: 09067590     EISSN: 16000587     Source Type: Journal    
DOI: 10.1111/j.1600-0587.2010.06354.x     Document Type: Article

References (108)

1. Agresti, A. 2002. Categorical data analysis, 2nd ed. Wiley.
2. Allouche, O. et al. 2006. Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS). J. Appl. Ecol. 43: 1223-1232.
3. Araújo, M. B. and Pearson, R. G. 2005. Equilibrium of species' distributions with climate. Ecography 28: 693-695.
4. 2: a useful measure of model performance when predicting a dichotomous outcome. Stat. Med. 18: 375-384.
5. Bandos, A. I. et al. 2005. A permutation test sensitive to differences in areas for comparing ROC curves from a paired design. Stat. Med. 24: 2873-2893.
6. Barnhart, H. X. and Williamson, J. M. 2002. Weighted least-squares approach for comparing correlated kappa. Biometrics 58: 1012-1019.
7. Biggerstaff, B. J. 2000. Comparing diagnostic tests: a simple graphic using likelihood ratios. Stat. Med. 19: 649-663.
8. Blackman, N. J. and Koval, J. J. 2000. Interval estimation for Cohen's kappa as a measure of agreement. Stat. Med. 19: 723-741.
9. Bloch, D. A and Kraemer, H. C. 1989. 2×2 kappa coefficients: measures of agreement or association. Biometrics 45: 269-287.
10. Böhning, B. et al. 2008. Revisiting Youden's index as a useful measure of the misclassification error in meta-analysis of diagnostic studies. Stat. Methods Med. Res. 17: 543-554.
11. Bradley, A. A. et al. 2008. Sampling uncertainty and confidence intervals for the Brier score and Brier skill score. Weather Forecast. 23: 992-1006.
12. Breiman, L. et al. 1984. Classification and regression trees. Wadsworth International Group, Belmont, CA.
13. Brier, G. W. 1950. Verification of forecasts expressed in terms of probability. Mon. Weather Rev. 78: 1-3.
14. Briggs, W. M. and Zaretzki, R. 2008. The skill plot: a graphical technique for evaluating continuous diagnostic tests. Biometrics 63: 250-261.
15. Brown, L. D. et al. 2001. Interval estimation for a binomial proportion. Stat. Sci. 16: 101-117.
16. Caruana, R. and Niculescu-Mizil, A. 2004. Data mining in metric space: an empirical analysis of supervised learning performance criteria. Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 22-25, 2004. Seattle, WA, USA, pp. 69-78.
17. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20: 37-40.
18. Coles, S. et al. 1999. Dependence measures for extreme value analyses. Extremes 2: 339-365.
19. Couto, P. 2003. Assessing the accuracy of spatial simulation models. Ecol. Model. 167: 181-198.
20. Daskalaki, S. et al. 2006. Evaluation of classifiers for an uneven class distribution problem. Appl. Artif. Intell. 20: 381-417.
21. DeLong, E. R. et al. 1988. Comparing the areas under two or more correlated receiver operating characteristic curves: a non parametric approach. Biometrics 44: 837-845.
22. Drake, J. M. et al. 2006. Modelling ecological niches with support vector machines. J. Appl. Ecol. 43: 424-432.
23. Elith, J. et al. 2006. Novel methods improve prediction of species' distributions from occurrence data. Ecography 29: 129-151.
24. Engler, R. et al. 2004. An improved approach for predicting the distribution of rare and endangered species from occurrence and pseudo-absence data. J. Appl. Ecol. 41: 263-274.
25. Fawcett, T. 2006. An introduction to ROC analysis. Pattern Recogn. Lett. 27: 861-874.
26. Ferro, C. A. T. 2007. A probability model for verifying deterministic forecasts of extreme events. Weather Forecast. 22: 1089-1100.
27. Fielding, A. H. 2007. Cluster and classification techniques for the biosciences. Cambridge Univ. Press.
28. Fielding, A. H. and Bell, J. F. 1997. A review of methods for the measurement of prediction errors in conservation presence/absence models. Environ. Conserv. 24: 38-49.
29. Finley, J. P. 1884. Tornado predictions. Am. Meteorol. J. 1: 85-88.
30. Finn, J. T. 1993. Use of the average mutual information index in evaluating classification error and consistency. Int. J. Inform. Syst. 7: 349-366.
31. Flack, V. 1987. Confidence intervals for the interrater agreement measure kappa. Commun. Stat. Theory 16: 953-968.
32. Fleiss, J. L. et al. 1969. Large-sample standard errors of kappa and weighted kappa. Psychol. Bull. 72: 323-327.
33. Flueck, J. A. 1987. A study of some measures of forecast verification. 10th Conference on Probability and Statistics in Atomaspheric Sciences. Edmonton, AB, Canada, American Meteorological Society, pp. 69-73.
34. Forbes, A. D. 1995. Classification-algorithm evaluation: five performance measures based on confusion matrices. J. Clin. Monit. 11: 189-206.
35. Fung, K. P. and Lee, J. 1991. Bootstrap estimate of the variance and confidence interval of kappa. Br. J. Ind. Med. 48: 503-504.
36. Garner, J. B. 1991. The standard error of Cohen's kappa. Stat. Med. 10: 767-775.
37. Gilbert, G. K. 1884. Finley's tornado predictions. Am. Meterol. J. 1: 166-172.
38. Glahn, B. 2004. Discussion of verification concepts in forecast verification: a practitioner's guide in atmospheric science. Weather Forecast. 19: 769-775.
39. Glas, A. S. et al. 2003. The diagnostic odds ratio: a single indicator of test performance. J. Clin. Epidemiol. 56: 1129-1135.
40. Glass, G. V. 1966. Note concerning rank biserial correlation. Educ. Psychol. Meas. 26: 623-631.
41. Gribskov, M. and Robinson, N. L. 1996. The use of receiver operating characteristic (ROC) analysis to evaluate sequence matching. Comput. Chem. 20: 25-33.
42. Guisan, A. and Zimmermann, N. E. 2000. Predictive habitat distribution models in ecology. Ecol. Model. 135: 147-186.
43. Guisan, A. et al. 1998. Predicting the potential distribution of plant species in an Alpine environment. J. Veg. Sci. 9: 65-74.
44. Hale, C. A. and Fleiss, J. L. 1993. Interval estimation under two study designs for kappa with binary classifications. Biometrics 49: 523-534.
45. Hand, D. J. 1992. Statistical methods in diagnosis. Stat. Methods Med. Res. 1: 49-67.
46. Hand, D. J. 2001. Measuring diagnostic accuracy of statistical prediction rules. Stat. Neerl. 55: 3-16.
47. Hand, D. J. and Hill, R. J. 2001. A simple generalization of the area under the ROC curve to multiple class classification problems. Mach. Learn. 45: 171-186.
48. Hanley, J. A. 1987. Standard error of the kappa statistic. Psychol. Bull. 102: 315-321.
49. Hanley, J. A. and McNeil, B. J. 1982. The meaning eand use of the area under a receiver operating characteristic (ROC) curve. Radiology 143: 29-36.
50. Hanley, J. A. and McNeil, B. J. 1983. A method of comparing the areas under receiver operating characterisitc curves derived from the same cases. Radiology 148: 839-848.
51. Hanley, J. A. and Hajian-Tilaki, K. O. 1997. Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves: an update. Acad. Radiol. 4: 49-58.
52. Hanssen, A. J. and Kuipers, W. J. 1965. On the relationship between the frequency of rain and various meteorological parameters. Meded Verhand 81: 2-15.
53. Hawass, N. E. D. 1997. Comparing the sensitivities and specificities of two diagnostic procedures performed on the same group of patients. Br. J. Radiol. 70: 360-366.
54. He, Y. and Escobar, M. 2008. Nonparametric statistical inference method for partial areas under receiver operating characteristic curves, with application to genomic studies. Stat. Med. 27: 5291-5308.
55. Heikkinen, R. K. et al. 2006. Methods and uncertainties in bioclimatic envelope modelling under climate change. Pro. Phys. Geogr. 30: 1-27.
56. Jiang, Y. et al. 1996. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology 143: 29-36.
57. Jolliffe, I. T. 2007. Uncertainty and inference for verification measures. Weather Forecast. 22: 637-650.
58. Klar, N. et al. 2002. An exact bootstrap confidence interval for κ in small samples. Statistician 51: 467-478.
59. Kraemer, H. C. 1992. Evaluating medical tests: objective and quantitative guidelines. Sage Publ.
60. Kraemer, H. C. 2006. Correlation coefficients in medical research: from product moment correlation to the odds ratio. Stat. Methods Med. Res. 15: 525-545.
61. Kundel, H. L. and Polansky, M. 2003. Measurement of observer agreement. Radiology 228: 303-308.
62. Lasko, T. A. et al. 2005. The use of receiver operating characteristic curves in biomedical informatics. J. Biomed. Inform. 38: 404-415.
63. Lee, J. J. and Tu, Z. N. 1994. A better confidence interval for kappa (κ) on measuring agreement between two raters with binary outcomes. J. Comput. Graph. Stat. 3: 301-321.
64. Lee, W. C. 1999. Probabilistic analysis of global performances of diagnostic tests: interpreting the Lorenz curve-based summary measures. Stat. Med. 18: 455-471.
65. Lee, W. C. and Hsiao, C. K. 1996. Alternative summary indices for the receiver operating characteristic curve. Epidemiology 7: 605-611.
66. Li, C. R. et al. 2008. On the exact interval estimation for the difference in paired areas under the ROC curves. Stat. Med. 27: 224-242.
67. Li, J. et al. 2007. Prevalence-dependent diagnostic accuracy measures. Stat. Med. 26: 3258-3273.
68. Liu, C. et al. 2005. Selecting thresholds of occurrence in the prediction of species distributions. Ecography 28: 385-393.
69. Liu, C. et al. 2007. Comparative assessment of the measures of thematic classification accuracy. Remote Sens. Environ. 107: 606-616.
70. Lobo, J. M. et al. 2008. AUC: a misleading measure of the performance of predictive distribution models. Global Ecol. Biogeogr. 17: 145-151.
71. Manel, S. et al. 2001. Evaluating presence-absence models in ecology: the need to account for prevalence. J. Appl. Ecol. 38: 921-931.
72. Mason, S. J. and Graham, N. E. 2002. Areas beneath the relative operating charcteristics (ROC) and relative operating levels (ROL) curves. statistical significance and interpretation. Q. J. R. Meteorol. Soc 128: 2145-2166.
73. McClish, D. K. 1989. Analyzing a portion of the ROC curve. Med. Decis. Making 9: 190-195.
74. McKenzie, D. P. et al. 1996. Comparing correlated kappas by resampling: is one level of agreement significantly different from another?. J. Psychiatr. Res. 30: 483-492.
75. Mittlböck, M. and Schemper, M. 1996. Explained variation for logistic regression. Stat. Med. 15: 1987-1997.
76. Molodianovitch, K. et al. 2006. Comparing the areas under two correlated ROC curves: parametric and non-parametric approaches. Biom. J. 48: 745-757.
77. Murphy, A. H. 1996. The Finley affair: a signal event in the history of forecast verification. Weather Forecast. 11: 3-20.
78. Murphy, A. H. and Winkler, R. L. 1984. Probability forecasting in meteorology. J. Am. Stat. Assoc. 79: 489-500.
79. Myers, J. L. and Well, A. 2003. Research design and statistical analysis, 2nd ed. Lawrence Erlbaum Associates, Hillsdale, NJ.
80. Newcombe, R. G. 1998a. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat. Med. 17: 857-872.
81. Newcombe, R. G. 1998b. Interval estimation for the difference between independent proportions: comparison of eleven methods. Stat. Med. 17: 873-890.
82. Newcombe, R. G. 1998c. Improved confidence intervals for the difference between binomial proportions based on paired data. Stat. Med. 17: 2635-2650.
83. Pearce, J. and Ferrier, S. 2000. Evaluating the predictive performance of habitat models developed using logistic regression. Ecol. Model. 133: 225-245.
84. Pearson, R. G. et al. 2007. Predicting species distributions from small numbers of occurrence records: a test case using cryptic geckos in Madagascar. J. Biogeogr. 34: 102-117.
85. Peirce, C. S. 1884. The numerical measure of the success of predictions. Science 4: 453-454.
86. Phillips, S. J. et al. 2009. Sample selection bias and presence-only distribution models: implications for background and pseudo-absence data. Ecol. Appl. 19: 181-197.
87. Pires, A. M. and Amado, C. 2008. Interval estimators for a binomial proportion: comparison of twenty methods. REVSTAT 6: 165-197.
88. Qin, G. and Hotilovac, L. 2008. Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test. Stat. Methods Med. Res. 17: 207-221.
89. Raes, N. and ter Steege, H. 2007. A null-model for significance testing of presence-only species distribution models. Ecography 30: 727-736.
90. Riddle, D. L. and Stratford, P. W. 1999. Interpreting validity indexes for diagnostic tests: an illustration using the Berg balance test. Phys. Ther. 79: 939-948.
91. Schemper, M. 2003. Predictive accuracy and explained variation. Stat. Med. 22: 2299-2308.
92. Seaman, R. et al. 1996. Confidence intervals for some performance measures of yes-no forecasts. Aust. Meteorol. Mag. 45: 49-53.
93. Sheskin, D. J. 2007. Handbook of parametric and nonparametric statistical procedures, 4 ed. Chapman and Hall/CRC.
94. Stephenson, D. B. 2000. Use of the "odds ratio" for diagnosing forecast skill. Weather Forecast. 15: 221-232.
95. Stephenson, D. B. et al. 2008. The extreme dependency score: a non-vanishing measure for forecasts of rare events. Meterol. Appl. 15: 41-50.
96. Stockwell, D. R. B. and Peterson, A. T. 2002. Effects of sample size on accuracy of species distribution models. Ecol. Model. 148: 1-13.
97. Tate, R. F. 1954. Correlation between a discrete and a continuous variable: point-biserial correlation. Ann. Math. Stat. 25: 603-607.
98. Thomas, D. G. 1971. Algorithm AS-36: exact confidence limits for the odds ratio in a 2×2 table. Appl. Stat. 20: 105-110.
99. Thuiller, W. et al. 2005. Niche-based modelling as a tool for predicting the risk of alien plant invasions at a global scale. Global Change Biol. 11: 2234-2250.
100. Vanbell, S. and Albert, A. 2008. A bootstrap method for comparing correlated kappa coefficients. J. Stat. Comput. Simul. 78: 1009-1015.
101. Vollset, S. E. 1993. Confidence intervals for a binomial proportion. Stat. Med. 12: 809-824.
102. Wieand, S. 1989. A family of non-parametric statistic for comparing diagnostic markers with paired or unpaired data. Biometrika 76: 585-592.
103. Williamson, J. M. et al. 2000. Modeling kappa for measuring dependent categorical agreement data. Biostatistics 1: 191-202.
104. Woodcock, F. 1976. The evaluation of yes-no forecasts for scientific and administrative purposes. Mon. Weather Rev. 104: 1209-1214.
105. Yerushalmy, J. 1947. Statistical problems in assessing methods of medical diagnosis. Public Health Rep. 62: 1432-1449.
106. Youden, W. J. 1951. Index for rating diagnostic tests. Cancer 3: 32-35.
107. Zhang, D. D. et al. 2002. A non-parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets. Stat. Med. 21: 701-715.
108. Zheng, B. and Agresti, A. 2000. Summarizing the predictive power of a generalized linear model. Stat. Med. 19: 1771-1781.