Assessing the accuracy of species distribution models: Prevalence, kappa and the true skill statistic (TSS)

Journal of Applied Ecology

Volumn 43, Issue 6, 2006, Pages 1223-1232

  Allouche, Omri ^a   Tsoar, Asaf ^a   Kadmon, Ronen ^a  

^a HEBREW UNIVERSITY OF JERUSALEM   (Israel)

Author keywords

AUC; Mahalanobis distance; Predictive maps; ROC curves; Sensitivity; Specificity; Woody plants

Indexed keywords

ACCURACY ASSESSMENT; POPULATION DISTRIBUTION; STATISTICAL ANALYSIS; WOODY PLANT;

ASIA; EURASIA; ISRAEL; MIDDLE EAST;

EID: 33750525529     PISSN: 00218901     EISSN: 13652664     Source Type: Journal    
DOI: 10.1111/j.1365-2664.2006.01214.x     Document Type: Article

References (68)

1. Accadia, C., Mariani, S., Casaioli, M., Lavaqnini, A. & Speranza, A. (2005) Verification of precipitation forecasts from two limited-area models over Italy and comparison with ECMWF forecasts using a resampling technique. Weather and Forecasting, 20, 276-300.
2. Araujo, M.B., Cabeza, M., Thuiller, W., Hannah, L. & Williams, P.H. (2004) Would climate change drive species out of reserves? An assessment of existing reserve-selection methods. Global Change Biology, 10, 1618-1626.
3. Araujo, M.B., Pearson, R.G., Thuiller, W. & Erhard, M. (2005) Validation of species-climate impact models under climate change. Global Change Biology, 11, 1504-1513.
4. Barry, S. & Elith, J. (2006) Error and uncertainty in habitat models. Journal of Applied Ecology, 43, 413-423.
5. Beaumont, L.J., Hughes, L. & Poulsen, M. (2005) Predicting species' distributions: use of climatic parameters in BIOCLIM and its impact on predictions of species' current and future distributions. Ecological Modelling, 186, 250-269.
6. Berg, A., Gardenfors, U. & von Proschwitz, T. (2004) Logistic regression models for predicting occurrence of terrestrial molluscs in southern Sweden: importance of environmental data quality and model complexity. Ecography, 27, 83-93.
7. Bomhard, B., Richardson, D.M., Donaldson, J.S., Hughes, G.O., Midgley, G.F., Raimondo, D.C., Rebelo, A.G., Rouget, M. & Thuiller, W. (2005) Potential impacts of future land use and climate change on the Red List status of the Proteaceae in the Cape Floristic Region, South Africa. Global Change Biology, 11, 1452-1468.
8. Brotons, L., Thuiller, W., Araújo, M.B. & Hirzel, A.H. (2004) Presence-absence versus presence-only modelling methods for predicting bird habitat suitability. Ecography, 27, 437-448.
9. Byrt, T., Bishop, J. & Carlin, J.B. (1993) Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46, 423-429.
10. Church, R.L., Stoms, D.M. & Davis, F.W. (1996) Reserve selection as a maximal covering location problem. Biological Conservation, 76, 105-112.
11. Cicchetti, D.V. & Feinstein, A.R. (1990) High agreement but low kappa. II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43, 551-558.
12. Cohen, J. (1960) A coefficient of agreement of nominal scales. Educational and Psychological Measurement, 20, 37-46.
13. Cumming, G.S. (2000a) Using between-model comparisons to fine-tune linear models of species ranges. Journal of Biogeography, 27, 441-455.
14. Cumming, G.S. (2000b) Using habitat models to map diversity: pan-African species richness of ticks (Acari: Ixodida). Journal of Biogeography, 27, 425-440.
15. Doswell, C.A., Daviesjones, R. & Keller, D.L. (1990) On summary measures of skill in rare event forecasting based on contingency tables. Weather and Forecasting, 5, 576-585.
16. Elmore, K.L., Weiss, S.J. & Banacos, P.C. (2003) Operational ensemble cloud model forecasts. Some preliminary results. Weather and Forecasting, 18, 953-964.
17. Engler, R., Guisan, A. & Rechsteiner, L. (2004) An improved approach for predicting the distribution of rare and endangered species from occurrence and pseudo-absence data. Journal of Applied Ecology, 41, 263-274.
18. Farber, O. & Kadmon, R. (2003) Assessment of alternative approaches for bioclimatic modelling with special emphasis on the Mahalanobis distance. Ecological Modelling, 160, 115-130.
19. Fielding, A.H. & Bell, J.F. (1997) A review of methods for the assessment of prediction errors in conservation presence/absence models. Environmental Conservation, 24, 38-49.
20. Guisan, A. & Hofer, U. (2003) Predicting reptile distributions at the mesoscale: relation to climate and topography. Journal of Biogeography, 30, 1233-1243.
21. Guisan, A. & Thuiller, W. (2005) Predicting species distribution: offering more than simple habitat models. Ecology Letters, 8, 993-1009.
22. Guisan, A., Lehmann, A., Ferrier, S., Austin, M., Overton, J.M.C.C., Aspinall, R. & Hastie, T. (2006) Making better biogeographical predictions of species' distributions. Journal of Applied Ecology, 43, 386-392.
23. Hanley, J.A. & McNeil, B.J. (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29-36.
24. Henderson, A.R. (1993) Assessing test accuracy and its clinical consequences: a primer for receiver operating characteristic curve analysis. Annals of Clinical Biochemistry, 30, 521-539.
25. Hoehler, F.K. (2000) Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. Journal of Clinical Epidemiology, 53, 499-503.
26. Howard, P.C., Viskanic, P., Davenport, T.R.B., Kigenyi, F.W., Baltzer, M., Dickinson, C.J., Lwanga, J.S., Matthews, R.A. & Balmford, A. (1998) Complementarity and the use of indicator groups for reserve selection in Uganda. Nature, 394, 472-475.
27. Huntley, B., Green, R.E., Collingham, Y.C., Hill, J.K., Willis, S.G., Bartlein, P.J., Cramer, W., Hagemeijer, W.J.M. & Thomas, C.J. (2004) The performance of models relating species' geographical distributions to climate is independent of trophic level. Ecology Letters, 7, 417-426.
28. Kadmon, R., Farber, O. & Danin, A. (2003) A systematic analysis of factors affecting the performance of climatic envelope models. Ecological Applications, 13, 853-867.
29. Lantz, C.A. & Nebenzahl, E. (1996) Behavior and interpretation of the k statistic: resolution of two paradoxes. Journal of Clinical Epidemiology, 49, 431-434.
30. Liu, C., Berry, P.M., Dawson, T.P. & Pearson, R.G. (2005) Selecting thresholds of occurrence in the prediction of species' distributions. Ecography, 28, 385-393.
31. Loiselle, B.A., Howell, C.A., Graham, C.H., Goerck, J.M., Brooks, T., Smith, K.G. & Williams, P.H. (2003) Avoiding pitfalls of using species distribution models in conservation planning. Conservation Biology, 17, 1591-1600.
32. McBride, J.L. & Ebert, E.E. (2000) Verification of quantitative precipitation forecasts from operational numerical weather prediction models over Australia. Weather and Forecasting, 15, 103-121.
33. McPherson, J.M., Jetz, W. & Rogers, D.J. (2004) The effects of species' range sizes on the accuracy of distribution models: ecological phenomenon or statistical artefact? Journal of Applied Ecology, 41, 811-823.
34. Manel, S., Dias, J.M. & Ormerod, S.J. (1999) Comparing discriminant analysis, neural networks and logistic regression for predicting species distributions: a case study with a Himalayan river bird. Ecological Modelling, 120, 337-347.
35. Manel, S., Williams, H.C. & Ormerod, S.J. (2001) Evaluating presence-absence models in ecology: the need to account for prevalence. Journal of Applied Ecology, 38, 921-931.
36. Margules, C.R. & Pressey, R.L. (2000) Systematic conservation planning. Nature, 405, 243-253.
37. Nelson, L.D. & Cicchetti, D.V. (1995) Assessment of emotional functioning in brain-impaired individuals. Psychological Assessment, 7, 404-413.
38. Nix, H.A. (1986) A biogeographic analysis of Australian elapid snakes. Atlas of Elapid Snakes of Australia (Ed. R. Longmore), pp. 4-15. Australian Government Publications Service, Canberra, Australia.
39. Norris, K. (2004) Managing threatened species: the ecological toolbox, evolutionary theory and declining-population paradigm. Journal of Applied Ecology, 41, 413-426.
40. Olden, J.D., Jackson, D.A. & Peres-Neto, P.R. (2002) Predictive models of fish species distributions: a note on proper validation and chance predictions. Transactions of the American Fisheries Society, 131, 329-336.
41. Ortega-Huerta, M.A. & Peterson, A.T. (2004) Modelling spatial patterns of biodiversity for conservation prioritization in north-eastern Mexico. Diversity and Distributions, 10, 39-54.
42. Parra, J.L., Graham, C.C. & Freile, J.F. (2004) Evaluating alternative data sets for ecological niche models of birds in the Andes. Ecography, 27, 350-360.
43. Pearce, J. & Ferrier, S. (2000) Evaluating the predictive performance of habitat models developed using logistic regression. Ecological Modelling, 133, 225-245.
44. Pearson, R.G., Dawson, T.P. & Liu, C. (2004) Modelling species distributions in Britain: a hierarchical integration of climate and land-cover data. Ecography, 27, 285-298.
45. Pearson, R.G., Thuiller, W., Araujo, M.B., Martinez-Meyer, E., Brotons, L., McClean, C., Miles, L., Segurado, P., Dawson, T.E. & Lees, D.C. (2006) Model-based uncertainty in species' range prediction. Journal of Biogeography, DOI: 10.1111/j.1365_2699.2006.01460x.
46. Peterson, A.T. & Robins, C.R. (2003) Using ecological-niche modelling to predict barred owl invasions with implications for spotted owl conservation. Conservation Biology, 17, 1161-1165.
47. Petit, S., Chamberlain, D., Haysom, K., Pywell, R., Vickery, J., Warman, L., Allen, D. & Firbank, L. (2003) Knowledge-based models for predicting species occurrence in arable conditions. Ecography, 26, 626-640.
48. Reese, G.C., Wilson, K.R., Hoeting, J.A. & Flather, C.H. (2005) Factors affecting species distribution predictions. A simulation modelling experiment. Ecological Applications, 15, 556-564.
49. Rouget, M., Richardson, D.M., Nel, J.L., Le Maitre, D.C., Egoh, B. & Mgidi, T. (2004) Mapping the potential ranges of major plant invaders in South Africa, Lesotho and Swaziland using climatic suitability. Diversity and Distributions, 10, 475-484.
50. Rushton, S.P., Ormerod, S.J. & Kerby, G. (2004) New paradigms for modelling species distributions? Journal of Applied Ecology, 41, 193-200.
51. Sanchez-Cordero, V., Cirelli, V., Munguial, M. & Sarkar, S. (2005) Place prioritization for biodiversity content using species ecological niche modelling. Biodiversity Informatics, 2, 11-23.
52. Saseendran, S.A., Singh, S.V., Rathore, L.S. & Das, S. (2002) Characterization of weekly cumulative rainfall forecasts over meteorological subdivisions of India using a GCM. Weather and Forecasting, 17, 832-844.
53. Schmidt, M., Kreft, H., Thiombiano, A. & Zizka, G. (2005) Herbarium collections and field data-based plant diversity maps for Burkina Faso. Diversity and Distributions, 11, 509-516.
54. Segurado, P. & Araujo, M.B. (2004) An evaluation of methods for modelling species' distributions. Journal of Biogeography, 31, 1555-1568.
55. Seoane, J., Carrascal, L.M., Alonso, C.L. & Palomino, D. (2005) Species-specific traits associated to prediction errors in bird habitat suitability modelling. Ecological Modelling, 185, 299-308.
56. Shao, G. & Halpin, P.N. (1995) Climatic controls of eastern North American coastal tree and shrub distributions. Journal of Biogeography, 22, 1083-1089.
57. Skov, F. & Svenning, J.C. (2004) Potential impact of climatic change on the distribution of forest herbs in Europe. Ecography, 27, 366-380.
58. Stockwell, D. & Peters, D. (1999) The GARP modelling system: problems and solutions to automated spatial prediction. International Journal of Geographical Information Science, 13, 143-158.
59. Stockwell, D.R.B. & Peterson, A.T. (2002) Effects of sample size on accuracy of species distribution models. Ecological Modelling, 148, 1-13.
60. Thuiller, W. (2003) BIOMOD: optimising predictions of species distributions and projecting potential future shifts under global change. Global Change Biology, 9, 1353-1362.
61. Thuiller, W., Lavorel, S. & Araújo, M.B. (2005) Niche properties and geographical extent as predictors of species sensitivity to climate change. Global Ecology and Biogeography, 14, 347-357.
62. Thuiller, W., Lavorel, S., Araujo, M.B., Sykes, M.T. & Prentice, I.C. (2005a) Climate change threats to plant diversity in Europe. Proceedings of the National Academy of Sciences USA, 102, 8245-8250.
63. Thuiller, W., Richardson, D.M., Pysek, P., Midgley, G.F., Hughes, G.O. & Rouget, M. (2005b) Niche-based modelling as a tool for predicting the risk of alien plant invasions at a global scale. Global Change Biology, 11, 2234-2250.
64. Tooth, L.R. & Ottenbacher, K.J. (2004) The kappa statistic in rehabilitation research: an examination. Archives of Physical Medicine and Rehabilitation, 85, 1371-1376.
65. Tsuji, N. & Tsubaki, Y. (2004) Three new algorithms to calculate the irreplaceability index for presence/absence data. Biological Conservation, 119, 487-494.
66. Vaughan, I.P. & Ormerod, S.J. (2003) Improving the quality of distribution models for conservation by addressing shortcomings in the field collection of training data. Conservation Biology, 17, 1601-1611.
67. Vaughan, I.P. & Ormerod, S.J. (2005) The continuing challenges of testing species distribution models. Journal of Applied Ecology, 42, 720-730.
68. Zweig, M.H. & Campbell, G. (1993) Receiver-operating characteristics (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39, 561-577.