Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure

Ecography

Volumn 40, Issue 8, 2017, Pages 913-929

  Roberts, David R ^a   Bahn, Volker ^c   Ciuti, Simone ^a   Boyce, Mark S ^d   Elith, Jane ^e   Guillera Arroita, Gurutzeta ^e   Hauenstein, Severin ^a   Lahoz Monfort, José J ^e   Schröder, Boris ^f   Thuiller, Wilfried ^g   Warton, David I ^h   Wintle, Brendan A ^e   Hartig, Florian ^a,b   Dormann, Carsten F ^a  

^a UNIVERSITY OF FREIBURG   (Germany)

^b UNIVERSITY OF REGENSBURG   (Germany)

^c WRIGHT STATE UNIVERSITY   (United States)

^d UNIVERSITY OF ALBERTA   (Canada)

^e UNIVERSITY OF MELBOURNE   (Australia)

^f TECHNICAL UNIVERSITY OF BRAUNSCHWEIG   (Germany)

^g UNIVERSITÉ JOSEPH FOURIER   (France)

^h UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

CORRELATION; DATA SET; ECOLOGICAL APPROACH; HIERARCHICAL SYSTEM; INTERPOLATION; LITERATURE REVIEW; MODEL VALIDATION; PERFORMANCE ASSESSMENT; PHYLOGENETICS; SPATIAL ANALYSIS; TEMPORAL ANALYSIS;

EID: 85014318718     PISSN: 09067590     EISSN: 16000587     Source Type: Journal    
DOI: 10.1111/ecog.02881     Document Type: Review

References (101)

1. Amesbury, M. J. et al. 2013. Statistical testing of a new testate amoeba-based transfer function for water-table depth reconstruction on ombrotrophic peatlands in northeastern Canada and Maine, United States. – J. Quat. Sci. 28: 27–39.
2. Anderson, R. P. 2013. A framework for using niche models to estimate impacts of climate change on species distributions. – Ann. N. Y. Acad. Sci. 1297: 8–28.
3. Araújo, M. B. et al. 2005. Validation of species-climate impact models under climate change. – Global Change Biol. 11: 1504–1513.
4. Arlot, S. and Celisse, A. 2010. A survey of cross-validation procedures for model selection. – Stat. Surv. 4: 40–79.
5. Austin, M. P. 2002. Spatial prediction of species distribution: an interface between ecological theory and statistical modelling. – Ecol. Modell. 157: 101–118.
6. BahnV. 2009. A new method for evaluating species distribution models. – 94th Ecol. Soc. Am. Ann. Meeting.
7. Bahn, V. and McGill, B. J. 2007. Can niche-based distribution models outperform spatial interpolation? – Global Ecol. Biogeogr. 16: 733–742.
8. Bahn, V. and McGill, B. J. 2013. Testing the predictive performance of distribution models. – Oikos 122: 321–331.
9. Bergmeir, C. and Benitez, J. M. 2012. On the use of cross- validation for time series predictor evaluation. – Inf. Sci. 191: 192–213.
10. Bjørnstad, O. N. and Grenfell, B. T. 2001. Noisy clockwork: time series analysis of population fluctuations in animals. – Science 293: 638–643.
11. Blois, J. L. et al. 2013. Space can substitute for time in predicting climate-change effects on biodiversity. – Proc. Natl Acad. Sci. USA 110: 9374–9379.
12. Boyce, M. S. et al. 2002. Evaluating resource selection functions. – Ecol. Modell. 157: 281–300.
13. Breiman, L. 2001. Random forests. – Machine Learning 45: 5–32.
14. Breslow, N. E. and Clayton, D. G. 1993. Approximate inference in generalized linear mixed models. – J. Am. Stat. Ass. 88: 9–25.
15. BrockwellP. J. and DavisR. A. 1996. Introduction to time series and forecasting. – Springer.
16. Broennimann, O. et al. 2012. Measuring ecological niche overlap from occurrence and spatial environmental data. – Global Ecol. Biogeogr. 21: 481–497.
17. Bulluck, L. et al. 2006. Spatial and temporal variations in species occurrence rate affect the accuracy of occurrence models. – Global Ecol. Biogeogr. 15: 27–38.
18. Burman, P. et al. 1994. A cross-validatory method for dependent data. – Biometrika 81: 351–358.
19. Buston, P. M. and Elith, J. 2011. Determinants of reproductive success in dominant pairs of clownfish: a boosted regression tree analysis. – J. Anim. Ecol. 80: 528–538.
20. Capinha, C. et al. 2012. Predicting the impact of climate change on the invasive decapods of the Iberian inland waters: an assessment of reliability. – Biol. Invas. 14: 1737–1751.
21. Chee, Y. E. and Elith, J. 2012. Spatial data for modelling and management of freshwater ecosystems. – Int. J. Geogr. Inf. Sci. 26: 2123–2140.
22. Coe, P. K. et al. 2011. Validation of elk resource selection models with spatially independent data. – J. Wildl. Manage. 75: 159–170.
23. Cornwell, W. K. et al. 2006. A trait-based test for habitat filtering: convex hull volume. – Ecology 87: 1465–1471.
24. CressieN. 1993. Statistics for spatial data. – Wiley.
25. Dormann, C. F. et al. 2007. Methods to account for spatial autocorrelation in the analysis of species distributional data: a review. – Ecography 30: 609–628.
26. Eiserhardt, W. L. et al. 2013. Dispersal and niche evolution jointly shape the geographic turnover of phylogenetic clades across continents. – Sci. Rep. UK 3: 1164.
27. Elith, J. and Graham, C. H. 2009. Do they? How do they? WHY do they differ? On finding reasons for differing performances of species distribution models. – Ecography 32: 66–77.
28. Elith, J. et al. 2006. Novel methods improve prediction of species’ distributions from occurrence data. – Ecography 29: 129–151.
29. Elith, J. et al. 2008. A working guide to boosted regression trees. – J. Anim. Ecol. 77: 802–813.
30. Elith, J. et al. 2010. The art of modelling range-shifting species. – Meth. Ecol Evol 1: 330–342.
31. Felsenstein, J. 1985. Phylogenies and the comparative method. – Am. Nat. 125: 1–15.
32. Fielding, A. H. and Bell, J. F. 1997. A review of methods for the assessment of prediction errors in conservation presence/absence models. – Environ. Conserv. 24: 38–49.
33. Fithian, W. et al. 2015. Bias correction in species distribution models: pooling survey and collection data for multiple species. – Meth. Ecol. Evol. 6: 424–438.
34. Fløjgaard, C. et al. 2009. Ice age distributions of European small mammals: insights from species distribution modelling. – J. Biogeogr. 36: 1152–1163.
35. Foley, A. M. et al. 2012. Current methods and advances in forecasting of wind power generation. – Renew. Energ. 37: 1–8.
36. FotheringhamA. S. et al. 2002. Geographically weighted regression: the analysis of spatially varying relationships. – Wiley.
37. Grafen, A. 1989. The phylogenetic regression. – Phil. Trans. R. Soc. B 326: 119–157.
38. HarveyP. H. and PagelM. D. 1991. The comparative method in evolutionary biology. – Oxford Univ. Press.
39. HastieT. et al. 2009. The elements of statistical learning: data mining, inference and prediction. – Springer.
40. Hawkins, D. M. 2004. The problem of overfitting. – J. Chem. Inf. Comp. Sci. 44: 1–12.
41. Heikkinen, R. K. et al. 2012. Does the interpolation accuracy of species distribution models come at the expense of transferability? – Ecography 35: 276–288.
42. Ives, A. R. and Zhu, J. 2006. Statistics for correlated data: phylogenies, space and time. – Ecol. Appl. 16: 20–32.
43. Kennard, R. W. and Stone, L. A. 1969. Computer aided design of experiments. – Technometrics 11: 137–148.
44. Koenig, W. D. 1999. Spatial autocorrelation of ecological phenomena. – Trends Ecol. Evol. 14: 22–26.
45. KohaviR. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. – 14th Int. Joint Conf. on Artifical Intelligence. Morgan Kaufmann Publ., San Francisco, CAUSA, pp. 1137–1143.
46. Koper, N. and Manseau, M. 2009. Generalized estimating equations and generalized linear mixed-effects models for modelling resource selection. – J. Appl. Ecol. 46: 590–599.
47. Kuhn, I. 2007. Incorporating spatial autocorrelation may invert observed patterns. – Divers. Distrib. 13: 66–69.
48. KuhnM. and JohnsonK. 2013. Applied predictive modeling. – Springer.
49. Larimore, W. E. and Mehra, R. K. 1985. The problem of overfitting data. – Byte 10: 167–178.
50. Le Rest, K. et al. 2014. Spatial leave-one-out cross-validation for variable selection in the presence of spatial autocorrelation. – Global Ecol. Biogeogr. 23: 811–820.
51. Leftwich, K. N. et al. 1997. Factors influencing behavior and transferability of habitat models for a benthic stream fish. – Trans. Am. Fish. Soc. 126: 725–734.
52. Legendre, P. 1993. Spatial autocorrelation – trouble or new paradigm. – Ecology 74: 1659–1673.
53. Legendre, P. and Fortin, M. J. 1989. Spatial pattern and ecological analysis. – Vegetatio 80: 107–138.
54. Lele, S. R. et al. 2013. Selection, use, choice and occupancy: clarifying concepts in resource selection studies. – J. Anim. Ecol. 82: 1183–1191.
55. Lieske, D. J. and Bender, D. J. 2011. A robust test of spatial predictive models: geographic cross-validation. – J. Environ. Inf. 17: 91–101.
56. Lundberg, P. et al. 2000. Population variability in space and time. – Trends Ecol. Evol. 15: 460–464.
57. Mackey, B. G. and Lindenmayer, D. B. 2001. Towards a hierarchical framework for modelling the spatial distribution of animals. – J. Biogeogr. 28: 1147–1166.
58. Maguire, K. C. et al. 2015. Modeling species and community responses to past, present, and future episodes of climatic and ecological change. – Annu. Rev. Ecol. Evol. Syst. 46: 343–368.
59. Mankin, J. B. et al. 1977. The importance of validation in ecosystem analysis. – In: Innis, G. (ed.), New directions in the analysis of ecological systems. Part I. The Society for Computer Simulation, pp. 63–71.
60. ManlyB. F. J. et al. 2002. Resource selection by animals: statistical design and analysis for field studies. – Kluwer.
61. Mesgaran, M. B. et al. 2014. Here be dragons: a tool for quantifying novelty due to covariate range and correlation change when projecting species distribution models. – Divers. Distrib. 20: 1147–1159.
62. Miller, J. et al. 2007. Incorporating spatial dependence in predictive vegetation models. – Ecol. Modell. 202: 225–242.
63. MostellerF. and TukeyJ. W. 1977. Data analysis and regression: a second course in statistics. – Addison Wesley.
64. Newbold, T. et al. 2015. Global effects of land use on local terrestrial biodiversity. – Nature 520: 45–50.
65. Ohlson, J. A. and Zhang, X. J. 1999. On the theory of forecast horizon in equity valuation. – J. Accounting Res. 37: 437–449.
66. Olden, J. D. et al. 2002. Predictive models of fish species distributions: a note on proper validation and chance predictions. – Trans. Am. Fish. Soc. 131: 329–336.
67. Otis, D. L. and White, G. C. 1999. Autocorrelation of location estimates and the analysis of radiotracking data. – J. Wildl. Manage. 63: 1039–1044.
68. Pearson, R. G. 2006. Climate change and the migration capacity of species. – Trends Ecol. Evol. 21: 111–113.
69. Petchey, O. L. et al. 2015. The ecological forecast horizon, and examples of its uses and determinants. – Ecol. Lett. 18: 597–611.
70. Picard, R. R. and Cook, R. D. 1984. Cross-validation of regression-models. – J. Am. Stat. Ass. 79: 575–583.
71. Power, M. 1993. The predictive validation of ecological and environmental-models. – Ecol. Modell. 68: 33–50.
72. Racine, J. 2000. Consistent cross-validatory model-selection for dependent data: hv-block cross-validation. – J. Econometrics 99: 39–61.
73. Radosavljevic, A. and Anderson, R. P. 2014. Making better MAXENT models of species distributions: complexity, overfitting and evaluation. – J. Biogeogr. 41: 629–643.
74. Reineking, B. and Schröder, B. 2003. Computer-intensive methods in the analysis of species–habitat relationships. – In: Breckling, B. et al. (eds), Gene Bits und Ökosysteme: Implikationen neuer Technologien für die ökologische Theorie. Peter Lang, pp. 165–182.
75. Revell, L. J. 2010. Phylogenetic signal and linear regression on species data. – Meth. Ecol. Evol. 1: 319–329.
76. Roberts, D. R. and Hamann, A. 2012a. Method selection for species distribution modelling: are temporally or spatially independent evaluations necessary? – Ecography 35: 792–802.
77. Roberts, D. R. and Hamann, A. 2012b. Predicting potential climate change impacts with bioclimate envelope models: a palaeoecological perspective. – Global Ecol. Biogeogr. 21: 121–133.
78. Roberts, D. R. et al. 2016. Data from: Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. – Dryad Digital Repository, < http://dx.doi.org/10.5061/dryad.737gk>.
79. Rooney, S. M. et al. 1998. Autocorrelated data in telemetry studies: time to independence and the problem of behavioural effects. – Mamm. Rev. 28: 89–98.
80. Rue, H. et al. 2009. Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations. – J. R. Stat. Soc. B 71: 319–392.
81. Rykiel, E. J. 1996. Testing ecological models: the meaning of validation. – Ecol. Modell. 90: 229–244.
82. Schibalski, A. et al. 2014. Climate change shifts environmental space and limits transferability of treeline models. – Ecography 37: 321–335.
83. Schröder, B. and Richter, O. 2000. Are habitat models transferable in space and time? – J. Nat. Conserv. 8: 195–205.
84. Shao, J. 1993. Linear-model selection by cross-validation. – J. Am. Stat. Ass. 88: 486–494.
85. Snee, R. D. 1977. Validation of regression-models – methods and examples. – Technometrics 19: 415–428.
86. Stephens, P. A. et al. 2016. Consistent response of bird populations to climate change on two continents. – Science 352: 84–87.
87. Stone, M. 1974. Cross-validatory choice and assessment of statistical predictions. – J. R. Stat. Soc. B 36: 111–147.
88. Sumpter, D. J. T. 2006. The principles of collective animal behaviour. – Phil. Trans. R. Soc. B 361: 5–22.
89. Telford, R. J. and Birks, H. J. B. 2009. Evaluation of transfer functions in spatially structured environments. – Quat. Sci. Rev. 28: 1309–1316.
90. Telford, R. J. et al. 2004. Biases in the estimation of transfer function prediction errors. – Paleoceanography 19: PA4014.
91. Thomas, J. A. and Bovee, K. D. 1993. Application and testing of a procedure to evaluate transferability of habitat suitability criteria. – River Res. Appl. 8: 285–294.
92. Thuiller, W. et al. 2004. Effects of restricting environmental range of data to project current and future species distributions. – Ecography 27: 165–172.
93. Trachsel, M. and Telford, R. J. 2016. Technical note: estimating unbiased transfer-function performances in spatially structured environments. – Climate Past 12: 1215–1223.
94. Veloz, S. D. 2009. Spatially autocorrelated sampling falsely inflates measures of accuracy for presence-only niche models. – J. Biogeogr. 36: 2290–2299.
95. Verbyla, D. L. and Litvaitis, J. A. 1989. Resampling methods for evaluating classification accuracy of wildlife habitat models. – Environ. Manage. 13: 783–787.
96. Wenger, S. J. and Olden, J. D. 2012. Assessing transferability of ecological models: an underappreciated aspect of statistical validation. – Meth. Ecol. Evol. 3: 260–267.
97. Wiens, T. S. et al. 2008. Three way k-fold cross-validation of resource selection functions. – Ecol. Modell. 212: 244–255.
98. Williams, J. W. and Jackson, S. T. 2007. Novel climates, no-analog communities and ecological surprises. – Front. Ecol. Environ. 5: 475–482.
99. Williams, J. W. et al. 2001. Dissimilarity analyses of late-Quaternary vegetation and climate in eastern North America. – Ecology 82: 3346–3362.
100. Wu, J. G. and David, J. L. 2002. A spatially explicit hierarchical approach to modeling complex ecological systems: theory and applications. – Ecol. Modell. 153: 7–26.
101. Zurell, D. et al. 2012. Predicting to new environments: tools for visualizing model behaviour and impacts on mapped distributions. – Divers. Distrib. 18: 628–634.