Evaluating the replicability of sample results: A tutorial of double cross-validation methods

Measurement in Physical Education and Exercise Science

Volumn 8, Issue 4, 2004, Pages 227-241

  Guan, Jianmin ^a   Xiang, Ping ^b   Keating, Xiaofen Deng ^c  

^a UNIVERSITY OF TEXAS AT SAN ANTONIO   (United States)

^b TEXAS A AND M UNIVERSITY   (United States)

^c CALIFORNIA STATE UNIVERSITY   (United States)

Author keywords

Internal replicability analysis; Invariance coefficient; Research; Shrinkage

Indexed keywords

CLINICAL STUDY; INTERNAL REPLICABILITY ANALYSIS; KINESIOLOGY; REGRESSION ANALYSIS; REPRODUCIBILITY; REVIEW; SAMPLE; STATISTICAL SIGNIFICANCE; VALIDATION PROCESS;

EID: 9144268040     PISSN: 1091367X     EISSN: None     Source Type: Journal    
DOI: 10.1207/s15327841mpee0804_4     Document Type: Review

References (28)

1. Ang. R. P. (1998). Use of double cross-validation and bootstrap methods to estimate replicability of results of multiple regression. Perceptual and Motor Skills, 86, 1143-1152.
2. Catting, P. (1980). Estimation of the predictive power of a regression model. Journal of Applied Psychology, 65, 407-414.
3. Cotter, K. L. (1982). An evaluation of formula-based population squared cross-validity estimates and factor score estimates in prediction. Educational and Psychological Measurement, 42, 493-519.
4. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 17, 155-169.
5. Fish, L. (1986, November). The importance of invariance procedures as against tests of statistical significance. Paper presented at the annual meeting of the Mid-South Educational Research Association, Memphis, TN. (ERIC Document Reproduction Service No. ED278707)
6. Gillaspy, J. A. (1996, January). Evaluating result replicability: Better alternatives to significance tests. Paper presented at the annual meeting of the Southwest Educational Research Association, New Orleans, LA. (ERIC Document Reproduction Service No. ED395956)
7. Herzberg, P. A. (1969). The parameters of cross validation. Psychometrika (Monograph Supplement, No. 16).
8. Huck, S., Cormier, W., & Bounds, W. (1974). Reading statistics and research. New York: Harper & Row.
9. Levin. J. R. (1995, April). The consultant's manual of researchers' common stat-illogical disorders. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
10. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
11. Mosier, C. I. (1951). Problems and designs of cross-validation. Educational and Psychological Measurement, 11, 5-11.
12. Pedhazur, E. J. (1982). Multiple regression in behavioral research: Explanation and prediction (2nd ed.). Fort Worth, TX: Harcourt Brace.
13. Pedhazur, E. J. (1997). Multiple regression in behavioral research: Explanation and prediction (3rd ed.). USA: Wadsworth, Thomoson Learning.
14. Reinhardt, B. M. (1992). Estimating result replicability using double cross-validation and bootstrap methods. Paper presented at the annual meeting of the American Educational Research Association, San Francisco. (ERIC Document Reproduction Service No. ED346164)
15. Robinson, D. H., & Levin, J. R. (1997). Reflections on statistical and substantive significance, with a slice of replication. Educational Researcher, 26(5), 21-26.
16. Rowell, R. K. (1991, April). Double-validation in multiple regression: A method of estimating the stability of results. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL. (ERIC Document Reproduction Service No. ED334239)
17. Schmitt, D. R. (1989, November). Three methods of cross-validation: Is three a difference in conclusions. Paper presented at the annual meeting of the Mid-South Educational Research Association, Rock, AR. (ERIC Document Reproduction Service No. ED314440)
18. Silverman, S., & Keating. X. D. (2002). An analysis of research methods classes in departments of physical education/kinesiology in the United States. Research Quarterly for Exercise and Sports, 73, 1-9.
19. Stevens, J. (2002). Applied multivariate statistics for the social science (4th ed.). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
20. Thomas, J. R., & Nelson, J. K. (2001). Research methods in physical activity (4th ed.). Champaign, IL: Human Kinetics.
21. Thompson, B. (1989). Statistical significance, result importance, and result generalizability: Three noteworthy but somewhat different issues. Measurement and Evaluation in Counseling and Development, 22, 2-6.
22. Thompson, B. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361-377.
23. Thompson, B. (1994). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62, 157-176.
24. Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26-30.
25. Tukey, J. W. (1958). Bias and confidence in not-quite large samples. Annals of Mathematical Statistics, 29, 614.
26. Wherry, R. J. (1931). A new formula for predicting the shrinkage of the coefficient of multiple correlation. Annals of Mathematical Statistics, 2, 440-457.
27. White, A. E. (2000, January). Result generalizability and detection of discrepant data points: Illustrating the jackknife method. Paper presented at the annual meeting of the Southwest Educational Research Association, Dallas, TX. (ERIC Document Reproduction Service No. ED445079)
28. Zhu, W. (1997). Making bootstrap statistical inferences: A tutorial. Research Quarterly for Exercise and Sports, 68, 44-55.