The influence of base rates on correlations: An evaluation of proposed alternative effect sizes with real-world data

Behavior Research Methods

Volumn 48, Issue 3, 2016, Pages 1021-1031

  Babchishin, Kelly M ^a   Helmus, Leslie Maaike ^b  

^a UNIVERSITY OF OTTAWA   (Canada)

^b Correctional Service of Canada ^*  (Canada)

Author keywords

AUC; Base rate; Cohen s d; Correlation; Statistics in prediction studies

Indexed keywords

ALGORITHM; AREA UNDER THE CURVE; CANADA; CRIME; FACTOR ANALYSIS; FORECASTING; HUMAN; PERSONNEL; PSYCHOLOGY; REFERENCE VALUE; STATISTICAL ANALYSIS; STATISTICS AND NUMERICAL DATA;

ALGORITHMS; AREA UNDER CURVE; CANADA; CRIME; DATA INTERPRETATION, STATISTICAL; FACTOR ANALYSIS, STATISTICAL; FORECASTING; HUMANS; PSYCHOLOGY; REFERENCE VALUES; RESEARCH PERSONNEL;

EID: 84937130241     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-015-0627-7     Document Type: Article

References (58)

1. Andrews, D. A., & Bonta, J. (2010). The psychology of criminal conduct (5th ed.). Newark, NJ: LexisNexus/Anderson.
2. Andrews, D. A., Guzzo, L., Raynor, P., Rowe, R. C., Rettinger, L. J., Brews, A., & Wormith, J. S. (2012). Are the major risk/need factors predictive of both female and male reoffending? A test with the eight domains of the Level of Service/Case Management Inventory. International Journal of Offender Therapy and Comparative Criminology, 56, 113–133. doi:10.1177/0306624X10395716
3. Andrews, D. A., Zinger, I., Hoge, R. D., Bonta, J., Gendreau, P., & Cullen, F. T. (1990). Does correctional treatment work? A clinically relevant and psychologically informed meta‐analysis. Criminology, 28(3), 369–404.
4. Aron, A., Aron, E., & Coups, E. J. (2006). Statistics for psychology. Upper Saddle River, NJ: Pearson Prentice Hall.
5. Association for the Treatment of Sexual Abusers. (2014). ATSA practice guidelines for assessment, treatment interventions, and management strategies for male adult sexual abusers. Beaverton, OR: Professional Issues Committee, ATSA.
6. Babchishin, K. M., Blais, J., & Helmus, L. (2012). Do static risk factors predict differently for Aboriginal sex offenders? A multi-site comparison using the original and revised Static-99 and Static-2002 scales. Canadian Journal of Criminology and Criminal Justice, 54, 1–43. doi:10.3138/cjccj.2010.E.40
7. Bonta, J., Law, M., & Hanson, R. K. (1998). The prediction of criminal and violent recidivism among mentally disordered offenders: A meta-analysis. Psychological Bulletin, 123, 123–142. doi:10.1037/0033-2909.123.2.123
8. Bonta, J., Rugge, T., Scott, T.-L., Bourgon, G., & Yessine, A. K. (2008). Exploring the black box of community supervision. Journal of Offender Rehabilitation, 47, 248–270. doi:10.1080/10509670802134085
9. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, West Sussex, United Kingdom: Wiley.
10. Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford.
11. Brown, M. B., & Benedetti, J. K. (1977). On the mean and variance of the tetrachoric correlation coefficient. Psychometrika, 42, 347–355. doi:10.1007/BF02293655
12. Campbell, M. A., French, S., & Gendreau, P. (2009). The prediction of violence in adult offenders: A meta-analytic comparison of instruments and methods of assessment. Criminal Justice and Behavior, 36, 567–590. doi:10.1177/0093854809333610
13. Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249–253. doi:10.1177/014662168300700301
14. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. doi:10.1037/0033-2909.112.1.155
15. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.
16. Coolican, H. (2013). Research methods and statistics in psychology. New York, NY: Routledge.
17. Correctional Service of Canada. (2012-06-13a). Correctional planning and criminal profile. Commissioner’s Directive 705-6. Ottawa, ON: Author.
18. Eysenck, M. W. (2013). Simply psychology (3rd ed.). London, UK: Taylor & Francis Group, Psychology Press.
19. Field, A., & Miles, J. (2010). Discovering statistics using SAS. London: Sage.
20. Finney, S. J., & DiStefano, C. (2006). Non-normal and categorical data in structural equation modeling. Structural equation modeling: A second course, 269–314.
21. Fleiss, J. L. (1994). Measures of effect size for categorical data. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 245–260). New York: Russell Sage Foundation.
22. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466–491. doi:10.1037/1082-989X.9.4.466
23. French, S. A., & Gendreau, P. (2006). Reducing prison misconducts: What works! Criminal Justice and Behavior, 33, 185–218.
24. Gendreau, P., Little, T., & Goggin, C. (1996). A meta‐analysis of the predictors of adult offender recidivism: What works! Criminology, 34, 575–608.
25. Gendreau, P., & Smith, P. (2007). Influencing the “people who count”: Some perspectives on the reporting of meta-analytic results for prediction and treatment outcomes for offenders. Criminal Justice and Behavior, 34, 1536–1559.
26. Ghiselli, E. E., Campbell, J. P., & Zedeck, S. (1981). Measurement theory for the behavioural sciences. San Francisco, CA: W. H Freeman and Company.
27. Glockner-Rist, A., & Hoijtink, H. (2003). The best of both worlds: Factor analysis of dichotomous data using item response theory and structural equation modeling. Structural Equation Modeling, 10, 544–565. doi:10.1207/S15328007SEM1004_4
28. Gonçalves, L. C., Gonçalves, R. A., Martins, C., & Dirkzwager, A. J. E. (2014). Predicting infractions and health care utilization in prison: A meta-analysis. Criminal Justice and Behavior, 41, 921–942. doi:10.1177/0093854814524402
29. Hanson, R. K. (2008). What statistics should we use to report predictive accuracy. Crime Scene, 15(1), 15–17.
30. Helmus, L. M., & Babchishin, K. M. (2015). Primer on risk assessment and the statistics used to evaluate its accuracy. Unpublished manuscript.
31. Helmus, L., Babchishin, K. M., & Blais, J. (2012). Predictive accuracy of dynamic risk factors for Aboriginal and non-Aboriginal sex offenders: A comparison using STABLE-2007. International Journal of Offender Therapy and Comparative Criminology, 56, 856–876. doi:10.1177/0306624X11414693
32. Helmus, L., & Forrester, T. (2014). Construct validity of the Static Factors Assessment. Unpublished manuscript. Ottawa, ON: Correctional Service of Canada.
33. Helmus, L., & Hanson, R. K. (2011). More fun with statistics! How to use logistic regression to predict criminal recidivism risk. Crime Scene, 18(2), 8–12. Retrieved from http://www.cpa.ca/docs/file/Sections/Criminal%20Justice%20Psychology/Crime%20Scene%202011-10.pdf
34. Holgado-Tello, F. P., Chacón-Moscoso, S., Barbero-García, I., & Vila-Abad, E. (2010). Polychoric versus Pearsons correlations in exploratory and confirmatory factor analysis of ordinal variables. Quality and Quantity, 44, 153–166. doi:10.1007/s11135-008-9190-y
35. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Newbury Park, CA: Sage.
36. Karabinus, R. A. (1975). The r-point biserial limitation. Educational and Psychological Measurement, 35, 277–282.
37. Karl, T. R., Groisman, P. Y., Knight, R. W., & Heim, R. R. (1993). Recent variations of snow cover and snowfall in North America and their relation to precipitation and temperature variations. Journal of Climate, 6, 1327–1344. doi:10.1175/1520-0442
38. Kubinger, K. D. (2003). On artificial results due to using factor analysis for dichotomous variables. Psychology Science, 45, 106–110.
39. Ley, P. (1972). Quantitative aspects of psychological assessments: An introduction. London, UK: Duckworth.
40. Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008). AUC: a misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography, 17, 145–151. doi:10.1111/j.1466-8238.2007.00358.x
41. Maydeu-Olivares, A., García-Forero, C., Gallardo-Pujol, D., & Renom, J. (2009). Testing categorized bivariate normality with two-stage polychoric correlation estimates. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5, 131.
42. McGrath, R. E., & Meyer, G. J. (2006). When effect sizes disagree: The case of the r and d. Psychological Methods, 11, 386–401.
43. McPhail, I. V., Hermann, C. A., & Fernandez, Y. M. (2014). Correlates of emotional congruence with children in sexual offenders against children: A test of theoretical models in an incarcerated sample. Child Abuse and Neglect, 38, 336–346. doi:10.1016/j.chiabu.2013.10.002
44. Motiuk, L. L. (1993). Where are we in our ability to assess risk? Forum on Corrections Research, 5(2), 14–18.
45. Olver, M. E., Wong, S. C. P., Nicholaichuk, T., & Gordon, A. (2007). The validity and reliability of the Violence Risk Scale – Sexual Offender Version: Assessing sex offender risk and evaluating therapeutic change. Psychological Assessment, 19, 318–329. doi:10.1037/1040-3590.19.3.318
46. Pearson, K. (1903). Mathematical contributions to the theory of evolution: II. On the influence of natural selection on the variability and correlation of organs. Royal Society Philosophical Transactions, 200(series A), 1–66. Available from https://archive.org/details/philtrans02398796
47. Pepe, M. S., Janes, H., Longton, G., Leisenring, W., & Newcomb, P. (2004). Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. American Journal of Epidemiology, 159, 882–890. doi:10.1093/aje/kwh101
48. Pintea, S., & Moldovan, R. (2009). The Receiver-Operating Characteristic (ROC) analysis: Fundamentals and applications in clinical psychology. Journal of Cognitive and Behavioral Psychotherapies, 9, 49–66.
49. Quinsey, V. L., Harris, G. T., Rice, M. E., & Cormier, C. A. (2006). Violent offenders: Appraising and managing risk (2nd ed.). Washington, DC: American Psychological Association.
50. Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen’s d, and r. Law and Human Behavior, 29, 615–620. doi:10.1007/s10979-005-6832-7
51. Rice, M. E., Harris, G. T., & Lang, C. (2013). Validation of and revision to the VRAG and SORAG: The Violence Risk Appraisal Guide – Revised (VRAG-R). Psychological Assessment, 25, 951–965. doi:10.1037/a0032878
52. Ruscio, J. (2008). A probability-based measure of effect size: Robustness to base rates and other factors. Psychological Methods, 13, 19–30. doi:10.1037/1082-989X.13.1.19
53. Sackett, D. L., Deeks, J. J., & Altman, D. G. (1996). Down with odds ratios! Evidence-Based Medicine, 1, 164–166.
54. Sánchez-Meca, J., Chacón-Moscoso, S., & Marín-Martínez, F. (2003). Effect-size indices for dichotomized outcomes in meta-analysis. Psychological Methods, 8, 448–467. doi:10.1037/1082-989X.8.4.448
55. Thorndike, R. L. (1949). Personnel selection: Test and measurement techniques. New York: Wiley.
56. Viljoen, J. L., Mordell, S., & Beneteau, J. L. (2012). Prediction of adolescent sexual reoffending: A meta-analysis of the J-SOAP-II, ERASOR, J-SORRAT-II, and Static-99. Law and Human Behavior, 36, 423–438. doi:10.1037/h0093938
57. Wall, M. M., Guo, J., & Amemiya, Y. (2012). Mixture factor analysis for approximating a nonnormally distributed continuous latent factor with continuous and dichotomous observed variables. Multivariate Behavioral Research, 47, 276–313. doi:10.1080/00273171.2012.658339
58. Yang, Y., Lu, X., & Qiao, H. (2014). A robust factor analysis model for dichotomous data. Journal of Systems Science and Information, 2, 437–450.