Using Bayesian statistical modelling as a bridge between quantitative and qualitative analyses: illustrated via analysis of an online teaching tool

Educational Media International

Volumn 54, Issue 4, 2017, Pages 317-359

  Low Choy, Samantha ^a   Riley, Tasha ^a   Alston Knox, Clair ^a  

^a GRIFFITH UNIVERSITY   (Australia)

Author keywords

graphical models; item response theory; mixed methods; randomization; Stereotyping

Indexed keywords

EID: 85034739520     PISSN: 09523987     EISSN: 14695790     Source Type: Journal    
DOI: 10.1080/09523987.2017.1397404     Document Type: Article

References (109)

1. Bayarri, M. J., & Berger, J. O., (2004). The interplay of Bayesian and Frequentist analysis. Statistical Science, 19(1), 58–80.
2. Bazeley, P., & Jackson, K., (2013). Qualitative data analysis with NVivo (2nd ed.). London: Sage.
3. Berger, J., (2006). The case for objective Bayesian analysis. Bayesian Analysis, 1, 385–402.10.1214/06-BA115
4. Besag, J., (1986). On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society. Series B (Methodological), 48(3), 259–302.
5. Bijak, J., & Bryant, J., (2016). Bayesian demography 250 years after Bayes. Population Studies, 70(1), 1–19.10.1080/00324728.2015.1122826
6. Box, G. E. P., Hunter, J. S., & Hunter, W. G., (2005). Statistics for experimenters: Design, innovation, and discovery (2nd ed.). Wiley Series in Probability and Statistics. Hoboken, NJ: Wiley.
7. Brown, C. H., Brincks, A., Huang, S., Perrino, T., Cruden, G., Pantin, H., … Sandler, I., (2016). Two-year impact of prevention programs on adolescent depression: An integrative data analysis approach. Prevention Science, 1–21. doi:10.1007/s11121-016-0737-1
8. Browne, W. J., & Draper, D., (2006, September). A comparison of Bayesian and likelihood-based methods for fitting multilevel models. Bayesian Analysis, 1(3), 473–514.
9. Bryman, A., (2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1(1), 8–22.10.1177/2345678906290531
10. Campitelli, G., & Macbeth, G., (2014). Hierarchical graphical Bayesian models in psychology. Revista Colombiana de Estadística, 37(2), 319–339.10.15446/rce.v37n2spe.47940
11. Chamberlin, T. C., (1890). The method of multiple working hypotheses. Science, 15, 92–96. (Reprinted in Science, 148, pp. 754–759 [1965]).
12. Chan, M. L., (2017). An explicit pragmatic approach to integrative data analysis strategies for mixed methods research. International Journal of Linguistics, 9(3), 166–184.10.5296/ijl.v9i3
13. Cohen, S., (2016). Bayesian analysis in natural language processing. San Rafael, CA: Morgan & Claypool.
14. Cooper, H., (2015). Research synthesis and meta-analysis: A step-by-step approach (Vol. 2). Thousand Oaks, CA: Sage.
15. Cooper, T. J., Baturo, A. R., Warren, E., & Doig, S. M., (2004). Young white teachers’ perceptions of mathematics learning of aboriginal and nonaboriginal students in remote communities. In M. J., Hoines, A., Fuglestad, & A., Berit (Eds.), 28th Annual Conference of the International Group for the Psychology of Mathematics Education (PME), July 14–18, 2004, Bergen, Norway. Cape Town: International Group for the Psychology of Mathematics Education. Retrieved from http://igpme.org
16. Creswell, J. W., (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Boston, MA: Pearson.
17. Dandy, J., Durkin, K., Barber, B., & Houghton, S., (2015). Academic expectations of Australian students from Aboriginal, Asian and Anglo backgrounds: Perspectives of teachers, trainee-teachers and students. International Journal of Disability, Development and Education, 62(1), 60–82.10.1080/1034912X.2014.984591
18. De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Hofman, A., Tuerlinckx, F., & Partchev, I., (2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39(12), 1–28.
19. Diamond, J., (1964). Bayesian statistics: A place in educational research? Retrieved from http://eric.ed.gov/?id=ED069707
20. Drabble, S. J., & O’Cathain, A., (2015). Moving from randomized controlled trials to mixed methods intervention evaluations. In S. N., Hesse-Biber & R. B., Johnson (Eds.), The Oxford handbook for multimethod and mixed methods research inquiry (pp. 406–425). Oxford: Oxford University Press.
21. Draper, D., Hodges, J. S., Mallows, C. L., & Pregibon, D., (1993). Exchangeability and data analysis. Journal of the Royal Statistical Society. Series A (Statistics in Society), 156(1), 9–37.10.2307/2982858
22. Düspohl, M., Frank, S., & Döll, P., (2012). A review of Bayesian networks as a participatory modeling approach in support of sustainable environmental management. Journal of Sustainable Development, 5(12), 1–18.
23. Elliott, L. P., & Brook, B. W., (2007). Revisiting Chamberlin: Multiple working hypotheses for the 21st century. BioScience, 57(7), 608–614.10.1641/B570708
24. Ellison, A. M., (1996). An introduction to Bayesian inference for ecological research and environmental decision-making. Ecological Applications, 6, 1036–1046.10.2307/2269588
25. Ellison, A. M., (2004). Bayesian inference in ecology. Ecology Letters, 7, 509–520.10.1111/ele.2004.7.issue-6
26. Etz, A., & Vandekerckhove, J., (2016). A Bayesian perspective on the reproducibility project: Psychology. PLoS ONE, 11(2), e0149794.10.1371/journal.pone.0149794
27. Fidler, F., & Cumming, G., (2005). Teaching confidence intervals: Problems and potential solutions. Paper presented at the 55th Session of the World Congress of the International Statistical Institute, Sydney, Australia. Retrieved from www.stat.auckland.ac.nz/~iase/publications.php
28. Finch, W. H., & French, B. F., (2012). Parameter estimation with mixture item response theory models: A Monte Carlo comparison of maximum likelihood and Bayesian methods. Journal of Modern Applied Statistical Methods, 11(1), 14.
29. Fisher, R., O’Leary, R. A., Low-Choy, S., Mengersen, K., Knowlton, N., Brainard, R. E., & Caley, M. J., (2015). Species richness on coral reefs and the pursuit of convergent global estimates. Current Biology, 25(4), 500–505.10.1016/j.cub.2014.12.022
30. Frigg, R., & Hartmann, S., (2017). Models in science. In E. N., Zalta (Ed.), The Stanford Encyclopedia of Philosophy (Spring 2017 ed.). Stanford, CA: Metaphysics Research Lab, Stanford University.
31. Gelman, A., (2006). Prior distributions for variance parameters in hierarchical models. Bayesian Analysis, 1(3), 515–533.10.1214/06-BA117A
32. Gelman, A., & Shalizi, C. R., (2013). Philosophy and the practice of Bayesian statistics. British Journal of Mathematical and Statistical Psychology, 66, 8–38.10.1111/j.2044-8317.2011.02037.x
33. Gigerenzer, G., Krauss, S., & Vitouch, O., (2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In D., Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 391–408). Thousand Oaks, CA: Sage.10.4135/9781412986311
34. Glaser, B., & Strauss, A., (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine.
35. Goldstein, M., (2006). Subjective Bayesian analysis: Principles and practice. Bayesian Analysis, 1, 403–420.10.1214/06-BA116
36. Goldstein, H., Browne, W. J., & Charlton, C., (2017). A Bayesian model for measurement and misclassification errors alongside missing data, with an application to higher education participation in Australia. Journal of Applied Statistics, 13, 1–14.10.1080/02664763.2017.1322558
37. Greene, J., (2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.
38. Greenland, S., & Brumback, B., (2002). An overview of relations among causal modelling methods. International Journal of Epidemiology, 31(5), 1030–1037.10.1093/ije/31.5.1030
39. Greenland, S., Daniel, R., & Pearce, N., (2016). Outcome modelling strategies in epidemiology: Traditional methods and basic alternatives. International Journal of Epidemiology, 45(2), 565–575.10.1093/ije/dyw040
40. Hesse-Biber, S., (2015). Introduction: Navigating a turbulent research landscape: Working the boundaries, tensions, diversity, and contradictions of multimethod and mixed methods inquiry. In S. N., Hesse-Biber & R. B., Johnson (Eds.), The Oxford handbook for multimethod and mixed methods research inquiry (pp. xxxiii–liii). Oxford: Oxford University Press.
41. Hesse-Biber, S., Rodriguez, D., & Frost, N. A., (2015). A qualitatively driven approach to multimethod and mixed methods research. In S. N., Hesse-Biber & R. B., Johnson (Eds.), The Oxford handbook for multimethod and mixed methods research inquiry (Chapter 1, pp. 3–20). Oxford: Oxford University Press.
42. Hilborn, R., & M., Mangel. (1997). The ecological detective: Confronting models with data. Monographs in Population Biology, 28. Princeton, NJ: Princeton University Press.
43. Hobbs, N. T., & Hilborn, R., (2006). Alternatives to statistical hypothesis testing in ecology: A guide to self teaching. Ecological Applications, 16, 5–19.10.1890/04-0645
44. Huff, D., (1993). How to lie with statistics. New York, NY: Norton.
45. Hulting, F. L., & Harville, D. A., (1991). Some Bayesian and non-Bayesian procedures for the analysis of comparative experiments and for small-area estimation: Computational aspects, Frequentist properties, and relationships. Journal of the American Statistical Association, 86(415), 557–568.10.1080/01621459.1991.10475080
46. Humphreys, M., & Jacobs, A. M., (2015). Mixing methods: A Bayesian approach. American Political Science Review, 109(4), 653–673.10.1017/S0003055415000453
47. Jackman, S., (2009). Bayesian analysis for the social sciences (Vol. 846 of Wiley Series in Statistics and Probability). Chichester: Wiley.10.1002/9780470686621
48. Kelle, U., (2015). Mixed methods and the problems of theory building and theory testing in the social sciences. In S. N., Hesse-Biber & R. B., Johnson (Eds.), The Oxford handbook for multimethod and mixed methods research inquiry (pp. 594–605). Oxford: Oxford University Press.
49. Kieftenbeld, V., Natesan, P., & Eddy, C., (2011). An item response theory analysis of the mathematics teaching efficacy beliefs instrument. Journal of Psychoeducational Assessment, 29(5), 443–454.10.1177/0734282910391062
50. Kim, N. J., Belland, B. R., & Walker, A. E., (2017, July). Effectiveness of computer-based scaffolding in the context of problem-based learning for STEM education: Bayesian meta-analysis. Educational Psychology Review, 133. doi:10.1007/s10648-017-9419-1
51. König, C., & van de Schoot, R., (2017). Bayesian statistics in educational research: A look at the current state of affairs. Educational Review, 14, 1–24.10.1080/00131911.2017.1350636
52. Kuo, T.-C., & Sheng, Y., (2015). Bayesian estimation of a multi-unidimensional graded response IRT model. Behaviormetrika, 42(2), 79–94.10.2333/bhmk.42.79
53. Levy, R., (2016). Advances in Bayesian modeling in educational research. Educational Psychologist, 51(3–4), 368–380.10.1080/00461520.2016.1207540
54. Lilford, R. J., & Braunholtz, D., (2003). Reconciling the quantitative and qualitative traditions–The Bayesian approach. Public Money & Management, 23(3), 203–208.10.1111/1467-9302.00369
55. Lindley, D. V., & Novick, M. R., (1981). The role of exchangeability in inference. Journal of the American Statistical Association, 9, 45–58.
56. Low Choy, S., (2013). Priors: Silent or active partners in Bayesian inference? In C. Alston, K., Mengersen, & A. N., Pettitt (Eds.), Case studies in Bayesian statistical modelling and analysis (pp. 30–65). London: Wiley.
57. Low Choy, S., O’Leary, R., & Mengersen, K., (2009). Elicitation by design for ecology: Using expert opinion to inform priors for Bayesian statistical models. Ecology, 90, 265–277.10.1890/07-1886.1
58. Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D., (2000). WinBUGS–A Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing, 10(4), 325–337.10.1023/A:1008929526011
59. Ly, A., & Crowshoe, L., (2015). Stereotypes are reality: addressing stereotyping in Canadian Aboriginal medical education. Medical Education, 49(6), 612–622.10.1111/medu.2015.49.issue-6
60. Mark, M. M., (2015). Mixed and multimethods in predominantly quantitative studies, especially experiments and quasi-experiments. In S. N., Hesse-Biber & R. B., Johnson (Eds.), The Oxford handbook for multimethod and mixed methods research inquiry (Chapter 2, pp. 21–41). Oxford: Oxford University Press.
61. Maxwell, J. A., Chmiel, M., & Rogers, S. E., (2015). Designing integration in multimethod and mixed methods research. In S. N., Hesse-Biber & R. B., Johnson (Eds.), The Oxford handbook for multimethod and mixed methods research inquiry (Chapter 13, pp. 223–239). Oxford: Oxford University Press.
62. McCambridge, J., Kypri, K., & Elbourne, D., (2014). In randomization we trust? There are overlooked problems in experimenting with people in behavioral intervention trials. Journal of Clinical Epidemiology, 67(3), 247–253.10.1016/j.jclinepi.2013.09.004
63. McElreath, R., (2016). Statistical rethinking: A Bayesian course with examples in R and STAN (Vol. 122). Boca Raton, FL: Chapman & Hall/CRC.
64. Merriam, S., & Tisdell, E., (2015). Qualitative research: A guide to design and implementation (4th ed.). San Francisco, CA: Jossey-Bass.10.4018/AKATM
65. Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., & Teller, E., (1953). Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21(6), 1087–1092.10.1063/1.1699114
66. Meyer, D. L., (1964). A Bayesian school superintendent. American Educational Research Journal, 1(4), 219–228.10.3102/00028312001004215
67. Mislevy, R. J., (1994). Evidence and inference in educational assessment. Psychometrika, 59(4), 439–483.10.1007/BF02294388
68. Moynihan, C., Lewis, R., Hall, E., Jones, E., Birtle, A., & Huddart, R., (2012, November). The patient deficit model overturned: A qualitative study of patients’ perceptions of invitation to participate in a randomized controlled trial comparing selective bladder preservation against surgery in muscle invasive bladder cancer (SPARE, CRUK/07/011). Trials, 13(1), 228.10.1186/1745-6215-13-228
69. Oakes, J., & Guiton, G., (1995). Matchmaking: The dynamics of high school tracking decisions. American Educational Research Journal, 32(1), 3–33.10.3102/00028312032001003
70. O’Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, R., Garthwaite, P., Jenkinson, D., … Rakow, T., (2006). Uncertain judgements: Eliciting experts’ probabilities. Chichester: Wiley.10.1002/0470033312
71. Oldehinkel, A. J., (2016). Editorial: Bayesian benefits for child psychology and psychiatry researchers. Journal of Child Psychology and Psychiatry, 57(9), 985–987.10.1111/jcpp.12619
72. Onwuegbuzie, A. J., & Hitchcock, J. H., (2015). Advanced mixed analysis approaches. In S. N., Hesse-Biber & R. B., Johnson (Eds.), The Oxford handbook for multimethod and mixed methods research inquiry (Chapter 16, pp. 275–295). Oxford: Oxford University Press.
73. Pickering, C., & Byrne, J., (2014). The benefits of publishing systematic quantitative literature reviews for PhD candidates and other early-career researchers. Higher Education Research & Development, 33(3), 534–548.10.1080/07294360.2013.841651
74. Pit-ten Cate, I. M., Krolak-Schwerdt, S., & Glock, S., (2016). Accuracy of teachers’ tracking decisions: Short- and long-term effects of accountability. European Journal of Psychology of Education, 31(2), 225–243.10.1007/s10212-015-0259-4
75. Press, S. J., (2003). Subjective and objective Bayesian statistics: Principles, models, and applications (2nd ed.). Hoboken, NJ: Wiley.
76. R Core Team. (2017). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.
77. Rainey, C., (2016). Dealing with separation in logistic regression models. Political Analysis, 24(3), 339–355.10.1093/pan/mpw014
78. Rasch, G., (1960). Probabilistic models for some intelligence and attainment tests. Copenhangen: Danmarks Paedogogiske Institut.
79. Raudenbush, S. W., Bryk, A. S., Cheong, Y. F., Congdon, R., & du Toit, M., (2004). HLM 6: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International. Retrieved from http://www.ssicentral.com/hlm/
80. Riley, T., (2010). Stigma, stereotypes, and attributional theory: A successful theoretical merger. Journal of Educational Thought, 44(2), 229–246.
81. Riley, T., (2017). Exceeding expectations: Teacher’s decision making regarding Aboriginal and Torres Strait Islander students. Unpublished manuscript, submitted for publication.
82. Riley, T., & Ungerleider, C., (2012). Self-fulfilling prophecy: How teachers’ attributions, expectations, and stereotypes influence the learning opportunities afforded Aboriginal students. Canadian Journal of Education, 35(2), 303–333.
83. Riley, T., & Low-Choy, S., (2017). Same grades, different placement: Linking a Bayesian ordinal regression with interview data to reveal the influence of teachers’ stereotypes. Unpublished manuscript, submitted for publication.
84. Rodgers, J. L., (2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65(1), 1–12.10.1037/a0018326
85. Royle, J. A., & Dorazio, R. M., (2008). Hierarchical modeling and inference in ecology: The analysis of data from populations, metapopulations and communities. London: Academic Press, Elsevier.
86. Schmidt, F. L., & Hunter, J. E., (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.10.4135/9781483398105
87. Sedimeier, P., & Gigerenzer, G., (2001). Teaching Bayesian reasoning in less than two hours. Journal of Experimental Psychology: General, 130(3), 380–400.10.1037/0096-3445.130.3.380
88. Shoemaker, J. S., Painter, I. S., & Weir, B. S., (1999). Bayesian statistics in genetics: A guide for the uninitiated. Trends in Genetics, 15(9), 354–358.10.1016/S0168-9525(99)01751-5
89. Smith, B. J., (2007). boa: An R package for MCMC output convergence assessment and posterior inference. Journal of Statistical Software, 21(11), 1–37.
90. Spiegelhalter, D. J., Adams, K. R., & Myles, J. P., (2004). Bayesian approaches to clinical trials and health-care evaluation. Statistics in Practice. Chichester: Wiley.
91. Spiegelhalter, D. J., Thomas, A., Best, N. G., & Lunn, D., (2003). WinBUGS version 14 user manual (Technical report). Cambridge: MRC Biostatistics Unit.
92. Stegmueller, D., (2013). How many countries for multilevel modeling? A comparison of Frequentist and Bayesian approaches. American Journal of Political Science, 57(3), 748–761.10.1111/ajps.2013.57.issue-3
93. Stone, L. D., Keller, C. M., Kratzke, T. M., & Strumpfer, J. P., (2014). Search for the wreckage of Air France Flight AF 447. Statistical Science, 29(1), 69–80.10.1214/13-STS420
94. Suess, E. A., Gardner, I. A., & Johnson, W. O., (2002). Hierarchical Bayesian model for prevalence inferences and determination of a country’s status for an animal pathogen. Preventive Veterinary Medicine, 55, 155–171.10.1016/S0167-5877(02)00092-2
95. TechSmith. (2016). Camtasia help. Technical Report Version 9.1. September 2017. TechSmith Corporation. Retrieved from https://support.techsmith.com/hc/en-us/article_attachments/115002225672/Camtasia_9.1_Help.pdf
96. Trafimow, D., & Marks, M., (2015). Editorial. Basic and Applied Social Psychology, 37(1), 1–2. doi:10.1080/01973533.2015.1012991
97. Tufte, E., (1983). The visual display of quantitative information. Cheshire, CT: Graphics Press.
98. Vakhitova, Z. I., & Alston-Knox, C., (2017). Non-significant p-values? Strategies to understand and better determine the importance of effects and interactions in logistic regression. Unpublished manuscript, under review.
99. van de Schoot, R., Winter, S. D., Ryan, O., Zondervan-Zwijnenburg, M., & Depaoli, S., (2017). A systematic review of Bayesian articles in psychology: The last 25 years. Psychological Methods, 22(2), 217–239.10.1037/met0000100
100. van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C., (1994). The think aloud method: A practical approach to modelling cognitive processes (Knowledge-based systems). London: Academic Press.
101. Villejoubert, G., & Mandel, D. R., (2002). The inverse fallacy: An account of deviations from Bayes’s theorem and the additivity principle. Memory and Cognition, 30(2), 171–178.10.3758/BF03195278
102. Waller, N. G., & Johnson, W. O., (1998). The non-significance of straw man arguments. Behavioral and Brain Sciences, 21(2), 226–227.10.1017/S0140525X98571166
103. Wang, C., & Nydick, S. W., (2015). Comparing two algorithms for calibrating the restricted non-compensatory multidimensional IRT model. Applied Psychological Measurement, 39, 119–134.
104. Wasserstein, R. L., & Lazar, N. A., (2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. doi:10.1080/00031305.2016.1154108
105. Weiner, B., (1974). Achievement motivation and attribution theory. Morristown, NJ: General Learning Press.
106. Western, B., & Jackman, S., (1994). Bayesian inference for comparative research. American Political Science Review, 88(02), 412–423.10.2307/2944713
107. Wollack, J. A., Bolt, D. M., Cohen, A. S., & Lee, Y. S., (2002). Recovery of item parameters in the nominal response model: A comparison of marginal maximum likelihood estimation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 26, 339–352.
108. Ziliak, S. T., & McCloskey, D. N., (2008). The psychology of psychological significance testing. In The cult of statistical significance: How the standard error costs us jobs, justice, and lives (Chapter 13, pp. 140–153). Ann Arbor: University of Michigan Press.
109. Zorn, C., (2005). A solution to separation in binary response models. Political Analysis, 13(2), 157–170.10.1093/pan/mpi009