Q-learning: A data analysis method for constructing adaptive interventions

Psychological Methods

Volumn 17, Issue 4, 2012, Pages 478-494

  Nahum Shani, Inbal ^a   Qian, Min ^b   Almirall, Daniel ^a   Pelham, William E ^c   Gnagy, Beth ^c   Fabiano, Gregory A ^d   Waxmonsky, James G ^e   Yu, Jihnhee ^d   Murphy, Susan A ^a  

^a UNIVERSITY OF MICHIGAN   (United States)

^b Columbia University ^*  (United States)

^c FLORIDA INTERNATIONAL UNIVERSITY   (United States)

^d UNIVERSITY AT BUFFALO   (United States)

^e FLORIDA INTERNATIONAL UNIVERSITY   (United States)

Author keywords

Adaptive interventions; Decision rules; Q learning; Regression

Indexed keywords

ARTICLE; DECISION SUPPORT SYSTEM; HUMAN; METHODOLOGY; RANDOMIZED CONTROLLED TRIAL (TOPIC); REGRESSION ANALYSIS; STATISTICAL ANALYSIS; STATISTICAL MODEL;

DATA INTERPRETATION, STATISTICAL; DECISION SUPPORT TECHNIQUES; HUMANS; LINEAR MODELS; RANDOMIZED CONTROLLED TRIALS AS TOPIC; REGRESSION ANALYSIS;

EID: 84874401815     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0029373     Document Type: Article

References (37)

1. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall.
2. Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182. doi:10.1037/0022-3514.51.6.1173
3. Bellman, R. E., & Dreyfus, S. E. (1962). Applied dynamic programming. Princeton, NJ: Princeton University Press.
4. Bierman, K. L., Nix, R. L., Maples, J. J., & Murphy, S. A. (2006). Examining clinical judgment in an adaptive intervention design: The Fast Track Program. Journal of Consulting and Clinical Psychology, 74, 468-481. doi:10.1037/0022-006X.74.3.468
5. Biernot, P., & Moodie, E. M. (2010). A comparison of variable selection approaches for dynamic treatment regimes. International Journal of Biostatistics, 6(1), Article 6.
6. Brown, C. H., Ten Have, T. R., Jo, B., Dagne, G., Wyman, P. A., Muthen, B.,... Gibbons, R. D. (2009). Adaptive designs for randomized trials in public health. Annual Review of Public Health, 30, 1-25. doi:10.1146/annurev.publhealth.031308.100223
7. Chakraborty, B., Murphy, S. A., & Strecher, V. (2010). Inference for non-regular parameters in optimal dynamic treatment regimes. Statistical Methods in Medical Research, 19, 317-343. doi:10.1177/0962280209105013
8. Chesney, A., & Locke, E. (1991). An examination of the relationship among goal difficulty, business strategies, and performance on a complex management simulation task. Academy of Management Journal, 34, 400-424. doi:10.2307/256448
9. Collins, L. M., Murphy, S. A., & Strecher, V. (2007). The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent ehealth interventions. American Journal of Preventive Medicine, 32(5, Suppl.), S112-S118. doi:10.1016/j.amepre.2007.01.022
10. Connell, A. M., Dishion,T. J., Yasui M., & Kavanagh, K. (2007). An adaptive approach to family intervention: Linking engagement in family-centered intervention to reductions in adolescent problem behavior. Journal of Consulting and Clinical Psychology, 75, 568-579. doi: 10.1037/0022-006X.75.4.568
11. Donovan, J. J., & Radosevich, D. J. (1998). The moderating role of goal commitment on the goal difficulty-performance relationship: A metaanalytic review and critical reanalysis. Journal of Applied Psychology, 83, 308-315. doi:10.1037/0021-9010.83.2.308
12. Erez, M. (1990). Performance quality and work motivation. In U. Kleinbeck, H. Thierry, H. Haecker, & H. Quast (Eds.), Work motivation (pp. 53-65). Hillsdale, NJ: Erlbaum.
13. Fabiano, G. A., Pelham, W. E., Waschbusch, D. A., Gnagy, E. M., Lahey, B. B., Chronis, A. M.,... Burrows-MacLean, L. (2006). A practical measure of impairment: Psychometric properties of the Impairment Rating Scale in samples of children with attention deficit hyperactivity disorder and two school-based samples. Journal of Clinical Child and Adolescent Psychology, 35, 369-385. doi:10.1207/s15374424jccp3503_3
14. Fried, Y., & Slowik, L. H. (2004). Enriching goal-setting theory with time: An integrated approach. Academy of Management Review, 29, 404-422.
15. Gunter, L., Zhu, J., & Murphy, S. A. (2011). Variable selection for qualitative interactions. Statistical Methodology, 8, 42-55, doi:10.1016/j.stamet.2009.05.003
16. Little, R. J. A., & Rubin, D. B. (1987). Statistical analysis with missing data. New York, NY: Wiley.
17. MacKinnon, D. P., Warsi, G., & Dwyer, J. H. (1995). A simulation study of mediated effect measures. Multivariate Behavioral Research, 30, 41-62. doi:10.1207/s15327906mbr3001_3
18. Marlowe, D. B., Festinger, D. S., Arabia, P. L., Dugosh, K. L., Benasutti, K. M., Croft, J. R., & McKay, J. R. (2008). Adaptive interventions in drug court: A pilot experiment. Criminal Justice Review, 33, 343-360. doi:10.1177/0734016808320325
19. Murphy, S. A. (2005). An experimental design for the development of adaptive treatment strategies. Statistics in Medicine, 24, 1455-1481. doi:10.1002/sim.2022
20. Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, B., Fabiano, G. A.,... Murphy, S. A. (2012). Experimental design and primary data analysis methods for comparing adaptive interventions. Psychological Methods, 17, 457-477.
21. Pelham, W. E., Evans, S. W., Gnagy, E. M., & Greenslade, K. E. (1992). Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders: Prevalence, factor analyses, and conditional probabilities in a special education sample. School Psychology Review, 21, 285-299.
22. Pelham, W. E., & Fabiano, G. A. (2008). Evidence-based psychosocial treatment for attention-deficit/hyperactivity disorder. Journal of Clinical Child and Adolescent Psychology, 37, 184-214. doi:10.1080/15374410701818681
23. Pelham, W. E., & Gnagy, E. M. (1999). Psychosocial and combined treatments for ADHD. Mental Retardation and Developmental Disabilities Research Reviews, 5, 225-236. doi:10.1002/(SICI)10982779(1999)5:3<225::AID-MRDD9>3.0.CO;2-E
24. Pelham, W. E., Gnagy, E. M., Greiner, A. R., Hoza, B., Hinshaw, S. P., Swanson, J. M.,... McBurnett, K. (2000). Behavioral vs. behavioral and pharmacological treatment in ADHD children attending a summer treatment program. Journal of Abnormal Child Psychology, 28, 507-525. doi:10.1023/A:1005127030251
25. Pelham, W. E., Hoza, B., Pillow, D. R., Gnagy, E. M., Kipp, H. L., Greiner, A. R.,... Fitzpatrick, E. (2002). Effects of methylphenidate and expectancy on children with ADHD: Behavior, academic performance, and attributions in a summer treatment program and regular classroom setting. Journal of Consulting and Clinical Psychology, 70, 320-335. doi:10.1037/0022-006X.70.2.320
26. Pliszka, S. (2007). Practice parameter for the assessment and treatment of children and adolescents with attention-deficit/hyperactivity disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 46, 894-921. doi:10.1097/chi.0b013e318054e724
27. Qin, J., Zhang, B., & Leung, D. H. Y. (2009). Empirical likelihood in missing data problems. Journal of the American Statistical Association, 104, 1492-1503. doi:10.1198/jasa.2009.tm08163
28. Rivera, D. E., Pew, M. D., & Collins, L. M. (2007). Using engineering control principles to inform the design of adaptive interventions: A conceptual introduction. Drug and Alcohol Dependence, 88, S31-S40. doi:10.1016/j.drugalcdep.2006.10.020
29. Robins, J. M. (1999). Association, causation, and marginal structural models. Synthese, 121, 151-179. doi:10.1023/A:1005285815569
30. Robins, J. M. (2004). Optimal structural nested models for optimal sequential decisions. In D. Y. Lin & P. Heagerty (Eds.), Proceedings of the Second Seattle Symposium on Biostatistics (pp. 189-326). New York, NY: Springer.
31. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York, NY: Wiley.
32. Schafer, J. L., & Olsen, M. K. (1998). Multiple imputation for multivariate missing-data problems: A data analyst's perspective. Multivariate Behavioral Research, 33, 545-571. doi:10.1207/s15327906mbr3304_5
33. Schaughency, E., & Ervin, R. (2006). Building capacity to implement and sustain effective practices to better serve children. School Psychology Review, 35, 155-166.
34. Shortreed, S. M., Laber, E., Pineau, J., & Murphy, S. A. (2010). Imputations methods for the clinical antipsychotic trials of intervention and effectiveness study (Technical Report No. SOCS-TR-2010.8). Montreal, Quebec, Canada: McGill University, School of Computer Science.
35. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
36. Velicer, W. F., Prochaska, J. O., & Redding, C. A. (2006). Tailored communications for smoking cessation: Past successes and future directions. Drug and Alcohol Review, 25, 49-57. doi:10.1080/09595230500459511
37. Watkins, C. J. C. H. (1989). Learning from delayed rewards (Unpublished doctoral dissertation). University of Cambridge, Cambridge, England.