Equivalence of electronic and paper administration of patient-reported outcome measures: A systematic review and meta-analysis of studies conducted between 2007 and 2013

Health and Quality of Life Outcomes

Volumn 13, Issue 1, 2015, Pages

  Muehlhausen, Willie ^a   Doll, Helen ^a   Quadri, Nuz ^a   Fordham, Bethany ^a   O'Donohoe, Paul ^b   Dogar, Nijda ^a   Wild, Diane J ^a  

^a ICON CLINICAL RESEARCH   (United States)

^b CRF Inc   (United Kingdom)

Author keywords

Equivalence; IVRS platform; Meta analysis; PDA smartphone platform; Pen and paper; Tablet touchscreen platform; Web computer platform

Indexed keywords

ARTICLE; CORRELATION COEFFICIENT; ELECTRONICS; HUMAN; INTERACTIVE VOICE RESPONSE SYSTEM; META ANALYSIS; PAPER; PATIENT REPORTED OUTCOME MEASURE; PERSONAL DIGITAL ASSISTANT; QUESTIONNAIRE; RHINO CONJUNCTIVITIS QUALITY OF LIFE QUESTIONNAIRE; SHORT FORM 36; SMARTPHONE; SYSTEMATIC REVIEW; ADULT; FEMALE; MALE; OUTCOME ASSESSMENT; QUALITY OF LIFE; REPRODUCIBILITY; STANDARDS; STATISTICS; STATISTICS AND NUMERICAL DATA;

ADULT; FEMALE; HUMANS; MALE; OUTCOME ASSESSMENT (HEALTH CARE); PAPER; PATIENT OUTCOME ASSESSMENT; QUALITY OF LIFE; REPRODUCIBILITY OF RESULTS; STATISTICS AS TOPIC; SURVEYS AND QUESTIONNAIRES;

EID: 84959156374     PISSN: None     EISSN: 14777525     Source Type: Journal    
DOI: 10.1186/s12955-015-0362-x     Document Type: Article

References (90)

1. Ofcom. Children and Parents: Media Use and Attitudes Report 2013 [ http://stakeholders.ofcom.org.uk/binaries/research/media-literacy/october-2013/research07Oct2013.pdf ]
2. Ofcom. Communications Market Report 2013 [ http://stakeholders.ofcom.org.uk/binaries/research/cmr/cmr13/2013_UK_CMR.pdf ]
3. Carlson L, Speca M, Hagen N, Taenzer P. Computerized Quality of Life Screening in a Cancer Care Clinic. J Palliat Care. 2001;17(1):46-52.
4. Ericsson. Mobility Report: on the Pulse of the Networked Society [ http://www.ericsson.com/res/docs/2013/ericsson-mobility-report-november-2013.pdf ]
5. Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient Non-Compliance with Paper Diaries. Br Med J. 2002;324:1193-4.
6. US Food and Drug Administration. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims 2009 [ http://www.fda.gov/Drugs/NewsEvents/ucm276110.htm ]
7. Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, Revicki DA, et al. Recommendations on Evidence Needed to Support Measurement Equivalence Between Electronic and Paper-Based Patient Reported Outcome (PRO) Measures: ISPOR ePRO Good Research Practices Task Force Report. Value Health. 2009;12:419-29.
8. Gwaltney CJ, Shields AL, Shiffman S. Equivalence of electronic and paper-and pencil administration of patient reported outcome measures: a meta-analytic view. Value Health. 2008;11:322-33.
9. Jamison R, Raymond S, Levine J, Slawsby E, Nedeljkovic S, Katz N. Electronic diaries for Monitoring Chronic Pain: 1-year validation study. Pain. 2000;91(3):277-85.
10. Bexellius C, Merk H, Sandin S, Nyren O, Kuhlmann-Berenzon S, Linde A, et al. Interactive voice response and web-based questionnaires for population-based infectious disease reporting. Eur J Epidemiol. 2010;25(10):693-702.
11. Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344-9.
12. Stata v12.1. StataCorp LP, 4905 Lakeway Drive, College Station, TX 77845, USA.
13. Comprehensive Meta Analysis v2. Biostat, Inc, 14 North Dean Street, Englewood, NJ 07631, USA
14. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analysis. Br Med J. 2003;327(7414):557-60.
15. Hedges LV, Olkin I. Statistical methods for meta-analysis. San Diego, CA: Academic; 1985.
16. Araujo L, Jacinto T, Moreira A, Castel-Branco MG, Delgado L, Costa-Pereira A, et al. Clinical efficacy of web-based versus standard asthma self-management. J Investig Allergol Clin Immunol. 2001;22:28-34.
17. Ashley L, Kedding A, Brown J, Velikova G, Wright P. Score equivalence of electronic and paper versions of the Social Difficulties Inventory (SDI-21): a randomised crossover trial in cancer patients. Qual Life Res. 2012;22(6):1435-40.
18. Basnov M, Kongsved SM, Bech P, Hjollund NH. Reliability of short form-36 in an Internet- and a pen-and-paper version. Inform Health Soc Care. 2009;34:53-8.
19. Beaumont JL, Victorson D, Su J, Baker CL, Wortman K, Shah H, et al. Examining web equivalence and risk factor sensitivity of the COPD population screener. Value Health. 2011;14:506-12.
20. th May 2013; New Orleans, LA United States. Conference Publication: 2013 (var. pagings).
21. Bernabe-Ortiz A, Curioso WH, Gonzales MA, Evangelista W, Castagnetto JM, Carcamo CP, et al. Handheld computers for self-administered sensitive data collection: A comparative study in Peru. BMC Med Inform Decis Mak. 2008;8:11.
22. Bernstein AN, Levinson AW, Hobbs AR, Lavery HJ, Samadi DB. Validation of online administration of the sexual health inventory for men. J Urol. 2013;189:1456-61.
23. Bishop FL, Lewis G, Harris S, McKay N, Prentice P, Thiel H, et al. A within-subjects trial to test the equivalence of online and paper outcome measures: the Roland Morris disability questionnaire. BMC Musculoskelet Disord. 2010;11:113.
24. Bushnell D, McCarrier K, Martin M, Paczkowski R, Shen W, Buesching D. Paper and web equivalence of the ensemble MDS - a tool used to collect phenotypic information prior to treatment. ISPOR 17th Annual International Meeting, June 2-6 2012; Washington, DC. Conference publication: 2012.
25. th May 2013; New Orleans, LA United States. Conference Publication: 2013: (var. pagings).
26. Bushnell DM, Martin M, Scanlon M, Chen TC, Chau D, Viswanathan H. Equivalence and measurement properties of an electronic version of the Psoriasis Symptom Inventory. Qual Life Res. 2013;23(3):897-906.
27. Carlbring P, Brunt S, Bohman S, Austin D, Richards J, Ost LG, et al. Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Comput Hum Behav. 2007;23(3):1421-34.
28. Chen TH, Li L, Sigle JM, Du YP, Wang HM, Lei J. Crossover randomized controlled trial of the electronic version of the Chinese SF-36. J Zhejiang Univ Sci B. 2007;8(8):604-8.
29. Clayer M, Davis A. Can the Toronto Extremity Salvage Score produce reliable results when used online? Clin Orthop Relat Res. 2011;469:1750-6.
30. Coles ME, Cook LM, Blake TR. Assessing obsessive compulsive symptoms and cognitions on the internet: evidence for the comparability of paper and Internet administration. Behav Res Ther. 2007;45:2232-40.
31. Cook IA, Balasubramani GK, Eng H, Friedman E, Young EA, Martin J, et al. Electronic source materials in clinical research: acceptability and validity of symptom self-rating in major depressive disorder. J Psychiatr Res. 2007;41:737-43.
32. Cubo E, Gabriel-Galan JM, Martinez JS, Alcubilla CR, Yang C, Arconada OF, et al. Comparison of office-based versus home Web-based clinical assessments for Parkinson's disease. Mov Disord. 2012;27:308-11.
33. Dalal AA, Nelson L, Gilligan T, McLeod L, Lewis S, DeMuro-Mercon C. Evaluating Patient-Reported Outcome Measurement Comparability between Paper and Alternate Versions, Using the Lung Function Questionnaire as an Example. Value Health. 2011;14:712-20.
34. Dunn JA, Arakawa R, Griest JH, Clayton AH. Assessing the Onset of Antidepressant-Induced Sexual Dysfunction Using Interactive Voice Response Technology. J Clin Psychiatr. 2007;68(4):525-32.
35. Dupont A, Wheeler J, Herndon II JE, Coan A, Zafar SY, Hood L, et al. Use of tablet personal computers for sensitive patient-reported information. J Support Oncol. 2009;7(3):91-7.
36. Gibbons C, Caudwell P, Finlayson G, King N, Blundell J. Validation of a new hand-held electronic data capture method for continuous monitoring of subjective appetite sensations. Int J Behav Nutr Phys Act. 2011;8:57.
37. Godfrey JM, Gines JL, McCarty EC. Comparison of computerized and paper versions of the Western Ontario Rotator Cuff (WORC) Index. J Shoulder Elbow Surg. 2013;22:500-4.
38. Griffiths-Jones W, Fern ED, Norton MR, Williams DH. The equivalence of remote electronic and paper collection of patient reported outcomes (PROS): A crossover trial. Value in Health.Conference: ISPOR 15th Annual European Congress: November 2012; Berlin Germany. Conference Publication: November 2012 (var. pagings).
39. Gudbergsen H, Bartels EM, Krusager P, Waehrens EE, Christensen R, Danneskiold-Samsoe B, et al. Test-retest of computerized health status questionnaires frequently used in the monitoring of knee osteoarthritis: a randomized crossover trial. BMC Musculoskelet Disord. 2011;12:190.
40. Handa VL, Barber MD, Young SB, Aronson MP, Morse A, Cundiff GW. Paper versus web-based administration of the Pelvic Floor Distress Inventory 20 and Pelvic Floor Impact Questionnaire 7. Int Urogynecol J. 2008;19:1331-5.
41. Heiberg T, Kvien TK, Dale O, Mowinckel P, Aanerud GJ, Songe-Moller AB, et al. Daily health status registration (patient diary) in patients with rheumatoid arthritis: a comparison between personal digital assistant and paper-pencil format. Arthritis Rheuma. 2007;57:454-60.
42. Hollandare F, Andersson G, Engstrom I. A comparison of psychometric properties between internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. J Med Int Res. 2010;12:49.
43. Hollen PJ, Gralla RJ, Stewart JA, Meharchand JM, Wierzbicki R, Leighl N. Can a computerized format replace a paper form in PRO and HRQL evaluation? Psychometric testing of the computer-assisted LCSS instrument (eLCSS-QL). Support Care Cancer. 2013;21:165-72.
44. Inman CJ, Wolfe F, Michaud K. Is there a difference in rheumatology patient reported outcomes when measured at home versus the clinic setting? Arthritis Rheuma. 2012;64(10):2068.
45. Jaspan HB, Flisher AJ, Myer L, Mathews C, Seebregts C, Berwick JR, et al. Brief report: Methods for collecting sexual behaviour information from South African adolescents-a comparison of paper versus personal digital assistant questionnaires. J Adolesc. 2007;30(2):353-9.
46. Jones SR, Fernyhough C, de-Wit L, Meins E. A message in the medium? Assessing the reliability of psychopathology e-questionnaires. Pers Indiv Differ. 2008;44(2):349-59.
47. Juniper EF, Riis B, Juniper BA. Development and validation of an electronic version of the Rhinoconjunctivitis Quality of Life Questionnaire. Allergy. 2007;62:1091-3.
48. Juniper EF, Langlands JM, Juniper BA. Patients may respond differently to paper and electronic versions of the same questionnaires. Respir Med. 2009;103:932-4.
49. Junker U, Freynhagen R, Langler K, Gockel U, Schmidt U, Tolle TR, et al. Paper versus electronic rating scales for pain assessment: A prospective, randomised, cross-over validation study with 200 chronic pain patients. Curr Med Res Opin. 2008;24:1797-806.
50. Kajander K, Latti M, Hatakka K, Korpela R. Letter to the Editor: An electronic diary versus a paper diary in measuring gastrointestinal symptoms. Dig Liver Dis. 2007;39:288-9.
51. Koolen BB, Pijnenburg MW, Brackel HJ, Landstra AM, van den Berg NJ, Merkus PJ, et al. Validation of a web-based version of the asthma control test and childhood asthma control test. Pediatr Pulmonol. 2011;46:941-8.
52. Lam MY, Lee H, Bright R, Korzenik R, Sands BE. Validation of interactive voice response system administration of the short inflammatory bowel disease questionnaire. Inflamm Bowel Dis. 2009;15(4):599-607.
53. Lee EH. Computerized measurement for asthma-specific quality of life: comparison with a conventional paper-and-pencil questionnaire. J Korean Acad Nurs. 2009;39:781-7.
54. Luce KH, Winzelberg AJ, Das S, Osborne MI, Bryson SW, Barr Taylor C. Reliability of self-report: paper versus online administration. Comput Hum Behav. 2007;23:1384-9.
55. Lundy JJ, Coons SJ. Measurement equivalence of interactive voice response and paper versions of the EQ-5D in a cancer patient sample. Value Health. 2011;14(6):867-71.
56. Lundy JJ, Coons SJ, Aaronson NK. Testing the measurement equivalence of paper and interactive voice response system versions of the EORTC QLQ-C30. Qual Life Res. 2014;23(1):229-37.
57. MacKenzie H, Thavaneswaran A, Chandran V, Gladman DD. Patient-reported outcome in psoriatic arthritis: A comparison of web-based versus paper-completed questionnaires. J Rheumatol. 2011;38(12):2619-24.
58. Marceau LD, Link C, Jamison RN, Carolan S. Electronic diaries as a tool to improve pain management: is there any evidence? Pain Med. 2007;8 Suppl 3:S101-9.
59. Matthew AG, Currie KL, Irvine J, Ritvo P, Santa MD, Jamnicky L, et al. Serial personal digital assistant data capture of health-related quality of life: A randomized controlled trial in a prostate cancer clinic. Health Qual Life Outcomes. 2007;5:38.
60. McCarrier KP, Bushnell DM, Martin ML, Paczkowski R, Nelson DR, Buesching D. Validation and psychometric evaluation of a brief combined assessment of depression and anxiety. Value in Health Conference: 16th Annual International Meeting of the International Society for Pharmacoeconomics and Outcomes Research, ISPOR: May 2011; Baltimore, MD United States. Conference Publication: 2011, 14.
61. McCarrier KP, Bushnell DM, Scanlon M, Nelson DR, Martin ML, Buesching D. Validation and Psychometric evaluation of a patient-reported assessment of disease burden: Evidence from a sample of adults with diabetes, depression, and rheumathoid arthritis. ISPOR 18th Annual International Meeting: May 2013; New Orleans, 2013.
62. McCarrier KP, Bushnell DM, Martin ML, Paczkowski R, Nelson DR, Buesching D. Validation and psychometric evaluation of a brief combined assessment of depression and anxiety. ISPOR 16th Annual International Meeting Research Abstracts: May 21-25 2011.
63. Mundt J, Hassan M, Lappalainen J, Morlock R. Evaluation of Psychometric equivalence between interactive voice-response (IVR) and paper versions of daily assessment scale for anxiety (DAS-A). Value Health. 2010, 13(3).
64. Parnell BA, Dunivan GC, Connolly A, Jannell ML, Wells EC, Geller EJ. Validation of web-based administration of the Pelvic Organ Prolapse/Urinary Incontinence Sexual Function Questionnaire (PISQ-12). Int Urogynecol J. 2010;22(3):357-61.
65. Raat H, Mangunkusumo RT, Landgraf JM, Kloek G, Brug J. Feasibility, reliability, and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): Internet administration compared with the standard paper version. Qual Life Res. 2007;16(4):675-85.
66. Raat H, Mangunkusumo RT, Mohangoo AD, Juniper EF, van Der Lei J. Internet and written respiratory questionnaires yield equivalent results for adolescents. Pediatr Pulmonol. 2007;42:357-61.
67. Ramachandran S, Lundy JJ, Coons SJ. Testing the measurement equivalence of paper and touch-screen versions of the EQ-5D visual analog scale (EQ VAS). Qual Life Res. 2008;17:1117-20.
68. Read JP, Farrow SM, Jaanimagi U, Ouimette P. Assessing trauma and traumatic stress via the Internet: Measurement equivalence and participant reactions. Traumatology. 2009;15(1):94-102.
69. Richardson CG, Johnson JL, Ratner PA, Zumbo BD. The influence of web-versus paper-based formats on the assessment of tobacco dependence: Evaluating the measurement invariance of the dimensions of tobacco Dependence Scale. Subst Abuse. 2009;3:1-14.
70. Richter JG, Becker A, Koch T, Nixdorf M, Widers R, Monser R, et al. Self-assessments of patients via Tablet PC in routine patient care: Comparison with standardised paper questionnaires. Ann Rheuma Dis. 2008;67(12):1739-41.
71. Salaffi F, Gasparini S, Grassi W. The use of computer touch-screen technology for the collection of patient-reported outcome data in rheumatoid arthritis: comparison with standardized paper questionnaires. Clin Exp Rheumatol. 2009;27:459-68.
72. Saunders G, Forsline A, Jacobs P. The Attitudes towards Loss of Hearing Questionnaire (ALHQ): a comparison of paper and electronic formats. J Am Acad Audiol. 2007;18:66-77.
73. Shervin N, Dorrwachter J, Bragdon CR, Shervin D, Zurakowski D, Malchau H. Comparison of paper and computer-based questionnaire modes for measuring health outcomes in patients undergoing total hip arthroplasty. J Bone Joint Surg. 2011;93(3):285-93.
74. Swartz RJ, de Moor C, Cook KF, Fouladi RT, Basen-Engquist K, Eng C, et al. Mode effects in the center for epidemiologic studies depression (CES-D) scale: personal digital assistant vs. paper and pencil administration. Qual Life Res. 2007;16:803-13.
75. Thoren ES, Andersson G, Lunner T. The use of research questionnaires with hearing impaired adults: Online vs. paper-and-pencil administration. BMC Ear Nose Throat Disord. 2012;12:14.
76. Tiplady B, Goodman K, Cummings G, Lyle D, Carrington R, Battersby C, et al. Patient-reported outcomes in rheumatoid arthritis: Assessing the equivalence of electronic and paper data collection. Patient. 2010;3(3):133-43.
77. Turvey C, Sheeran T, Dindo Wakefield B, Klein D. Validity of the Patient Health Questionnaire, PHQ-9, administered through interactive-voice-response technology. J Telemed Telecare. 2012;18(6):348-51.
78. Vallejo MA, Mananes G, Isabel CM, Diaz MI. Comparison between administration via Internet and paper-and-pencil administration of two clinical instruments: SCL-90-R and GHQ-28. J Behav Ther Exp Psychiatr. 2008;39(3):201-8.
79. Vallejo MA, Jordan CM, Diaz MI, Comeche MI, Ortega J. Psychological assessment via the internet: a reliability and validity study of online (vs paper-and-pencil) versions of the General Health Questionnaire-28 (GHQ-28) and the Symptoms Check-List-90-Revised (SCL-90-R). J Med Internet Res. 2007;9:2.
80. Varni JW, Limbers CA, Burwinkle TM, Bryant WP, Wilson DP. The ePedsQLin type 1 and type 2 diabetes. Diabetes Care. 2008;31:672-7.
81. Vinney LA, Grad JD, Connor NP. Feasibility of using a handheld electronic device for the collection of patient reported outcomes data from children. J Commun Disord. 2012;45(1):12-9.
82. Whitehead L. Methodological issues in Internet-mediated research: A randomized comparison of Internet versus mailed questionnaires. J Med Internet Res. 2011;13(4):303-8.
83. Wijndaele K, Matton L, Duvigneaud N, Lefevre J, Duquet W, Thomis M, et al. Reliability, equivalence and respondent preference of computerized versus paper-and-pencil mental health questionnaires. Comput Hum Behav. 2007;23(4):1958-70.
84. Wu RC, Thorpe K, Ross H, Micevski V, Marquez C, Straus SE. Comparing administration of questionnaires via the Internet to pen-and-paper in patients with heart failure: Randomized controlled trial. J Med Internet Res. 2009;11(1):21-9.
85. Young NL, Varni JW, Snider L, McCormick A, Sawatzky B, Scott M, et al. The Internet is valid and reliable for child-report: an example using the Activities Scale for Kids (ASK) and the Pediatric Quality of Life Inventory (PedsQL). J Clin Epidemiol. 2009;62:314-20.
86. Yu SC, Yu MN. Comparison of Internet-based and paper-based questionnaires in Taiwan using multisample invariance approach. CyberPsychol Behav. 2007;10(4):501-7.
87. Zimmerman M, Martinez JH. Web-based assessment of depression in patients treated in clinical practice: Reliability, validity, and patient acceptance. J Clin Psychiatr. 2012;73:333-38.
88. Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, Ware JE. Difference in method of administration did not significantly impact item response: an IRT-based analysis from the Patient-Reported Outcomes Measurement Information System (PROMIS) initiative. Qual Life Res 2013. doi: 10.1007/s11136-013-0451-4.
89. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420-8.
90. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1(1):30-46.