Methodologies for establishing validity in surgical simulation studies

Surgery

Volumn 147, Issue 5, 2010, Pages 622-630

  Van Nortwick, Sara S ^a   Lendvay, Thomas S ^a   Jensen, Aaron R ^a   Wright, Andrew S ^a   Horvath, Karen D ^a   Kim, Sara ^a  

^a UNIVERSITY OF WASHINGTON SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; HUMAN; MEDLINE; PERFORMANCE MEASUREMENT SYSTEM; PREDICTIVE VALIDITY; PRIORITY JOURNAL; RELIABILITY; SIMULATION; SURGICAL TECHNIQUE; SURGICAL TRAINING; VALIDITY; VIDEOTAPE;

COMPETENCY-BASED EDUCATION; COMPUTER-ASSISTED INSTRUCTION; EDUCATION, MEDICAL, GRADUATE; EDUCATIONAL MEASUREMENT; GENERAL SURGERY; HUMANS; REPRODUCIBILITY OF RESULTS;

EID: 77950863678     PISSN: 00396060     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.surg.2009.10.068     Document Type: Article

References (99)

1. American College of Surgeons, Division of Education. Accredited Education Institutes™: enhancing patient safety through simulation. Available at: http://www.facs.org/education/accreditationprogram/. Accessed November 27, 2009.
2. Scott D.J., Cendan J.C., Pugh C.M., Minter R.M., Dunnington G.L., and Kozar R.A. The changing face of surgical education: simulation as the new paradigm. J Surg Res 147 (2008) 189-193
3. Okuda Y., Bryson E.O., DeMaria Jr. S., Jacobson L., Quinones J., Shen B., and Levine A.I. The utility of simulation in medical education: what is the evidence?. Mt Sinai J Med 76 (2009) 330-343
4. Feldman L.S., Sherman V., and Fried G.M. Using simulators to assess laparoscopic competence: ready for widespread use?. Surgery 135 (2004) 28-42
5. Gurusamy K.S., Aggarwal R., Palanivelu L., and Davidson B.R. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev 21 (2009) CD006575
6. Sutherland L.M., Middleton P.F., Anthony A., Hamdorf J., Cregan P., Scott D., and Maddern G.J. Surgical simulation: a systematic review. Ann Surg 243 (2006) 291-300
7. Sturm L.P., Windsor J.A., Cosman P.H., Cregan P., Hewett P.J., and Maddern G.J. A systematic review of skills transfer after surgical simulation training. Ann Surg 248 (2008) 166-179
8. Aucar J.A., Groch N.R., Troxel S.A., and Eubanks S.W. A review of surgical simulation with attention to validation methodology. Surg Laparosc Endosc Percutan Tech 15 (2005) 82-89
9. American Educational Research Association. American Psychological Association. National Council on Measurement in Education. Standards for educational and psychological testing (1999), American Educational Research Association, Washington (DC)
10. Gall M.D., Borg W.R., and Gall J.P. Educational research: an introduction. 6th ed. (2006), Longman Publishers, White Plains (NY)
11. In: Linn R.L. (Ed). Educational measurement (American Council on Education/Oryx series on higher education). 3rd ed (1988), Oryx Press, Phoenix (AZ)
12. Messick S. Validity. In: Linn R.L. (Ed). Educational Measurement (American Council on Education/Oryx series on higher education). 3rd ed (1988), Oryx Press, Phoenix (AZ)
13. Mitchell M., and Jolley J. Research design explained. Vol 2 (1992), Harcourt Brace Jovanovich College Publishers, Fort Worth (TX)
14. Borsboom D., Mellenbergh G.J., and van Heerden J. The concept of validity. Psychol Rev 111 (2004) 1061-1071
15. Reznick R., Regehr G., MacRae H., Martin J., and McCulloch W. Testing technical skill via an innovative "bench station" examination. Am J Surg 173 (1997) 226-230
16. Adrales G.L., Park A.E., Chu U.B., Witzke D.B., Donnelly M.B., Hoskins J.D., et al. A valid method of laparoscopic simulation training and competence assessment. J Surg Res 114 (2003) 156-162
17. Aggarwal R., Grantcharov T.P., Eriksen J.R., Blirup D., Kristiansen V.B., Funch-Jensen P., et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 244 (2006) 310-314
18. Aggarwal R., Hance J., Undre S., Ratnasothy J., Moorthy K., Chang A., and Darzi A. Training junior operative residents in laparoscopic suturing skills is feasible and efficacious. Surgery 139 (2006) 729-734
19. Aggarwal R., Ward J., Balasundaram I., Sains P., Athanasiou T., and Darzi A. Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Ann Surg 246 (2007) 771-779
20. Ahlberg G., Enochsson L., Gallagher A.G., Hedman L., Hogman C., McClusky III D.A., et al. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg 193 (2007) 797-804
21. Anastakis D.J., Regehr G., Reznick R.K., Cusimano M., Murnaghan J., Brown M., et al. Assessment of technical skills transfer from the bench training model to the human model. Am J Surg 177 (1999) 167-170
22. Andreatta P.B., Woodrum D.T., Birkmeyer J.D., Yellamanchilli R.K., Doherty G.M., Gauger P.G., et al. Laparoscopic skills are improved with LapMentor training: results of a randomized, double-blinded study. Ann Surg 243 (2006) 854-860 discussion 860-3
23. Ault G., Reznick R., MacRae H., Leadbetter W., DaRosa D., Joehl R., et al. Exporting a technical skills evaluation technology to other sites. Am J Surg 182 (2001) 254-256
24. Banks E.H., Chudnoff S., Karmin I., Wang C., and Pardanani S. Does a surgical simulator improve resident operative performance of laparoscopic tubal ligation?. Am J Obstet Gynecol 197 (2007) 541.e1-5
25. Bann S., Kwok K.F., Lo C.Y., Darzi A., and Wong J. Objective assessment of technical skills of surgical trainees in Hong Kong. Br J Surg 90 (2003) 1294-1299
26. Bann S.D., Khan M.S., and Darzi A.W. Measurement of surgical dexterity using motion analysis of simple bench tasks. World J Surg 27 (2003) 390-394
27. Beard J.D., Jolly B.C., Newble D.I., Thomas W.E., Donnelly J., and Southgate L.J. Assessing the technical skills of surgical trainees. Br J Surg 92 (2005) 778-782
28. Black M., and Gould J.C. Measuring laparoscopic operative skill in a video trainer. Surg Endosc 20 (2006) 1069-1071
29. Broe D., Ridgway P.F., Johnson S., Tierney S., and Conlon K.C. Construct validation of a novel hybrid surgical simulator. Surg Endosc 20 (2006) 900-904
30. Brydges R., Kurahashi A., Brummer V., Satterthwaite L., Classen R., and Dubrowski A. Developing criteria for proficiency-based training of surgical technical skills using simulation: changes in performances as a function of training year. J Am Coll Surg 206 (2008) 205-211
31. Chaudhry A., Sutton C., Wood J., Stone R., and McCloy R. Learning rate for laparoscopic surgical skills on MIST VR, a virtual reality simulator: quality of human-computer interface. Ann R Coll Surg Engl 81 (1999) 281-286
32. Datta V., Bann S., Aggarwal R., Mandalia M., Hance J., and Darzi A. Technical skills examination for general surgical trainees. Br J Surg 93 (2006) 1139-1146
33. Datta V., Bann S., Beard J., Mandalia M., and Darzi A. Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg 199 (2004) 603-606
34. Datta V., Bann S., Mandalia M., and Darzi A. The surgical efficiency score: a feasible, reliable, and valid method of skills assessment. Am J Surg 192 (2006) 372-378
35. Datta V., Mackay S., Mandalia M., and Darzi A. The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg 193 (2001) 479-485
36. Dauster B., Steinberg A.P., Vassiliou M.C., Bergman S., Stanbridge D.D., Feldman L.S., et al. Validity of the MISTELS simulator for laparoscopy training in urology. J Endourol 9 (2005) 541-545
37. Derossis A.M., Fried G.M., Abrahamowicz M., Sigman H.H., Barkun J.S., and Meakins J.L. Development of a model for training and evaluation of laparoscopic skills. Am J Surg 175 (1998) 482-487
38. Dubrowski A., Sidhu R., Park J., and Carnahan H. Quantification of motion characteristics and forces applied to tissues during suturing. Am J Surg 190 (2005) 131-136
39. Duffy A.J., Hogle N.J., McCarthy H., Lew J.I., Egan A., Christos P., et al. Construct validity for the LAPSIM laparoscopic surgical simulator. Surg Endosc 19 (2005) 401-405
40. Eriksen J.R., and Grantcharov T. Objective assessment of laparoscopic skills using a virtual reality stimulator. Surg Endosc 19 (2005) 1216-1219
41. Faulkner H., Regehr G., Martin J., and Reznick R. Validation of an objective structured assessment of technical skill for surgical residents. Acad Med 71 (1996) 1363-1365
42. Feldman L.S., Hagarty S.E., Ghitulescu G., Stanbridge D., and Fried G.M. Relationship between objective assessment of technical skills and subjective in-training evaluations in surgical residents. J Am Coll Surg 198 (2004) 105-110
43. Fichera A., Prachand V., Kives S., Levine R., and Hasson H. Physical reality simulation for training of laparoscopists in the 21st century. A multispecialty, multi-institutional study. JSLS 9 (2005) 125-129
44. Fried G.M., Feldman L.S., Vassiliou M.C., Fraser S.A., Stanbridge D., Ghitulescu G., et al. Proving the value of simulation in laparoscopic surgery. Ann Surg 240 (2004) 518-525 discussion 525-8
45. Gallagher A.G., Lederman A.B., McGlade K., Satava R.M., and Smith C.D. Discriminative validity of the Minimally Invasive Surgical Trainer in Virtual Reality (MIST-VR) using criteria levels based on expert performance. Surg Endosc 18 (2004) 660-665
46. Gallagher A.G., Richie K., McClure N., and McGuigan J. Objective psychomotor skills assessment of experienced, junior, and novice laparoscopists with virtual reality. World J Surg 25 (2001) 1478-1483
47. Ganai S., Donroe J.A., St Louis M.R., Lewis G.M., and Seymour N.E. Virtual-reality training improves angled telescope skills in novice laparoscopists. Am J Surg 193 (2007) 260-265
48. Goff B.A., Lentz G.M., Lee D., Fenner D., Morris J., and Mandel L.S. Development of a bench station objective structured assessment of technical skills. Obstet Gynecol 98 (2001) 412-416
49. Grantcharov T.P., Bardram L., Funch-Jensen P., and Rosenberg J. Learning curves and impact of previous operative experience on performance on a virtual reality simulator to test laparoscopic surgical skills. Am J Surg 185 (2003) 146-149
50. Grantcharov T.P., Kristiansen V.B., Bendix J., Bardram L., Rosenberg J., and Funch-Jensen P. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg 91 (2004) 146-150
51. Grantcharov T.P., Rosenberg J., Pahle E., and Funch-Jensen P. Virtual reality computer simulation. Surg Endosc 15 (2001) 242-244
52. Haluck R.S., Gallagher A.G., Satava R.M., Webster R., Bass T.L., and Miller C.A. Reliability and validity of Endotower, a virtual reality trainer for angled endoscope navigation. Stud Health Technol Inform 85 (2002) 179-184
53. Haluck R.S., Webster R.W., Snyder A.J., Melkonian M.G., Mohler B.J., Dise M.L., et al. A virtual reality surgical trainer for navigation in laparoscopic surgery. Stud Health Technol Inform 81 (2001) 171-176
54. Hyltander A., Liljegren E., Rhodin P.H., and Lonroth H. The transfer of basic skills learned in a laparoscopic simulator to the operating room. Surg Endosc 16 (2002) 1324-1328
55. Katz R., Hoznek A., Salomon L., Antiphon P., de la Taille A., and Abbou C.C. Skill assessment of urological laparoscopic surgeons: can criterion levels of surgical performance be determined using the pelvic box trainer?. Eur Urol 47 (2005) 482-487
56. Khan M.S., Bann S.D., Darzi A., and Butler P.E. Use of suturing as a measure of technical competence. Ann Plast Surg 50 (2003) 304-308 discussion 308-9
57. Kirby T.O., Numnum T.M., Kilgore L.C., and Straughn J.M. A prospective evaluation of a simulator-based laparoscopic training program for gynecology residents. J Am Coll Surg 206 (2008) 343-348
58. Kolkman W., van de Put M., Wolterbeek R., Trimbos J., and Jansen F. Laparoscopic skills simulator: construct validity and establishment of performance standards for residency training. Gynecol Surg 5 (2008) 109-114
59. Korndorffer Jr. J.R., Clayton J.L., Tesfay S.T., Brunner W.C., Sierra R., Dunne J.B., et al. Multicenter construct validity for southwestern laparoscopic videotrainer stations. J Surg Res 128 (2005) 114-119
60. Korndorffer Jr. J.R., Dunne J.B., Sierra R., Stefanidis D., Touchard C.L., and Scott D.J. Simulator training for laparoscopic suturing using performance goals translates to the operating room. J Am Coll Surg 201 (2005) 23-29
61. Larsen C.R., Grantcharov T., Aggarwal R., Tully A., Sorensen J.L., Dalsgaard T., et al. Objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator. Surg Endosc 20 (2006) 1460-1466
62. Lehmann K.S., Ritz J.P., Maass H., Cakmak H.K., Kuehnapfel U.G., Germer C.T., et al. A prospective randomized study to test the transfer of basic psychomotor skills from virtual reality to physical reality in a comparable training setting. Ann Surg 241 (2005) 442-449
63. Lendvay T.S., Casale P., Sweet R., and Peters C. VR robotic surgery: randomized blinded study of the dV-Trainer robotic simulator. Stud Health Technol Inform 132 (2008) 242-244
64. Lentz G.M., Mandel L.S., Lee D., Gardella C., Melville J., and Goff B.A. Testing surgical skills of obstetric and gynecologic residents in a bench laboratory setting: validity and reliability. Am J Obstet Gynecol 184 (2001) 1462-1468 discussion 1468-70
65. Mackay S., Datta V., Chang A., Shah J., Kneebone R., and Darzi A. Multiple Objective Measures of Skill (MOMS): a new approach to the assessment of technical ability in surgical trainees. Ann Surg 238 (2003) 291-300
66. Martin J.A., Regehr G., Reznick R., MacRae H., Murnaghan J., Hutchison C., et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84 (1997) 273-278
67. McClusky III D.A., Ritter E.M., Lederman A.B., Gallagher A.G., and Smith C.D. Correlation between perceptual, visuo-spatial, and psychomotor aptitude to duration of training required to reach performance goals on the MIST-VR surgical simulator. Am Surg 71 (2005) 13-20 discussion 20-1
68. McDougall E.M., Corica F.A., Boker J.R., Sala L.G., Stoliar G., Borin J.F., et al. Construct validity testing of a laparoscopic surgical simulator. J Am Coll Surg 202 (2006) 779-787
69. Munz Y., Almoudaris A.M., Moorthy K., Dosis A., Liddle A.D., and Darzi A.W. Curriculum-based solo virtual reality training for laparoscopic intracorporeal knot tying: objective assessment of the transfer of skill from virtual reality to reality. Am J Surg 193 (2007) 774-783
70. Munz Y., Moorthy K., Bann S., Shah J., Ivanova S., and Darzi S.A. Ceiling effect in technical skills of surgical residents. Am J Surg 188 (2004) 294-300
71. Neary P.C., Boyle E., Delaney C.P., Senagore A.J., Keane F.B., and Gallagher A.G. Construct validation of a novel hybrid virtual-reality simulator for training and assessing laparoscopic colectomy; results from the first course for experienced senior laparoscopic surgeons. Surg Endosc 22 (2008) 2301-2309
72. O'Toole R.V., Playter R.R., Krummel T.M., Blank W.C., Cornelius N.H., Roberts W.R., et al. Measuring and developing suturing technique with a virtual reality surgical simulator. J Am Coll Surg 189 (1999) 114-127
73. Paisley A.M., Baldwin P.J., and Paterson-Brown S. Validity of surgical simulation for the assessment of operative skill. Br J Surg 88 (2001) 1525-1532
74. Ritter E.M., Kindelan T.W., Michael C., Pimentel E.A., and Bowyer M.W. Concurrent validity of augmented reality metrics applied to the fundamentals of laparoscopic surgery (FLS). Surg Endosc 21 (2007) 1441-1445
75. Ro C.Y., Toumpoulis I.K., Ashton Jr. R.C., Jebara T., Schulman C., Todd G.J., et al. The LapSim: a learning environment for both experts and novices. Stud Health Technol Inform 111 (2005) 414-417
76. Rosen J., Brown J.D., Barreca M., Chang L., Hannaford B., and Sinanan M. The Blue DRAGON-a system for monitoring the kinematics and the dynamics of endoscopic tools in minimally invasive surgery for objective laparoscopic skill assessment. Stud Health Technol Inform 85 (2002) 412-418
77. Rosenthal R., Gantert W.A., Hamel C., Hahnloser D., Metzger J., Kocher T., et al. Assessment of construct validity of a virtual reality laparoscopy simulator. J Laparoendosc Adv Surg Tech A 17 (2007) 407-413
78. Schijven M., and Jakimowicz J. Construct validity: experts and novices performing on the Xitact LS500 laparoscopy simulator. Surg Endosc 17 (2003) 803-810
79. Schijven M.P., Jakimowicz J.J., and Carter F.J. How to select aspirant laparoscopic surgical trainees: establishing concurrent validity comparing Xitact LS500 index performance scores with standardized psychomotor aptitude test battery scores. J Surg Res 121 (2004) 112-119
80. Scott D.J., Bergen P.C., Rege R.V., Laycock R., Tesfay S.T., Valentine R.J., et al. Laparoscopic training on bench models: better and more cost effective than operating room experience?. J Am Coll Surg 191 (2000) 272-283
81. Scott D.J., Young W.N., Tesfay S.T., Frawley W.H., Rege R.V., and Jones D.B. Laparoscopic skills training. Am J Surg 182 (2001) 137-142
82. Seymour N.E., Gallagher A.G., Roman S.A., O'Brien M.K., Bansal V.K., Andersen D.K., et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236 (2002) 458-463 discussion 463-4
83. Sherman V., Feldman L.S., Stanbridge D., Kazmi R., and Fried G.M. Assessing the learning curve for the acquisition of laparoscopic skills on a virtual reality simulator. Surg Endosc 19 (2005) 678-682
84. Sokollik C., Gross J., and Buess G. New model for skills assessment and training progress in minimally invasive surgery. Surg Endosc 18 (2004) 495-500
85. Stefanidis D., Acker C., and Heniford B.T. Proficiency-based laparoscopic simulator training leads to improved operating room skill that is resistant to decay. Surg Innov 15 (2008) 69-73
86. Stylopoulos N., Cotin S., Maithel S.K., Ottensmeye M., Jackson P.G., Bardsley R.S., et al. Computer-enhanced laparoscopic training system (CELTS): bridging the gap. Surg Endosc 18 (2004) 782-789
87. Sugiono M., Teber D., Anghel G., Gozen A.S., Stock C., Hruza M., et al. Assessing the predictive validity and efficacy of a multimodal training programme for laparoscopic radical prostatectomy (LRP). Eur Urol 51 (2007) 1332-1339 discussion 1340
88. Swanstrom L.L., Fried G.M., Hoffman K.I., and Soper N.J. Beta test results of a new system assessing competence in laparoscopic surgery. J Am Coll Surg 202 (2006) 62-69
89. Sweet R., Kowalewski T., Oppenheimer P., Weghorst S., and Satava R. Face, content and construct validity of the University of Washington virtual reality transurethral prostate resection trainer. J Urol 172 5 Pt 1 (2004) 1953-1957
90. Taffinder N., Sutton C., Fishwick R.J., McManus I.C., and Darzi A. Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: results from randomised controlled studies using the MIST VR laparoscopic simulator. Stud Health Technol Inform 50 (1998) 124-130
91. Torkington J., Smith S.G., Rees B.I., and Darzi A. Skill transfer from virtual reality to a real laparoscopic task. Surg Endosc 15 (2001) 1076-1079
92. Uchal M., Raftopoulos Y., Tjugum J., and Bergamaschi R. Validation of a six-task simulation model in minimally invasive surgery. Surg Endosc 19 (2005) 109-116
93. van Dongen K.W., Tournoij E., van der Zee D.C., Schijven M.P., and Broeders I.A. Construct validity of the LapSim: can the LapSim virtual reality simulator distinguish between novices and experts?. Surg Endosc 21 (2007) 1413-1417
94. Van Sickle K.R., McClusky III D.A., Gallagher A.G., and Smith C.D. Construct validation of the ProMIS simulator using a novel laparoscopic suturing task. Surg Endosc 19 (2005) 1227-1231
95. Woodrum D.T., Andreatta P.B., Yellamanchilli R.K., Feryus L., Gauger P.G., and Minter R.M. Construct validity of the LapSim laparoscopic surgical simulator. Am J Surg 191 (2006) 28-32
96. Yamaguchi S., Konishi K., Yasunaga T., Yoshida D., Kinjo N., Kobayashi K., et al. Construct validity for eye-hand coordination skill on a virtual reality laparoscopic surgical simulator. Surg Endosc 21 (2007) 2253-2257
97. Zhang A., Hunerbein M., Dai Y., Schlag P.M., and Beller S. Construct validity testing of a laparoscopic surgery simulator (Lap Mentor): evaluation of surgical skill with a virtual laparoscopic training simulator. Surg Endosc 22 (2008) 1440-1444
98. Schout BM, Hendrikx AJ, Scheele F, Bemelmans BL, Scherpbier AJ. Validation andimplementation of surgical simulators: a critical review of present, past, and future. Surg Endosc. 2009 Jul 25 [Epub ahead of print]
99. Kneebone R. Simulation in surgical training: educational issues and practical implications. Med Educ 37 (2003) 267-277