Introducing Item Response Theory for measuring usability inspection processes

Conference on Human Factors in Computing Systems - Proceedings

Volumn , Issue , 2008, Pages 893-902

  Schmettow, Martin ^a   Vietze, Wolfgang ^a  

^a UNIVERSITY OF PASSAU   (Germany)

Author keywords

Experiment; Item Response Theory; Latent trait model; Measurement; Quality process; Usability inspection

Indexed keywords

HUMAN ENGINEERING; PROGRAMMING THEORY;

EXPERIMENT; ITEM RESPONSE THEORY; LATENT TRAIT MODEL; MEASUREMENT; QUALITY PROCESS; USABILITY INSPECTION;

INSPECTION;

EID: 57649208558     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1357054.1357196     Document Type: Conference Paper

References (41)

1. Alan Woolrych, Gilbert Cockton, and Mark Hindmarch. Falsification Testing for Usability Inspection Method Assessment. In Proceedings of the HCI 2004, 2004.
2. Alan Woolrych, Gilbert Cockton, and Mark Hindmarch. Knowledge Resources in Usability Inspection. In Proceedings of the HCI 2005, 2005.
3. E. B. Andersen. Conditional inference for multiple-choice questionnaires. British Journal of Mathematical and Statistical Psychology, 26:31-44, 1973.
4. Anne Boomsma, Marijtje A. J. van Duijn, and Tom A. B. Snijders, editors. Essays on Item Response Theory, volume 157 of Lecture Notes in Statistics. Springer, 2001.
5. David A. Caulton. Relaxing the homogeneity asssumption in usability testing. Behaviour & Information Technology, 20(1): 1-7, 2001.
6. Gilbert Cockton, Darryn Lavery, and Alan Woolrych. Inspection-based evaluations. In Julie A. Jacko and Andrew Sears, editors, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, pages 1118-1138. Lawrence Erlbaum Associates, Inc., Mahwah, NJ, USA, 2003.
7. Nancy S. Cole. History and Development of DIF, chapter 2, pages 25-29. In Holland and Wainer[17], 1993.
8. Neil J. Dorans and Paul W. Holland. DIF Detection and Description: Mantel-Haenszel and Standardization, chapter 3, pages 35-66. In Holland and Wainer[17], 1993.
9. Laura Faulkner. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments & Computers, 35(3):379-383, 2003.
10. Gerhard H. Fischer. Derivations of the Rasch Model. In Gerhard H. Fischer and Ivo W. Molenaar[11], pages 15-38.
11. Gerhard H. Fischer and Ivo W Molenaar, editors. Rasch models: Foundations, recent developments and applications. Springer-Verlag, New York Berlin Heidelberg, 1995.
12. Wayne D. Gray and Marilyn C. Salzman. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13(3):203-261, 1998.
13. H. Rex Hartson, Terence S. Andre, and Robert C. Williges. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 15(1): 145-181, 2003.
14. Morten Hertzum, Niels Ebbe Jacobsen, and Rolf Molich. Usability inspections by groups of specialists: perceived agreement in spite of disparate observations. In CHI '02: CHI '02 extended abstracts on Human factors in computing systems, pages 662-663, New York, NY, USA, 2002." ACM Press.
15. Herber Hoijtink and Anne Boomsma. On Person Parameter Estimation in the Dichotomous Rasch Model, chapter 4, pages 53-68. In Gerhard H. Fischer and Ivo W. Molenaar[11], 1995.
16. Herbert Hoijtink. Conditional Independence and Differential Item Functioning in the Two-Parameter Logistic Model, chapter 6, pages 109-130. Volume 157 of Boomsma et al.[4], 2001.
17. Paul W. Holland and Howard Wainer, editors. Differential Item Functioning. Lawrence Erlbaum Associates, 1993.
18. Hans Irtel. The uniqueness structure of simple latent trait models. In G. H. Fischer and D. Laming, editors, Contributions to Mathematical Psychology, Psychometrics, and Methodology, pages 265-275. Springer, 1994.
19. Susan L. Keenan, H. Rex Hartson, Dennis G. Kafura, and Robert S. Schulman. The Usability Problem Taxonomy: A framework for classification and analysis. Empirical Software Engineering, 4(1):71-104, 1999.
20. K. C. Klauer. Asymptotic properties of the ML estimator of the ability parameter when item parameters are known. Methodika, 4:23-26, 1990.
21. James R. Lewis. Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. International Journal of Human-Computer Interaction, 13(4):445-479, 2001.
22. Patrick Mair and Reinhold Hatzinger. CML based estimation of extended Rasch models with the eRm package in R. Psychology Science, 49(1):26-43, 2007.
23. Patrick Mair and Reinhold Hatzinger. Extended Rasch Modeling: The eRm Package for the Application of IRT Models in R. Journal of Statistical Software, 20(9): 1-20, May 2007.
24. I. W. Molenaar. Estimation of Item Parameters, chapter 3, pages 39-52. In Gerhard H. Fischer and Ivo W. Molenaar[11], 1995.
25. Jakob Nielsen. Finding usability problems through heuristic evaluation. In CHI '92: Proc. of the SIGCHI conference on Human factors in computing systems, pages 373-380, New York, NY, USA, 1992. ACM Press.
26. Jakob Nielsen and Thomas K. Landauer. A mathematical model of the finding of usability problems. In CHI '93: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 206-213, 1993. ACM Press.
27. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2006. ISBN 3-900051-07-0.
28. Jürgen Rost. The Growing family of Rasch Models, chapter 2, pages 25-42. Volume 157 of Boomsma et al.[4], 2001.
29. Martin Schmettow. Towards a pattern based usability inspection method for industrial practitioners. In Proceedings of the Workshop on Integrating Software Engineering and Usability Engineering (held on Interact 2005), 2005. http://www.se-hci.org/bridging/interact2005/03-Schmettow-Towards-UPI.pd%f.
30. Martin Schmettow. Introducing IRT for measuring usability inspection processes - accompaning programs. Website, September 2007. http://schmettow. info/IRT/.
31. Martin Schmettow and Sabine Niebuhr. A pattern-based usability inspection method: First empirical performance measures and future issues. In Devina Ramduny-Ellis and Dorothy Rachovides, editors, Proceedings of the HCI 2007, volume 2 of People and Computers, pages 99-102. BCS, September 2007.
32. Alicia P. Schmitt, Paul W. Holland, and Neil J. Dorans. Evaluating Hypotheses about Differential Item Functioning, chapter 14, pages 281-315. In Holland and Wainer[17], 1993.
33. Andrew Sears. Heuristic walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9(3):213-234, 1997.
34. Jared Spool and Will Schroeder. Testing web sites: Five users is nowhere near enough. In Proceedings of ACM CHI Conference on Human Factors in Computing, Seattle, WA, USA, 2001.
35. Hoi K. Suen. Principles of Test Theories. Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1990.
36. David Thissen, Lynne Steinberg, and Howard Wainer. Detection of Differential Item Functioning Using the Parameters of Item Response Model, chapter 4, pages 67-113. In Holland and Wainer[17], 1993.
37. N. D. Verhelst and C. A. D. Glas. A dynamic generalization of the Rasch model. Psychometrika, 58(3):395-415, 1993.
38. N. D. Verhelst and C. A. W Glas. Testing the Rasch Model, chapter 5, pages 69-96. In Gerhard H. Fischer and Ivo W. Molenaar[11], 1995.
39. Robert A. Virzi. Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34(4):457-468, 1992.
40. Cathleen Wharton, John Rieman, Clayton Lewis, and Peter Poison. The cognitive walkthough method: A practitioner's guide. In Jakob Nielsen and Robert L. Mack, editors, Usability Inspection Methods, pages 105-140. John Wiley & Son, 1994.
41. A. Woolrych and G. Cockton. Why and when five test users aren't enough. In J. Vanderdonckt, A. Blandford, and A. Derycke, editors, Proceedings of IHM-HCI 2001 Conference, volume 2, pages 105-108. Cepadeus, Toulouse, France, 2001.