Heuristic principles and differential judgments in the quality assessment of information quality

Journal of the Association for Information Systems

Volumn 18, Issue 5, 2017, Pages 403-432

  Arazy, Ofer ^a   Kopak, Rick ^b   Hadar, Irit ^a  

^a UNIVERSITY OF HAIFA   (Israel)

^b UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Agreement; Assessment; Consistency; Heuristic principles; Information quality; Rating

Indexed keywords

CONTRACTS; INFORMATION SYSTEMS; RATING;

ASSESSMENT; CONSISTENCY; HEURISTIC PRINCIPLE; INFORMATION QUALITY; INFORMATION QUALITY DIMENSIONS; INTER-RATER AGREEMENTS; QUALITY ASSESSMENT; WIKIPEDIA ARTICLES;

INFORMATION ANALYSIS;

EID: 85020113725     PISSN: 15369323     EISSN: 15583457     Source Type: Journal    
DOI: 10.17705/1jais.00458     Document Type: Article

References (72)

1. Agichtein, E., Castillo, C., Donato, D., Gionis, A., & Mishne, G. (2008). Finding high-quality content in social media. In Proceedings of the International Conference on Web Search and Data Mining (pp. 183-194).
2. Arazy, O., & Kopak, R. (2011). On the measurability of information quality. Journal of the American Society for Information Science and Technology, 62(1), 89-99.
3. Benkler, Y. (2006). The wealth of networks: How social production transforms markets and freedom. New Haven, CT: Yale University Press.
4. Chaiken, S. (1980). Heuristic versus systematic information processing and the use of source versus message cues in persuasion. Journal of Personality and Social Psychology, 39(5), 752-766.
5. Chen, S., & Chaiken, S. (1999). The heuristic-systematic model in its broader context. In S. Chaiken & Y. Trope (Eds.), Dual-process theories in social psychology. New York: The Guildford Press.
6. Chesney, T. (2006). An empirical examination of Wikipedia’s credibility. First Monday, 11(11).
7. Cicchetti, D. V., & Heavens, R. (1981). A computer program for determining the significance of the difference between pairs of independently derived values of kappa or weighted kappa. Educational and Psychological Measurement, 41(1), 189-193.
8. Eppler, M. J. (2006). Managing information quality: Increasing the value of information in knowledge-intensive products and processes (2nd ed.). Berlin: Springer.
9. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis. Cambridge, MA: MIT Press.
10. Evans, J. S. B. (2006). The heuristic-analytic theory of reasoning: Extension and evaluation. Psychonomic Bulletin & Review, 13(3), 378-395.
11. Eysenbach, G., Powell, J., Kuss, O., & Sa, E. R. (2002). Empirical studies assessing the quality of health information for consumers on the World Wide Web. Journal of the American Medical Association, 287(20), 2691-2700.
12. Fallis, D. (2008). Toward an epistemology of Wikipedia. Journal of the American Society for Information Science and Technology, 59(10), 1662-1674.
13. Fink-Shamit, N., & Bar-Ilan, J. (2008). Information quality assessment on the Web-an expression of behavior. Information Research, 13(4).
14. Flanagin, A. J., & Metzger, M. J. (2003). The perceived credibility of personal web page information as influenced by the sex of the source. Computers in Human Behavior, 19(6), 683-701.
15. Flanagin, A. J., & Metzger, M. J. (2007). The role of site features, user attributes, and information verification behaviors on the perceived credibility of Web-based information. New Media & Society, 9(2), 319-342.
16. Fleiss, J. L. (1981). The measurement of interrater agreement. In J. L. Fleiss (Ed.), Statistical methods for rates and proportions (pp. 212-236). New York: John Wiley.
17. Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33(3), 613-619.
18. Fleiss, J. L., Cohen, J., & Everitt, B. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72(5), 323-327.
19. Gagliardi, A., & Jadad, A. R. (2002). Examination of instruments used to rate quality of health information on the internet: chronicle of a voyage with an unclear destination. BMJ, 324(9), 569-573.
20. Gigerenzer, G. (1994). Where do new ideas come from? In M. A. Boden (Ed.), Dimensions of creativity (pp. 53-74). Cambridge, MA: MIT Press.
21. Gigerenzer, G. (2008). Rationality for mortals: How people cope with uncertainty. Oxford, UK: Oxford University Press.
22. Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology, 62, 451-482.
23. Gigerenzer, G., & Todd, P. M. (1999). Fast and frugal heuristics: The adaptive toolbox. In G. Gigerenzer & P. M. Todd (Eds.), Simple heuristics that make us smart (pp. 3-34). Oxford, UK: Oxford University Press.
24. Gigerenzer, G., Hertwig, R., & Pachur, T. (Eds.). (2011). Heuristics: The foundations of adaptive behavior. New York: Oxford University Press.
25. Giles, J. (2005). Internet encyclopaedias go head to head. Nature, 438(7070), 900-901.
26. Gilovich, T., Griffin, D. W., & Kahneman, D. (2002). Heuristics and biases: The psychology of intuitive judgement. New York: Cambridge University Press.
27. Haggard, E. A. (1958). Intraclass correlation and the analysis of variance. New York, NY: Dryden Press.
28. Harper, F. M., Raban, D., Rafaeli, S., & Konstan, J. A. (2008). Predictors of answer quality in online Q&A sites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 865-874).
29. Hilligoss, B., & Rieh, S. Y. (2008). Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context. Information Processing & Management, 44(4), 1467-1484.
30. Kahneman, D. (2003). Maps of bounded rationality: psychology for behavioral economics. The American Economic Review, 93(5), 1449-1475.
31. Kahneman, D., & Frederick, S. (2002). Representativeness revisited: Attribute substitution in intuitive judgment. In In T. Gilovich, D. Griffin, & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment. New York: Cambridge University Press.
32. Kahneman, D., Slovic, P., & Tversky, A. (1983). Judgment under uncertainty: Heuristics and biases. New York: Cambridge University Press.
33. Kim, P., Eng, T. R., Deering, M. J., & Maxfield, A. (1999). Published criteria for evaluating health related web sites: Review. BMJ, 318(7184), 647-649.
34. Klein, K. J., Conn, A. B., Smith, D. B., & Sorra, J. S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86(1), 3-16.
35. Knight, S., & Burn, J. (2005). Developing a framework for assessing information quality on the World Wide Web. Informing Science, 8, 159-172.
36. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.
37. Lankes, R. D. (2008). Trusting the internet: New approaches to credibility tools. In M. J. Metzger & A. J. Flanagin (Eds.), Digital media, youth, and credibility (pp. 101-122). Boston, MA: MIT Press.
38. LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815-852.
39. Lee, Y. W., Strong, D. M., Kahn, B. K., & Wang, R. Y. (2002). AIMQ: A methodology for information quality assessment. Information & Management, 40(2), 133-146.
40. Lim, S. (2009). How and why do college students use Wikipedia? Journal of the American Society for Information Science and Technology, 60(11), 2189-2202.
41. Lim, S. (2013). College students’ credibility judgments and heuristics concerning Wikipedia. Information Processing & Management, 49(2), 405-419.
42. Linstone, H. A., & Turoff, M. (1976). The Delphi method: Techniques and applications. New York: Addison-Wesley.
43. Liu, Z. (2004). Perceptions of credibility of scholarly information on the Web. Information Processing & Management, 40(6), 1027-1038.
44. Lukyanenko, R., Parsons, J., & Wiersma, Y. (2014). The IQ of the crowd: Understanding and improving information quality in structured user-generated content. Information Systems Research, 25(4), 669-689.
45. Luyt, B., Aaron, T. C. H., Thian, L. H., & Hong, C. K. (2008). Improving Wikipedia’s accuracy: Is edit age a solution? Journal of the American Society for Information Science and Technology, 59(2), 318-330.
46. McDowell, L. (2002). Electronic information resources in undergraduate education: An exploratory study of opportunities for student learning and independence. British Journal of Educational Technology, 33(3), 255-266.
47. Metzger, M. J. (2007). Making sense of credibility on the Web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, 58(13), 2078-2091.
48. Metzger, M. J., & Flanagin, A. J. (2013). Credibility and trust of information in online environments: The use of cognitive heuristics. Journal of Pragmatics, 59, 210-220.
49. Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010). Social and heuristic approaches to credibility evaluation online. Journal of Communication, 60(3), 413-439.
50. Michnik, J., & Lo, M. C. (2009). The assessment of the information quality with the aid of multiple criteria analysis. European Journal of Operational Research, 195(3), 850-856.
51. Moskal, B. M. (2000). Scoring rubrics: What, when and how? Practical Assessment, Research & Evaluation, 7(10)
52. Nurse, J. R. C., Creese, S., Goldsmith, M., & Lamberts, K. (2011). Trustworthy and effective communication of cybersecurity risks: A review. In Proceedings of The 1st Workshop on Socio-Technical Aspects in Security and Trust (pp. 60-68).
53. Oakleaf, M. (2009). Using rubrics to assess information literacy: An examination of methodology and interrater reliability. Journal of the American Society for Information Science and Technology, 60(5), 969-983.
54. Petty, R. E., & Cacioppo, J. T. (1986). The elaboration likelihood model of persuasion. In L. Berkowitz (Ed.), Advances in experimental social psychology (pp. 123-205). New York: Academic Press.
55. Petty, R. E., & Wegener, D. T. (1999). The elaboration likelihood model: Current status and controversies. In S. Chaiken & Y. Trope (Eds.), Dual-process theories in social psychology. New York: The Guildford Press.
56. Rains, S. A., & Karmikel, C. D. (2009). Health information-seeking and perceptions of website credibility: Examining Web-use orientation, message characteristics, and structural features of websites. Computers in Human Behavior, 25(2), 544-553.
57. Rieh, S. Y., & Danielson, D. R. (2007). Credibility: A multidisciplinary framework. Annual Review of Information Science and Technology, 41(1), 307-364.
58. Rowley, J., & Johnson, F. (2013). Understanding trust formation in digital information sources: The case of Wikipedia. Journal of Information Science, 39(4), 494-508.
59. Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical Therapy, 85(3), 257-268.
60. Simon, H. A. (1980). Cognitive science: The newest science of the artificial. Cognitive Science, 4(1), 33-46.
61. Stanford, J., Tauber, E., Fogg, B., & Marable, L. (2002). Experts vs. online consumers: A comparative credibility study of health and finance web sites. Consumer WebWatch.
62. Stvilia, B., Twidale, M., Smith, L., & Gasser, L. (2008). Information quality work organization in Wikipedia. Journal of the American Society for Information Science and Technology, 59(6), 983-1001.
63. Sundar, S. S. (2007). The MAIN model: A heuristic approach to understanding technology effects on credibility. In M. J. Metzger (Ed.), Digital media, youth, and credibility (pp. 73-100). Cambridge, MA: MIT Press.
64. Taylor, R. S., & Voigt, M. J. (1986). Value added processes in information systems. Westport, CT: Greenwood Publishing Group.
65. Todd, P. M., & Gigerenzer, G. (2012). Ecological rationality: Intelligence in the world. New York: Oxford University Press.
66. Tversky, A., & Kahneman, D. (2000). Choices, values, and frames. Russell Sage Foundation.
67. Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5-33.
68. West, K., & Williamson, J. (2009). Wikipedia: Friend or foe? Reference Services Review, 37(3), 260-271.
69. Wong, S. S. (2008). Task knowledge overlap and knowledge variety: the role of advice network structures and impact on group effectiveness. Journal of Organizational Behavior, 29(5), 591-614.
70. Wong, S. S. (2008). Task knowledge overlap and knowledge variety: the role of advice network structures and impact on group effectiveness. Journal of Organizational Behavior, 29(5), 591-614.
71. Xu, S. X., & Zhang, X. (2013). Impact of Wikipedia on market information environment: Evidence on management disclosure and investor reaction. MIS Quarterly, 37(4), 1043-1068.
72. Yaari, E., Baruchson-Arbib, S., & Bar-Ilan, J. (2011). Information quality assessment of community generated content: A user study of Wikipedia. Journal of Information Science, 37(5), 487-498.