Uncertainty Representation in Visualizations of Learning Analytics for Learners: Current Approaches and Opportunities

IEEE Transactions on Learning Technologies

Volumn 8, Issue 3, 2015, Pages 242-260

  Demmans Epp, Carrie ^a   Bull, Susan ^b  

^a UNIVERSITY OF TORONTO   (Canada)

^b UNIVERSITY OF BIRMINGHAM   (United Kingdom)

Author keywords

educational reporting; learning analytics; learning dashboards; open learner models; uncertainty; visual analytics

Indexed keywords

DECISION MAKING; EDUCATION; LEARNING SYSTEMS; VISUALIZATION;

EDUCATIONAL REPORTING; LEARNING ANALYTICS; LEARNING DASHBOARDS; OPEN LEARNER MODELS; UNCERTAINTY; VISUAL ANALYTICS;

UNCERTAINTY ANALYSIS;

EID: 84961794709     PISSN: 19391382     EISSN: None     Source Type: Journal    
DOI: 10.1109/TLT.2015.2411604     Document Type: Article

References (94)

1. K. Brodlie, R. Allendes Osorio, and A. Lopes, "A review of uncertainty in data visualization," in Expanding the Frontiers Visual Analytics and Visualization, J. Dill, R. Earnshaw, D. Kasik, J. Vince, and P. C. Wong, Eds. London, U.K.: Springer, 2012, pp. 81-109.
2. R. Zwick, D. Zapata-Rivera, and M. Hegarty, "Comparing graphical and verbal representations of measurement error in test score reports," Educ. Assess., vol. 19, no. 2, pp. 116-138, Apr. 2014.
3. G. E. Newman and B. J. Scholl, "Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias," Psychon. Bull. Rev., vol. 19, no. 4, pp. 601-607, May 2012.
4. P. F. Fisher, "Visualizing uncertainty in soil maps by animation," Cartogr. Int. J. Geogr. Inf. Geovisualization, vol. 30, no. 2, pp. 20-27, Oct. 1993.
5. S. Belia, F. Fidler, J. Williams, and G. Cumming, "Researchers misunderstand confidence intervals and standard error bars," Psychol. Methods, vol. 10, no. 4, pp. 389-396, 2005.
6. M. Skeels, B. Lee, G. Smith, and G. Robertson, "Revealing uncertainty for information visualization," in Proc. Working Conf. Adv. Vis. Interfaces, New York, NY, USA, 2008, pp. 376-379.
7. H. Wainer, "Depicting error," Am. Stat., vol. 50, no. 2, p. 101, May 1996.
8. S. Basapur, A. M. Bisantz, and T. Kesavadas, "The effect of display modality on decision-making with uncertainty," Proc. Hum. Factors Ergon. Soc. Annu. Meet., vol. 47, no. 3, pp. 558-561, Oct. 2003.
9. A. M. Bisantz, D. Cao, M. Jenkins, P. R. Pennathur, M. Farry, E. Roth, S. S. Potter, and J. Pfautz, "Comparing uncertainty visualizations for a dynamic decision-making task," J. Cogn. Eng. Decis. Mak., vol. 5, no. 3, pp. 277-293, Aug. 2011.
10. D. Spiegelhalter, M. Pearson, and I. Short, "Visualizing uncertainty about the future," Science, vol. 333, no. 6048, pp. 1393-1400, Sep. 2011.
11. J. Wood, P. Isenberg, T. Isenberg, J. Dykes, N. Boukhelifa, and A. Slingsby, "Sketchy rendering for information visualization," IEEE Trans. Vis. Comput. Graph., vol. 18, no. 12, pp. 2749-2758, Dec. 2012.
12. C. Demmans Epp, S. Bull, and M. D. Johnson, "Visualising uncertainty for open learner model users," in Proc. CEUR. User Model., Adaptation Personalization, Aalborg, Denmark, 2014, vol. 1181, pp. 9-12.
13. J. D. Zapata-Rivera, J. S. Underwood, and M. Bauer, "Advanced reporting systems in assessment environments," in Proc. Workshop Learner Model. Reflection, Int. Conf. Artif. Intell. Educ., Amsterdam, Netherlands, 2005, pp. 23-31.
14. S. Bull and J. Kay, "Chapter 15: Open learner models," in Advances in Intelligent Tutoring Systems, R. Nkambou, J. Bourdeau, and R. Mizoguchi, Eds. New York, NY, USA: Springer, 2010, pp. 301-322.
15. E. Duval, J. Klerkx, K. Verbert, T. Nagel, S. Govaerts, G. A. Parra Chico, J. L. Santos Odriozola, and B. Vandeputte, "Learning dashboards & learnscapes," in Proc. Workshop Educ. Interfaces, Softw., Technol. ACM SIGCHI Conf. Human Factors Comput. Syst., Austin, TX, USA, 2012, pp. 1-5.
16. A. F. Wise, "Designing pedagogical interventions to support student use of learning analytics," in Proc. 4th Int. Conf. Learn. Analytics Knowl., New York, NY, USA, 2014, pp. 203-211.
17. S. Bull and A. T. McEvoy, "An Intelligent learning environment with an open learner model for the desktop PC and pocket PC," in Proc. Int. Conf. Artif. Intell. Educ., 2003, pp. 389-391.
18. E. Duval, "Attention Please!: Learning analytics for visualization and recommendation," in Proc. 1st Int. Conf. Learn. Analytics Knowl., New York, NY, USA, 2011, pp. 9-17.
19. D. Edge, E. Searle, K. Chiu, J. Zhao, and J. A. Landay, "MicroMandarin: Mobile language learning in context," in Proc. Conf. Human Factors Comput. Syst., Vancouver, BC, Canada, 2011, pp. 3169-3178.
20. C. Hansen and G. McCalla, "Active open learner modelling," in Proc. Workshop Learner Model. Reflection (Supplemental Proc.) 11th Int. Conf. Artif. Intell. Educ., 2003, vol. 5, pp. 248-257.
21. J. Hewitt, C. Brett, and K. MacKinnon, "A study of private messaging within an asynchronous discussion environment," in Proc. To See World and a Grain Sand: Learn. Across Levels Space, Time Scale: CSCL Conf., Sydney, Australia, 2013, vol. 2, pp. 46-49.
22. K. Upton and J. Kay, "Narcissus: group and individual models to support small group work," in Proc. 17th Int. Conf. User Model., Adaptation, Personalization, 2009, pp. 54-65.
23. J. Vassileva and L. Sun, "Evolving a social visualization design aimed at increasing participation in a class-based online community," Int. J. Coop. Inf. Syst., vol. 17, no. 4, pp. 443-466, Dec. 2008.
24. S. Bull and J. Kay, "Student models that invite the learner in: The SMILI:() Open learner modelling framework," Int. J. Artif. Intell. Educ., vol. 17, no. 2, pp. 89-120, 2007.
25. L. Bastin, P. F. Fisher, and J. Wood, "Visualizing uncertainty in multi-spectral remotely sensed imagery," Comput. Geosci., vol. 28, no. 3, pp. 337-350, Apr. 2002.
26. J. Bertin, Semiology of graphics. Madison, WI, USA: Univ. Wisconsin Press, 1983.
27. C. Ware, Information Visualization: Perception for Design. San Francisco, CA, USA: Morgan Kaufman, 2004.
28. C. Chen, "Information visualization," Wiley Interdiscip. Rev. Comput. Stat., vol. 2, no. 4, pp. 387-403, Jul. 2010.
29. D. J. Greenberg and S. Z. Blue, "Visual complexity in infancy: Contour or numerosity?" Child Dev., vol. 46, no. 2, p. 357, Jun. 1975.
30. E. R. Tufte, The Visual Display of Quantitative Information. Cheshire, CT, USA: Graphics Press, 1999.
31. S. Bateman, R. L. Mandryk, C. Gutwin, A. Genest, D. McDine, and C. Brooks, "Useful Junk?: The effects of visual embellishment on comprehension and memorability of charts," in Proc. SIGCHI Conf. Human Factors in Computing Systems, New York, NY, USA, 2010, pp. 2573-2582.
32. C. M. Wittenbrink, A. T. Pang, and S. K. Lodha, "Glyphs for visualizing uncertainty in vector fields," IEEE Trans. Vis. Comput. Graph., vol. 2, no. 3, pp. 266-279, Sep. 1996.
33. T. D. Zuk, "Visualizing uncertainty," PhD, Dept. Comput. Sci., Univ. Calgary, 2008.
34. T. Gebuis and B. Reynvoet, "The role of visual information in numerosity estimation," PLoS ONE, vol. 7, no. 5, p. e37426, May 2012.
35. A. M. Bisantz, S. S. Marsiglio, and J. Munch, "Displaying uncertainty: Investigating the effects of display format and specificity," Hum. Factors, vol. 47, no. 4, pp. 777-796, 2005.
36. N. Boukhelifa, A. Bezerianos, T. Isenberg, and J.-D. Fekete, "Evaluating sketchiness as a visual variable for the depiction of qualitative uncertainty," IEEE Trans. Vis. Comput. Graph., vol. 18, no. 12, pp. 2769-2778, Dec. 2012.
37. C. Collins, S. Carpendale, and G. Penn, "Visualization of uncertainty in lattices to support decision-making," in Proc. 9th Joint Eurograph./ IEEE VGTC Conf. Vis., Aire-la-Ville, Switzerland, Switzerland, 2007, pp. 51-58.
38. R. A. Becker and W. S. Cleveland, "Brushing scatterplots," Technometrics, vol. 29, no. 2, p. 127, May 1987.
39. S. Bateman, C. Gutwin, and M. Nacenta, "Seeing things in the clouds: The effect of visual features on tag cloud selections," in Proc. 19th ACM Conf. Hypertext Hypermedia, New York, NY, USA, 2008, pp. 193-202.
40. G. Siemens. (2011, Aug. 5). Learning and academic analytics, Learning and Knowledge Analytics [Online]. Available: http:// www.learninganalytics.net/?p=131.
41. V. Kodaganallur, R. R. Weitz, and D. Rosenthal, "A comparison of model-tracing and constraint-based intelligent tutoring paradigms," Int. J. Artif. Intell. Educ., vol. 15, no. 2, pp. 117-144, 2005.
42. G. McCalla, "The ecological approach to the design of e-learning environments: Purpose-based capture and use of information about learners," J. Interact. Media Educ., vol. 2004, no. 7, pp. 1-23, 2004.
43. S. Ohlsson, "Constraint-based student modelling," in Student Modelling: The Key to Individualized Knowledge-Based Instruction, J. Greer and G. McCalla, Eds. New York, NY, USA: Springer-Verlag, 1993, pp. 165-189.
44. J. A. Self, "Bypassing the intractable problem of student modelling," Artif. Intell., vol. 41, pp. 1-26, 1990.
45. K. VanLehn, R. Freedman, P. Jordan, C. Murray, R. Osan, M. Ringenberg, C. Rosé, K. Schulze, R. Shelby, D. Treacy, A. Weinstein, and M. Wintersgill, "Fading and deepening: The next steps for andes and other model-tracing tutors," in Proc. 5th Int. Conf. Intell. Tutoring Syst., 2000, pp. 474-483.
46. G. Weber, "Episodic learner modeling," Cogn. Sci., vol. 20, no. 2, pp. 195-236, 1996.
47. K. Chrysafiadi and M. Virvou, "Student modeling approaches: A literature review for the last decade," Expert Syst. Appl., vol. 40, no. 11, pp. 4715-4729, Sep. 2013.
48. C. Demmans Epp and G. I. McCalla, "ProTutor: Historic open learner models for pronunciation tutoring," in Proc. 15th Int. Conf. Artif. Intel. Educ., Auckland, New Zealand, 2011, vol. 6738, pp. 441-443.
49. W. L. Johnson, S. Marsella, and H. Vilhjálmsson, "The DARWARS Tactical language training system," presented at the Interservice/ Industry Training, Simulation and Education Conf., Orlando, FL, USA, 2004.
50. A. Mitrovic. (2007). SQL-Tutor: an ITS for SQL programming, CAPIT-University of Canterbury [Online]. Available: http://www. cosc.canterbury.ac.nz/tanja.mitrovic/sql-tutor.html.
51. R. S. J. D. Baker and K. Yacef, "The state of educational data mining in 2009: A review and future visions," J. Educ. Data Min., vol. 1, no. 1, pp. 3-17, 2009.
52. D. Barrow, A. Mitrovic, S. Ohlsson, and M. Grimley, "Assessing the impact of positive feedback in constraint-based tutors," in Intelligent Tutoring Systems. Berlin, Germany: Springer, 2008, pp. 250-259.
53. M. Mathews and A. Mitrovic, "Investigating the effectiveness of problem templates on learning in intelligent tutoring systems," B. Sc. Honors Rep. Univ. Canterb. Dept. Comput. Sci., Softw. Eng. Christchchurch, NewZealand, 2006.
54. A. Mitrovic, K. R. Koedinger, and B. Martin, "A comparative analysis of cognitive tutoring and constraint-based modelling," in Proc. 9th Int. Conf. User Model., Johnstown, PA, USA, 2003, pp. 313-322.
55. K. VanLehn, "The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems," Educ. Psychol., vol. 46, no. 4, pp. 197-221, 2011.
56. A. Corbett, A. Wagner, S. Lesgold, H. Ulrich, and S. Stevens, "The impact on learning of generating vs. selecting descriptions in analyzing algebra example solutions," in Proc. 7th Int. Conf. Learn. Sci., 2006, pp. 99-105.
57. S. Govaerts, K. Verbert, J. Klerkx, and E. Duval, "Visualizing activities for self-reflection and awareness," in Proc. Int. Conf. Adv. Web-Based Learn., Berlin, Germany, 2010, vol. 6483, pp. 91-100.
58. A. Paiva, J. A. Self, and R. Hartley, "Externalising learner models," in Proc. World Conf. Artif. Intell. Educ., Washington, D.C., USA, 1995, pp. 509-516.
59. J. Kay, "A scrutable user modelling shell for user-adapted interaction," Ph.D. Basser, Dept. Comput. Sci., Univ. Sydney, N.S.W., Australia, 1999.
60. A. Mitrovic and B. Martin, "Evaluating the effect of open student models on self-assessment," Int. J. Artif. Intell. Educ. IJAIED, vol. 17, no. 2, pp. 121-144, Jan. 2007.
61. Blackboard. Washington, D.C., Blackboard Inc, 1997.
62. Moodle. Perth, Australia: Moodle Trust, 2002.
63. N. Ahmad and S. Bull, "Do students trust their open learner models?" in Proc. 5th Int. Conf. Adaptive Hypermedia Adaptive Web-Based Syst., 2008, pp. 255-258.
64. J. B. Uther, "On the visualisation of large user models in web based systems," Ph.D. dissertation, Dept. Comput. Sci., Univ. Sydney, N.S.W., Australia, 2001.
65. J. D. Zapata-Rivera and J. Greer, "Interacting with inspectable Bayesian student models," Int. J. Artif. Intell. Educ., vol. 14, no. 2, pp. 127-163, 2004.
66. A. Paramythis, S. Weibelzahl, and J. Masthoff, "Layered evaluation of interactive adaptive systems: Framework and formative methods," User Model. User-Adapt. Interact., vol. 20, no. 5, pp. 383-453, Nov. 2010.
67. K. Potter, P. Rosen, and C. R. Johnson, "From quantification to visualization: A taxonomy of uncertainty visualization approaches," in Uncertainty Quantification in Scientific Computing, A. M. Dienstfrey and R. F. Boisvert, Eds. Berlin, Germany: Springer, 2012, pp. 226-249.
68. C. Correa, Y.-H. Chan, and K.-L. Ma, "A framework for uncertainty-aware visual analytics," in Proc. IEEE Symp. Vis. Analytics Sci. Technol., 2009, pp. 51-58.
69. A. M. MacEachren, R. E. Roth, J. O'Brien, B. Li, D. Swingley, and M. Gahegan, "Visual semiotics & uncertainty visualization: An empirical study," IEEE Trans. Vis. Comput. Graph., vol. 18, no. 12, pp. 2496-2505, Dec. 2012.
70. K. Silius, T. Miilumaki, J. Huhtamaki, T. Tebest, J. Merilainen, and S. Pohjolainen, "Students' motivations for social media enhanced studying and learning," Knowl. Manag. E-Learn. Int. J. KMEL, vol. 2, no. 1, pp. 51-67, Feb. 2010.
71. J. Thomson, E. Hetzler, A. MacEachren, M. Gahegan, and M. Pavel, "A typology for visualizing uncertainty," Proc. SPIE, vol. 146, pp. 146-157, 2005.
72. N. Gershon, "Visualization of an imperfect world," IEEE Comput. Graph. Appl., vol. 18, no. 4, pp. 43-45, Jul. 1998.
73. S. K. Lodha, A. Pang, R. E. Sheehan, and C. M. Wittenbrink, "UFLOW: Visualizing uncertainty in fluid flow," in Proc. Vis., 1996, pp. 249-254.
74. S. Bull, M. D. Johnson, C. Demmans Epp, D. Masci, M. Alotaibi, and S. Girard, "Formative assessment and meaningful learning analytics-An independent open learner model solution," in Proc. IEEE Int. Conf. Adv. Learn. Technol., Athens, Greece, 2014, pp. 327-329.
75. S. Deitrick and R. Edsall, "The influence of uncertainty visualization on decision making: An empirical evaluation," in Progress in Spatial Data Handling, D. A. Riedl, P. W. Kainz, and P. G. A. Elmes, Eds. Berlin, Germany: Springer 2006, pp. 719-738.
76. C. R. Johnson and A. R. Sanderson, "A next step: Visualizing errors and uncertainty," IEEE Comput. Graph. Appl., vol. 23, no. 5, pp. 6-10, Sep./Oct. 2003.
77. C. Munteanu, J. Lumsden, H. Fournier, R. Leung, D. D'Amours, D. McDonald, and J. Maitland, "ALEX: mobile language assistant for low-literacy adults," in Proc. Int. Conf. Human-Comput. Interaction Mobile Dev. Services, Lisbon, Portugal, 2010, pp. 427-430.
78. I.-H. Hsiao, F. Bakalov, P. Brusilovsky, and B. König-Ries, "Open social student modeling: Visualizing student models with parallel introspective views," in Proc. 19th Int. Conf. User Model., Adaption Personalization, 2011, pp. 171-182.
79. Z.-H. Chen, C.-Y. Chou, Y.-C. Deng, and T.-W. Chan, "Active open learner models as animal companions: Motivating children to learn through interacting with my-pet and our-pet," Int. J. Artif. Intell. Educ., vol. 17, no. 2, pp. 145-167, Apr. 2007.
80. S. Bull, M. Mangat, A. Mabbott, A. S. Abu Issa, and J. Marsh, "Reactions to inspectable learner models: Seven year olds to university students," in Proc. Workshop Learner Model. Reflection, Int. Conf. Artif. Intel. Educ., Amsterdam, Netherlands, 2005, pp. 1-10.
81. S. J. H. Lee and S. Bull, "An open learner model to help parents help their children," Technol. Instr. Cogn. Learn., vol. 6, no. 1, pp. 29-51, 2008.
82. T. Lloyd and S. Bull, "A haptic learner model," Int. J. Contin. Eng. Educ. Life Long Learn., vol. 16, no. 1, pp. 137-149, Jan. 2006.
83. S. Bull, M. D. Johnson, M. Alotaibi, W. Byrne, and G. Cierniak, "Visualising multiple data sources in an independent open learner model," in Proc. 16th Intell. Conf. Artif. Intel. Educ., 2013, pp. 199-208.
84. N. Ahmad and S. Bull, "Learner trust in learner model externalisations," Frontiers Artif. Intell. Educ., vol. 200, pp. 617-619, 2009.
85. N. Van Labeke, P. Brna, and R. Morales, "Opening up the interpretation process in an open learner model," Int. J. Artif. Intell. Educ., vol. 17, no. 3, pp. 305-338, 2007.
86. J. Kay and B. Kummerfeld, "Creating personalized systems that people can scrutinize and control: drivers, principles and experience," ACM Trans. Interact. Intell. Syst., vol. 2, no. 4, pp. 24:1-24:42, Jan. 2013.
87. S. Bull and H. Pain, "Did I say what I think I said, and do you agree with me?': Inspecting and questioning the student model," in Proc. World Conf. Artif. Intell. Educ., Washington, DC, USA, 1995, pp. 501-508.
88. S. Mohanarajah, R. Kemp, and E. Kemp, "Opening a fuzzy learner model," in Proc. Workshop Learner Model. Reflection Int. Conf. Artif. Intel. Educ., 2005, pp. 62-71.
89. J. Kay and A. Lum, "Exploiting readily available web data for scrutable student models," in Proc. Artif. Intell. Educ., Amsterdam, Netherlands, 2005, pp. 338-345.
90. J. Burston, "BonPatron: An online spelling, grammar, and expression checker," Comput. Assist. Lang. Instr. Consort. CALICO J., vol. 25, no. 2, pp. 337-347, 2008.
91. A. Mabbott and S. Bull, "Student preferences for editing, persuading, and negotiating the open learner model," in Intelligent Tutoring Systems. Heidelberg, Germany: Springer, 2006, pp. 481-490.
92. S. Bull and M. Britland, "Group interaction prompted by a simple assessed open learner model that can be optionally released," in Proc. PING Workshop User Model., 2007, pp. 24-32.
93. M. Correll and M. Gleicher, "Error bars considered harmful: Exploring alternate encodings for mean and error," IEEE Trans. Vis. Comput. Graph., vol. 20, no. 12, pp. 2142-2151, Dec. 2014.
94. G. Cumming, F. Fidler, and D. L. Vaux, "Error bars in experimental biology," J. Cell Biol., vol. 177, no. 1, pp. 7-11, Apr. 2007.