What you see is what you can change: Human-centered machine learning by interactive visualization

Neurocomputing

Volumn 268, Issue , 2017, Pages 164-175

  Sacha, Dominik ^a   Sedlmair, Michael ^b   Zhang, Leishi ^c   Lee, John A ^d   Peltonen, Jaakko ^e   Weiskopf, Daniel ^f   North, Stephen C ^g   Keim, Daniel A ^a  

^a UNIVERSITY OF KONSTANZ   (Germany)

^b UNIVERSITY OF VIENNA   (Austria)

^c MIDDLESEX UNIVERSITY   (United Kingdom)

^d UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

^e AALTO UNIVERSITY   (Finland)

^f UNIVERSITY OF STUTTGART   (Germany)

^g Infovisible   (United States)

Author keywords

Information visualization; Interaction; Machine learning; Visual analytics

Indexed keywords

DATA HANDLING; DATA VISUALIZATION; INFORMATION ANALYSIS; INFORMATION SYSTEMS; LEARNING SYSTEMS; MACHINE LEARNING;

AUTOMATED ALGORITHMS; AUTOMATED DATA ANALYSIS; CONCEPTUAL FRAMEWORKS; INFORMATION VISUALIZATION; INTERACTION; INTERACTIVE VISUALIZATIONS; VISUAL ANALYTICS; VISUALIZATION RESEARCH;

VISUALIZATION;

ALGORITHM; ARTICLE; AUTOMATION; DATA ANALYSIS; DATA MINING; FEEDBACK SYSTEM; HUMAN COMPUTER INTERACTION; MACHINE LEARNING; PRIORITY JOURNAL; PROBLEM SOLVING;

EID: 85019013025     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2017.01.105     Document Type: Article

References (68)

1. Keim, D.A., Kohlhammer, J., Ellis, G.P., Mansmann, F., Mastering the Information Age – Solving Problems with Visual Analytics. 2010, Eurographics Association.
2. Tam, G.L., Kothari, V., Chen, M., An analysis of machine- and human-analytics in classification. IEEE Trans. Visual. Comput. Gr. (TVCG), PP(99), 2016, 1, 10.1109/TVCG.2016.2598829.
3. Endert, A., North, C., Chang, R., Zhou, M., Toward usable interactive analytics: coupling cognition and computation. Proceedings of the ACM SIGKDD Workshop on Interactive Data Exploration and Analytics (IDEA), 2014.
4. Sacha, D., Stoffel, A., Stoffel, F., Kwon, B., Ellis, G., Keim, D., Knowledge generation model for visual analytics. IEEE Trans. Visual. Comput. Gr. (TVCG) 20:12 (2014), 1604–1613, 10.1109/TVCG.2014.2346481.
5. Amershi, S., Cakmak, M., Knox, W., Kulesza, T., Power to the people: the role of humans in interactive machine learning. AI Mag. 35:4 (2014), 105–120.
6. Keim, D.A., Munzner, T., Rossi, F., Verleysen, M., Bridging information visualization with machine learning (Dagstuhl Seminar 15101). Dagstuhl Rep. 5:3 (2015), 1–27, 10.4230/DagRep.5.3.1.
7. Sacha, D., Sedlmair, M., Zhang, L., Lee, J.A., Weiskopf, D., North, S.C., Keim, D.A., Human-centered machine learning through interactive visualization: review and open challenges. Proceedings of the Twenty-Fourth European Symposium on Artificial Neural Networks, ESANN, 2016, 641–646.
8. Card, S.K., Mackinlay, J.D., Shneiderman, B., Readings in Information Visualization – Using Vision to Think. 1999, Academic Press.
9. Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., From data mining to knowledge discovery in databases. AI Mag. 17:3 (1996), 37–54.
10. Pirolli, P., Card, S., The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. Proceedings of the International Conference on Intelligence Analysis, 5, 2005, 2–4.
11. Pike, W.A., Stasko, J.T., Chang, R., O'Connell, T.A., The science of interaction. Inf. Visual. 8:4 (2009), 263–274, 10.1057/ivs.2009.22.
12. Endert, A., Hossain, M.S., Ramakrishnan, N., North, C., Fiaux, P., Andrews, C., The human is the loop: new directions for visual analytics. J. Intell. Inf. Syst. 43:3 (2014), 411–435.
13. Chen, M., Golan, A., What may visualization processes optimize?. IEEE Trans. Visual. Comput. Gr. (TVCG) 22:12 (2016), 2619–2632, 10.1109/TVCG.2015.2513410.
14. Brehmer, M., Munzner, T., A multi-level typology of abstract visualization tasks. IEEE Trans. Visual. Comput. Gr. (TVCG) 19:12 (2013), 2376–2385, 10.1109/TVCG.2013.124.
15. von Landesberger, T., Fiebig, S., Bremm, S., Kuijper, A., Fellner, D.W., Interaction taxonomy for tracking of user actions in visual analytics applications. Handbook of Human Centric Visualization, 2014, Springer, 653–670.
16. Bertini, E., Lalanne, D., Surveying the complementary role of automatic data analysis and visualization in knowledge discovery. Proceedings of the ACM SIGKDD Workshop on Visual Analytics and Knowledge Discovery: Integrating Automated Analysis with Interactive Exploration, 2009, 12–20, 10.1145/1562849.1562851.
17. Muhlbacher, T., Piringer, H., Gratzl, S., Sedlmair, M., Streit, M., Opening the black box: strategies for increased user involvement in existing algorithm implementations. IEEE Trans. Visual. Comput. Gr. (TVCG) 20:12 (2014), 1643–1652, 10.1109/TVCG.2014.2346578.
18. Jawaheer, G., Weller, P., Kostkova, P., Modeling user preferences in recommender systems: a classification framework for explicit and implicit user feedback. ACM Trans. Interact. Intell. Syst. 4:2 (2014), 8:1–8:26, 10.1145/2512208.
19. Melville, P., Sindhwani, V., Recommender systems. Sammut, C., Webb, G.I., (eds.) Encyclopedia of Machine Learning, 2010, Springer, 829–838, 10.1007/978-0-387-30164-8.
20. Ricci, F., Rokach, L., Shapira, B., Kantor, P.B., Recommender Systems Handbook. first, 2010, Springer-Verlag.
21. Cohn, D., Atlas, L., Ladner, R., Improving generalization with active learning. Mach. Learn. 15:2 (1994), 201–221, 10.1023/A:1022673506211.
22. Settles, B., Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 2012, Morgan & Claypool Publishers, 10.2200/S00429ED1V01Y201207AIM018.
23. Groce, A., Kulesza, T., Zhang, C., Shamasunder, S., Burnett, M.M., Wong, W., Stumpf, S., Das, S., Shinsel, A., Bice, F., McIntosh, K., You are the only possible oracle: effective test selection for end users of interactive machine learning systems. IEEE Trans. Softw. Eng. 40:3 (2014), 307–323, 10.1109/TSE.2013.59.
24. Norman, D., The Design of Everyday Things. 2002, Basic Books.
25. Shneiderman, B., Direct manipulation: a step beyond programming languages. IEEE Comput. 16:8 (1983), 57–69, 10.1109/MC.1983.1654471.
26. Kandel, S., Heer, J., Plaisant, C., Kennedy, J., van Ham, F., Riche, N., Weaver, C., Lee, B., Brodbeck, D., Buono, P., Research directions in data wrangling: visualizations and transformations for usable and credible data. Inf. Visual. 10:4 (2011), 271–288, 10.1177/1473871611415994.
27. Guyon, I., Elisseeff, A., An introduction to variable and feature selection. J. Mach. Learn. Res. 3 (2003), 1157–1182.
28. Kaden, M., Lange, M., Nebel, D., Riedel, M., Geweniger, T., Villmann, T., Aspects in classification learning – review of recent developments in learning vector quantization. Found. Comput. Decis. Sci. 39:2 (2014), 79–105.
29. Schneider, P., Hammer, B., Biehl, M., Adaptive relevance matrices in learning vector quantization. Neural Comput. 21 (2009), 3532–3561.
30. Bunte, K., Schneider, P., Hammer, B., Schleif, F.-M., Villmann, T., Biehl, M., Limited rank matrix learning, discriminative dimension reduction and visualization. Neural Netw. 26:1 (2012), 159–173.
31. Rakotomamonjy, A., Bach, F.R., Canu, S., Grandvalet, Y., Simplemkl. J. Mach. Learn. Res. 9:Nov (2008), 2491–2521.
32. Efron, B., Tibshirani, R., Improvements on cross-validation: the .632+ bootstrap method. J. Am. Stat. Assoc. 92:438 (1997), 548–560, 10.2307/2965703.
33. Fröhlich, H., Chapelle, O., Schölkopf, B., Feature selection for support vector machines by means of genetic algorithm. Proceedings of the Fifteenth IEEE International Conference on Tools with Artificial Intelligence, 2003, 142–148.
34. Jeong, D.H., Ziemkiewicz, C., Fisher, B.D., Ribarsky, W., Chang, R., iPCA: an interactive system for PCA-based visual analytics. Comput. Gr. Forum 28:3 (2009), 767–774, 10.1111/j.1467-8659.2009.01475.x.
35. Mühlbacher, T., Piringer, H., A partition-based framework for building and validating regression models. IEEE Trans. Visual. Comput. Gr. (TVCG) 19:12 (2013), 1962–1971, 10.1109/TVCG.2013.125.
36. Endert, A., Fiaux, P., North, C., Semantic interaction for visual text analytics. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2012, 473–482.
37. van den Elzen, S., van Wijk, J., Baobabview: interactive construction and analysis of decision trees. Proceedings of the IEEE Conference on Visual Analytics in Science and Technology (VAST), 2011, 151–160, 10.1109/VAST.2011.6102453.
38. Aupetit, M., Sedlmair, M., SepMe: 2002 new visual separation measures. Proceedings of the IEEE Pacific Visualization Symposium, 2016, 1–8.
39. Crouser, R.J., Chang, R., An affordance-based framework for human computation and human–computer collaboration. IEEE Trans. Visual. Comput. Gr. (TVCG) 18:12 (2012), 2859–2868, 10.1109/TVCG.2012.195.
40. Boy, J., Eveillard, L., Détienne, F., Fekete, J., Suggested interactivity: seeking perceived affordances for information visualization. IEEE Trans. Visual. Comput. Gr. (TVCG) 22:1 (2016), 639–648, 10.1109/TVCG.2015.2467201.
41. Xu, K., Attfield, S., Jankun-Kelly, T.J., Wheat, A., Nguyen, P.H., Selvaraj, N., Analytic provenance for sensemaking: a research agenda. IEEE Comput. Gr. Appl. 35:3 (2015), 56–64, 10.1109/MCG.2015.50.
42. Kapoor, A., Lee, B., Tan, D.S., Horvitz, E., Learning to learn: algorithmic inspirations from human problem solving. Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.
43. Höferlin, B., Netzel, R., Höferlin, M., Weiskopf, D., Heidemann, G., Inter-active learning of ad-hoc classifiers for video visual analytics. Proceedings of the IEEE Conference on Visual Analytics in Science and Technology (VAST), 2012, 23–32, 10.1109/VAST.2012.6400492.
44. Kandel, S., Paepcke, A., Hellerstein, J.M., Heer, J., Wrangler: interactive visual specification of data transformation scripts. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2011, 3363–3372, 10.1145/1978942.1979444.
45. Brown, E., Liu, J., Brodley, C., Chang, R., Dis-function: learning distance functions interactively. Proceedings of the IEEE Conference on Visual Analytics in Science and Technology (VAST), 2012, 83–92, 10.1109/VAST.2012.6400486.
46. Talbot, J., Lee, B., Kapoor, A., Tan, D., EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2009, 1283–1292, 10.1145/1518701.1518895.
47. Wongsuphasawat, K., Moritz, D., Anand, A., Mackinlay, J.D., Howe, B., Heer, J., Voyager: exploratory analysis via faceted browsing of visualization recommendations. IEEE Trans. Visual. Comput. Gr. (TVCG) 22:1 (2016), 649–658, 10.1109/TVCG.2015.2467191.
48. Ingram, S., Munzner, T., Irvine, V., Tory, M., Bergner, S., Möller, T., DimStiller: workflows for dimensional analysis and reduction. Proceedings of the IEEE Conference on Visual Analytics in Science and Technology (VAST), 2010, 3–10, 10.1109/VAST.2010.5652392.
49. Simon, S., Mittelstdt, S., Keim, D.A., Sedlmair, M., Bridging the gap of domain and visualization experts with a liaison. Bertini, E., Kennedy, J., Puppo, E., (eds.) Proceedings of the Eurographics Conference on Visualization (EuroVis) – Short Papers, 2015, The Eurographics Association, 10.2312/eurovisshort.20151137.
50. Sedlmair, M., Meyer, M., Munzner, T., Design study methodology: reflections from the trenches and the stacks. IEEE Trans. Visual. Comput. Gr. (TVCG) 18:12 (2012), 2431–2440, 10.1109/TVCG.2012.213.
51. Vellido, A., Martín-Guerrero, J.D., Lisboa, P.J.G., Making machine learning models interpretable. Proceedings of the Twentieth European Symposium on Artificial Neural Networks, ESANN 2012, 2012.
52. Amershi, S., Fogarty, J., Weld, D., ReGroup: interactive machine learning for on-demand group creation in social networks. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2012, ACM, 21–30, 10.1145/2207676.2207680.
53. Behrisch, M., Korkmaz, F., Shao, L., Schreck, T., Feedback-driven interactive exploration of large multidimensional data supported by visual classifier. Proceedings of the IEEE Conference on Visual Analytics in Science and Technology (VAST), 2014, 43–52, 10.1109/VAST.2014.7042480.
54. Liu, Z., Stasko, J., Mental models, visual reasoning and interaction in information visualization: a top-down perspective. IEEE Trans. Visual. Comput. Gr. (TVCG) 16:6 (2010), 999–1008, 10.1109/TVCG.2010.177.
55. Venna, J., Peltonen, J., Nybo, K., Aidos, H., Kaski, S., Information retrieval perspective to nonlinear dimensionality reduction for data visualization. J. Mach. Learn. Res. 11 (2010), 451–490.
56. Lee, J.A., Renard, E., Bernard, G., Dupont, P., Verleysen, M., Type 1 and 2 mixtures of Kullback–Leibler divergences as cost functions in dimensionality reduction based on similarity preservation. Neurocomputing 112 (2013), 92–108, 10.1016/j.neucom.2012.12.036.
57. Bertini, E., Tatu, A., Keim, D.A., Quality metrics in high-dimensional data visualization: an overview and systematization. IEEE Trans. Visual. Comput. Gr. (TVCG) 17:12 (2011), 2203–2212.
58. Sips, M., Neubert, B., Lewis, J.P., Hanrahan, P., Selecting good views of high-dimensional data using class consistency. Comput. Gr. Forum 28:3 (2009), 831–838.
59. Zuk, T., Carpendale, S., Visualization of uncertainty and reasoning. Proceedings of the Eighth International Symposium on Smart Graphics, 2007, Springer-Verlag, Berlin, Heidelberg, 164–177, 10.1007/978-3-540-73214-3_15.
60. Sacha, D., Senaratne, H., Kwon, B.C., Ellis, G.P., Keim, D.A., The role of uncertainty, awareness, and trust in visual analytics. IEEE Trans. Visual. Comput. Gr. (TVCG) 22:1 (2016), 240–249, 10.1109/TVCG.2015.2467591.
61. Correa, C.D., Chan, Y., Ma, K., A framework for uncertainty-aware visual analytics. Proceedings of the IEEE Conference on Visual Analytics in Science and Technology (VAST), 2009, 51–58, 10.1109/VAST.2009.5332611.
62. Pang, A., Wittenbrink, C.M., Lodha, S.K., Approaches to uncertainty visualization. Vis. Comput. 13:8 (1997), 370–390.
63. Brodlie, K., Osorio, R.A., Lopes, A., A review of uncertainty in data visualization. Dill, J., Earnshaw, R., Kasik, D., Vince, J., Wong, P.C., (eds.) Expanding the Frontiers of Visual Analytics and Visualization, 2012, Springer, 81–109.
64. Skeels, M., Lee, B., Smith, G., Robertson, G.G., Revealing uncertainty for information visualization. Inf. Visual. 9:1 (2010), 70–81.
65. Pezzotti, N., Lelieveldt, B., van der Maaten, L., Hollt, T., Eisemann, E., Vilanova, A., Approximated and user steerable tSNE for progressive visual analytics. IEEE Trans. Visual. Comput. Gr. (TVCG), PP(99), 2016, 1, 10.1109/TVCG.2016.2570755.
66. Turkay, C., Kaya, E., Balcisoy, S., Hauser, H., Designing progressive and interactive analytics processes for high-dimensional data analysis. IEEE Trans. Visual. Comput. Gr. (TVCG) 23:1 (2017), 131–140, 10.1109/TVCG.2016.2598470.
67. Popper, K., The Logic of Scientific Discovery. 2005, Routledge.
68. Dix, A., Theoretical analysis and theory creation. Res. Methods Human–Comput. Interact., 2008, 175–195.