Comparing Visual-Interactive Labeling with Active Learning: An Experimental Study

IEEE Transactions on Visualization and Computer Graphics

Volumn 24, Issue 1, 2018, Pages 298-308

  Bernard, Jürgen ^a   Hutter, Marco ^a   Zeppelzauer, Matthias ^b   Fellner, Dieter ^c   Sedlmair, Michael ^d  

^a DARMSTADT UNIVERSITY OF TECHNOLOGY   (Germany)

^b ST PÖLTEN UNIVERSITY OF APPLIED SCIENCES   (Austria)

^c FRAUNHOFER IGD   (Germany)

^d UNIVERSITY OF VIENNA   (Austria)

Author keywords

Active Learning; Classification; Dimensionality Reduction; Evaluation; Experiment; Information Visualization; Labeling; Machine Learning; Visual Analytics; Visual Interactive Labeling

Indexed keywords

ARTIFICIAL INTELLIGENCE; DATA STRUCTURES; DATA VISUALIZATION; ENCODING (SYMBOLS); EXPERIMENTS; INFORMATION SYSTEMS; LABELING; LEARNING SYSTEMS; VISUALIZATION;

ACTIVE LEARNING; DIMENSIONALITY REDUCTION; EVALUATION; INFORMATION VISUALIZATION; INTERACTIVE LABELING; UNCERTAINTY; VISUAL ANALYTICS;

CLASSIFICATION (OF INFORMATION);

EID: 85028696903     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2017.2744818     Document Type: Article

References (72)

1. American Psychological Association. Publication manual of the American psychological association (6th edition). American Psychological Association Washington, 2010.
2. M. Behrisch, F. Korkmaz, L. Shao, and T. Schreck. Feedback-driven interactive exploration of large multidimensional data supported by visual classifier. In IEEE Visual Analytics Science and Technology (VAST), pp. 43-52, 2014. doi: 10.1109/VAST.2014.7042480
3. J. Bernard, C. Ritter, D. Sessler, M. Zeppelzauer, J. Kohlhammer, and D. Fellner. Visual-interactive similarity search for complex objects by example of soccer player analysis. In Proc. ofIVAPP, VISIGRAPP, pp. 75-87, 2017. doi: 10.5220/0006116400750087
4. J. Bernard, D. Sessler, A. Bannach, T. May, and J. Kohlhammer. A visual active learning system for the assessment of patient well-being in prostate cancer research. In IEEE VIS Workshop on Visual Analytics in Healthcare (VAHC), pp. 1-8. ACM, 2015. doi: 10.1145/2836034.2836035
5. J. Bernard, D. Sessler, T. Ruppert, J. Davey, A. Kuijper, and J. Kohlhammer. User-based visual-interactive similarity definition for mixed data objects-concept and first implementation. In WSCG, vol. 22. Eurographics, 2014.
6. J. Bernard, M. Steiger, S. Mittelstadt, S. Thum, D. Keim, and J. Kohlhammer. A survey and task-based quality assessment of static 2d colormaps. In Electronic Imaging, SPIE Conference on Visualization and Data Analysis, vol. 9397, pp. 93970M-93970M-16. SPIE Press, 2015. doi: 10.1117/12. 2079841
7. J. Bernard, M. Zeppelzauer, M. Sedlmair, and W. Aigner. A Unified Process for Visual-Interactive Labeling. In M. Sedlmair and C. Tominski, eds., EuroVis Workshop on Visual Analytics (EuroVA). The Eurographics Association, 2017. doi: 10.2312/eurova.20171123
8. R. Borgo, J. Kehrer, D. H. S. Chung, E. Maguire, R. S. Laramee, H. Hauser, M. Ward, and M. Chen. Glyph-based visualization: Foundations, design guidelines, techniques and applications. In Eurographics State of the Art Reports, EG STARs, pp. 39-63. Eurographics Association, May 2013. doi: 10.2312/conf/EG2013/stars/039-063
9. L. Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
10. E. T. Brown, J. Liu, C. E. Brodley, and R. Chang. Dis-function: Learning distance functions interactively. In IEEE Visual Analytics Science and Technology (VAST), pp. 83-92. IEEE, 2012. doi: 10.1109/VAST.2012. 6400486
11. K. Charmaz. Constructing grounded theory. Sage, 2014.
12. K. Chen and L. Liu. Clustermap: Labeling clusters in large datasets via visualization. In ACM Conference on Information and Knowledge Management (CIKM), pp. 285-293. ACM, New York, NY, USA, 2004. doi: 10.1145/1031171.1031233
13. C. Cortes and V. Vapnik. Support-vector networks. Machine learning, 20(3):273-297, 1995.
14. A. Culotta and A. McCallum. Reducing labeling effort for structured prediction tasks. In Conference on Artificial Intelligence (AAAI), pp. 746-751. AAAI Press, 2005.
15. G. Cumming and S. Finch. Inference by eye: confidence intervals and how to read pictures of data. American Psychologist, 60(2):170, 2005. doi: 10. 1002/sim.3471
16. I. Dagan and S. P. Engelson. Committee-based sampling for training probabilistic classifiers. In Conference on Machine Learning (ICML), pp. 150-157. Morgan Kaufmann Pub., 1995.
17. C. K. Dagli, S. Rajaram, and T. S. Huang. Leveraging active learning for relevance feedback using an information theoretic diversity measure. In Conference on Image and Video Retrieval, pp. 123-132. Springer, 2006. doi: 10.1007/11788034.13
18. R. O. Duda, P. E. Hart, D. G. Stork, et al. Pattern classification, vol. 2. Wiley New York, 1973.
19. C. Ferri, J. Hernandez-Orallo, and R. Modroiu. An experimental comparison of performance measures for classification. Pattern Recogn. Lett., 30(1):27-38, 2009. doi: 10.1016/j.patrec.2008.08.010
20. R. A. FISHER. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2):179-188, 1936. doi: 10.1469-1809.1936. Tb02137.x
21. S. Garg, I. Ramakrishnan, and K. Mueller. A visual analytics approach to model learning. na, pp. 67-74, 2010. doi: 10.1109/VAST.2010.5652484
22. F. Heimerl, S. Koch, H. Bosch, and T. Ertl. Visual classifier training for text document retrieval. IEEE Transactions on Visualization and Computer Graphics (TVCG), 18(12):2839-2848, 2012. doi: 10.1109/TVCG.2012. 277
23. B. Hoferlin, R. Netzel, M. Hoferlin, D. Weiskopf, and G. Heidemann. Inter-active learning of ad-hoc classifiers for video visual analytics. In IEEE Visual Analytics Science and Technology (VAST), pp. 23-32, 2012. doi: 10.1109/VAST.2012.6400492
24. S. C. Hoi, R. Jin, and M. R. Lyu. Large-scale text categorization by batch mode active learning. In World Wide Web, pp. 633-642. ACM, 2006. doi: 10.1145/1135777.1135870
25. K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989. doi: 10.1016/0893-6080(89)90020-8
26. S. Ingram, T. Munzner, V. Irvine, M. Tory, S. Bergner, and T. Moller. Dimstiller: Workflows for dimensional analysis and reduction. In IEEE Visual Analytics Science and Technology (VAST), pp. 3-10. IEEE, 2010. doi: 10.1109/VAST.2010.5652392
27. I. T. Jolliffe. Principal Component Analysis. Springer, 3rd ed., 2002. doi: 10.1007/978-1-4757-1904-8.7
28. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing userstudies withmechan-ical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pp. 453-156. ACM, New York, NY, USA, 2008. doi: 10.1145/1357054.1357127
29. J. Krause, A. Perer, and E. Bertini. Infuse: Interactive feature selection for predictive modeling of high dimensional data. IEEE Trans on Vis. & Comp. Graph., 20(12):1614-1623, 2014. doi: 10.1109/TVCG.2014. 2346482
30. J. B. Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1):1-27, 1964. doi: 10. 1007/BF02289565
31. N. Landwehr, M. Hall, and E. Frank. Logistic model trees. Machine Learning, 59(1-2):161-205, 2005. doi: 10.1007/s10994-005-0466-3
32. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998. doi: 10.1109/5.726791
33. J. M. Lewis, M. Ackerman, and V. R. de Sa. Human cluster evaluation and formal quality measures: A comparative study. In Annual Meeting of the Cognitive Science Society (CogSci), pp. 1870-1875, 2012.
34. J. M. Lewis, L. van der Maaten, and V. R. de Sa. A behavioral investigation of dimensionality reduction. In Annual Meeting of the Cognitive Science Society (CogSci), 2012.
35. G. M. Mamani, F. M. Fatore, L. G. Nonato, and F. V. Paulovich. User-driven feature space transformation. In CGF, pp. 291-299, 2013. doi: 10. 1111/cgf.12116
36. N. A. H. Mamitsuka. Query learning strategies using boosting and bagging. In Conference on Machine Learning (ICML), vol. 1. Morgan Kaufmann Pub., 1998.
37. A. McCallum and K. Nigam. Employing em and pool-based active learning for text classification. In Conference on Machine Learning (ICML), pp. 350-358. Morgan Kaufmann Pub., San Francisco, CA, USA, 1998.
38. J. Mohrmann, S. Bernstein, T. Schlegel, G. Werner, and G. Heidemann. Improving the usability of interfaces for the interactive semi-automatic labeling of large image data sets. In Human Computer Interaction. Design and Development Approaches, pp. 618-627. Springer, 2011.
39. F. Olsson. A literature survey of active machine learning in the context of natural language processing. Technical Report, no. 3600, 2009.
40. S. Pajer, M. Streit, T. Torsney-Weir, F. Spechtenhauser, T. Mller, and H. Piringer. Weightlifter: Visual weight space exploration for multi-criteria decision making. IEEE Transactions on Visualization and Computer Graphics (TVCG), 23(1):611-620, 2017. doi: 10.1109/TVCG.2016. 2598589
41. F. V. Paulovich, M. C. F. Oliveira, and R. Minghim. The projection explorer: A flexible tool for projection-based multidimensional visualization. In Symposium on Computer Graphics and Image Processing (SIBGRAPI), pp. 27-36. IEEE Computer Society, Washington, DC, USA, 2007. doi: 10. 1109/SIBGRAPI.2007.21
42. G.-J. Qi, X.-S. Hua, Y. Rui, J. Tang, and H.-J. Zhang. Two-dimensional multilabel active learning with an efficient online adaptation model for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(10):1880-1897, 2009. doi: 10.1109/TPAMI.2008.218
43. D. Sacha, M. Sedlmair, L. Zhang, J. A. Lee, D. Weiskopf, S. C. North, and D. A. Keim. Human-Centered Machine Learning Through Interactive Visualization: Review and Open Challenges. In Symp. on Artificial Neural Networks, Computational Intelligence and Machine Learning, 2016.
44. D. Sacha, L. Zhang, M. Sedlmair, J. A. Lee, J. Peltonen, D. Weiskopf, S. C. North, and D. A. Keim. Visual Interaction with Dimensionality Reduction: A Structured Literature Analysis. IEEE Transactions on Visualization and Computer Graphics (TVCG), 23(01):241-250, 2016. doi: 10.1109/TVCG. 2016.2598495
45. G. Salton and C. Buckley. Improving retrieval performance by relevance feedback. Readings in Information Retrieval, 24:5, 1997. doi: 10.1002/(SICI)1097-4571(199006)41:4288::AID-ASI83.0.CO;2-H
46. J. W. Sammon. A nonlinear mapping for data structure analysis. IEEE Trans. Comput., 18(5):401-409, 1969. doi: 10.1109/T-C.1969.222678
47. A. Sarkar, M. Spott, A. F. Blackwell, and M. Jamnik. Visual discovery and model-driven explanation of time series patterns. In Visual Languages and Human-Centric Computing (VL/HCC), pp. 78-86. IEEE, 2016. doi: 10.1109/VLHCC.2016.7739668
48. T. Scheffer, C. Decomain, and S. Wrobel. Active hidden markov models for information extraction. In Conference on Advances in Intelligent Data Analysis (IDA), pp. 309-318. Springer-Verlag, London, UK, UK, 2001. doi: 10.1007/3-540-44816-0.31
49. T. Schreck and C. Panse. A new metaphor for projection-based visual analysis and data exploration. In Visualization and Data Analysis 2007, SPIE Proceedings, pp. 64950L-64950L-12, 2007. doi: 10.1117/12.697879
50. T. Schreck, M. Schler, F. Zeilfelder, and K. Worm. Butterfly plots for visual analysis of large point cloud data. In WSCG, pp. 33-40. University of West Bohemia, Plzen, 2008.
51. T. Schreck, T. von Landesberger, and S. Bremm. Techniques for precision-based visual analysis of projected data. Information Visualization, 9(3):181-193, 2010. doi: 10.1057/ivs.2010.2
52. M. Sedlmair, T. Munzner, and M. Tory. Empirical guidance on scatterplot and dimension reduction technique choices. IEEE Transactions on Visualization and Computer Graphics (TVCG), 19(12):2634-2643, 2013. doi: 10.1109/TVCG.2013.153
53. M. Sedlmair, A. Tatu, T. Munzner, and M. Tory. A taxonomy of visual cluster separation factors. Computer Graphics Forum, 31(3pt4):1335-1344, 2012. doi: 10.1111/j.1467-8659.2012.03125.x
54. D. Seebacher, M. Stein, H. Janetzko, and D. A. Keim. Patent Retrieval: A Multi-Modal Visual Analytics Approach. In EuroVis Workshop on Visual Analytics (EuroVA), pp. 013-017. Eurographics, 2016. doi: 10. 2312/eurova.20161118
55. C. Seifert and M. Granitzer. User-based active learning. In IEEE Conference on Data Mining Workshops (ICDMW), pp. 418-425, 2010. doi: 10. 1109/ICDMW.2010.181
56. B. Settles. Active learning literature survey. Tech. Report 1648, Univ. of Wisconsin-Madison, 2009.
57. B. Settles. Closing the loop: Fast, interactive semi-supervised annotation with queries on features and instances. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1467-1478. Computational Linguistics, 2011.
58. B. Settles and M. Craven. An analysis of active learning strategies for sequence labeling tasks. In Empirical Methods in Natural Language Processing, pp. 1070-1079. Computational Linguistics, 2008.
59. B. Settles, M. Craven, and S. Ray. Multiple-instance active learning. In Advances in neural information processing systems, pp. 1289-1296, 2008.
60. H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In Proc. of the 5th Ann. Worksh. on Comput. Learning Theory, COLT '92, pp. 287-294. ACM, New York, NY, USA, 1992. doi: 10.1145/130385.130417
61. C. Shannon. A mathematical theory of communication. Bell system technical journal, 27, 1948.
62. E. Simpson. Measurement of diversity. Nature, 163(4148):688-688, 1949. doi: doi:10.1038/163688a0
63. M. Sips, B. Neubert, J. P. Lewis, and P. Hanrahan. Selecting good views of high-dimensional data using class consistency. Computer Graphics Forum, 28(3):831-838, 2009. doi: 10.1111/j.1467-8659.2009.01467.x
64. J. Stahnke, M. Drk, B. Mller, and A. Thom. Probing projections: Interaction techniques for interpreting arrangements and errors of dimensionality reductions. IEEE Transactions on Visualization and Computer Graphics (TVCG), 22(1):629-638, 2016. doi: 10.1109/TVCG.2015.2467717
65. D. Tuia, M. Volpi, L. Copa, M. Kanevski, and J. Munoz-Mari. A survey of active learning algorithms for supervised remote sensing image classification. IEEE Selected Topics in Signal Proc., 5(3):606-617, 2011. doi: 10. 1109/JSTSP.2011.2139193
66. L. van der Maaten and G. E. Hinton. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008.
67. J. Vendrig, I. Patras, C. Snoek, M. Worring, J. den Hartog, S. Raaijmakers, J. van Rest, and D. A. van Leeuwen. Trec feature extraction by active learning. In TREC, 2002.
68. M. Wang and X.-S. Hua. Active learning in multimedia annotation and retrieval: A survey. ACM Trans. Intell. Syst. Technol., 2(2):10:1-10:21, 2011. doi: 10.1145/1899412.1899414
69. Y. Wang, K. Feng, C. Xiaowei, J. Zhang, C.-W. Fu, M. Sedlmair, X. Yu, and B. Chen. A Perception-Driven Approach to Supervised Dimensionality Reduction for Visualization. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2017. doi: 10.1109/TVCG.2017.2701829
70. M. Ware, E. Frank, G. Holmes, M. Hall, and I. H. Witten. Interactive machine learning: letting users build classifiers. Human-Computer Studies, pp. 281-292, 2001. doi: 10.1006/ijhc.2001.0499
71. N. Weber, M. Waechter, S. C. Amend, S. Guthe, and M. Goesele. Rapid, Detail-Preserving Image Downscaling. In ACM SIGGRAPH Asia, 2016. doi: 10.1145/2980179.2980239
72. Y. Wu, I. Kozintsev, J.-Y. Bouguet, and C. Dulong. Sampling strategies for active learning in personal photo retrieval. In Multimedia and Expo, 2006 IEEE International Conference on, pp. 529-532. IEEE, 2006. doi: 10.1109/ICME.2006.262442