Towards better analysis of machine learning models: A visual analytics perspective

Visual Informatics

Volumn 1, Issue 1, 2017, Pages 48-56

  Liu, Shixia ^a   Wang, Xiting ^a   Liu, Mengchen ^a   Zhu, Jun ^a  

^a TSINGHUA UNIVERSITY   (China)

Author keywords

Diagnosis; Interactive model analysis; Interactive visualization; Machine learning; Refinement; Understanding

Indexed keywords

DATA ANALYTICS; DATA MINING; DATA VISUALIZATION; DIAGNOSIS; LEARNING SYSTEMS; MACHINE LEARNING; TURING MACHINES; VISUALIZATION;

COMPREHENSIVE ANALYSIS; DATA MINING PROBLEMS; INTERACTIVE MODELING; INTERACTIVE VISUALIZATIONS; MACHINE LEARNING MODELS; REFINEMENT; RESEARCH OPPORTUNITIES; UNDERSTANDING;

ADVANCED ANALYTICS;

EID: 85066750614     PISSN: 25432656     EISSN: 2468502X     Source Type: Journal    
DOI: 10.1016/j.visinf.2017.01.006     Document Type: Article

References (53)

1. Alsallakh, B., Hanbury, A., Hauser, H., Miksch, S., Rauber, A., Visual methods for analyzing probabilistic classification data. IEEE TVCG 20:12 (2014), 1703–1712, 10.1109/TVCG.2014.2346660.
2. Amershi, S., Chickering, M., Drucker, S.M., Lee, B., Simard, P., Suh, J. 2015. Modeltracker: Redesigning performance analysis tools for machine learning. In: CHI, pp. 337–346. http://dx.doi.org/10.1145/2702123.2702509.
3. Cao, N., Shi, C., Lin, S., Lu, J., Lin, Y.R., Lin, C.Y., Targetvue: Visual analysis of anomalous user behaviors in online communication systems. IEEE TVCG 22:1 (2016), 280–289.
4. Chen, J., Zhu, J., Wang, Z., Zheng, X., Zhang, B. 2013. Scalable inference for logistic-normal topic models. In: NIPS.
5. Choo, J., Lee, C., Reddy, C.K., Park, H., Utopian: User-driven topic modeling based on interactive nonnegative matrix factorization. IEEE TVCG 19:12 (2013), 1992–2001, 10.1109/TVCG.2013.212.
6. Chuang, J., Gupta, S., Manning, C.D., Heer, J. 2013. Topic model diagnostics: Assessing domain relevance via topical alignment. In: ICML, pp. 612–620.
7. Cohn, D., Atlas, L., Ladner, R., Improving generalization with active learning. Mach. Learn. 15:2 (1994), 201–221.
8. Cohn, D.A., Ghahramani, Z., Jordan, M.I., Active learning with statistical models. JAIR 4:1 (1996), 129–145.
9. Correa, C.D., Chan, Y.H., Ma, K.L. 2009. A framework for uncertainty-aware visual analytics. In: VAST, pp. 51–58.
10. Craven, M.W., Shavlik, J.W., Visualizing learning and computation in artificial neural networks. Int. J. Artif. Intell. Tools 1:03 (1992), 399–425.
11. Cui, W., Liu, S., Tan, L., Shi, C., Song, Y., Gao, Z., Qu, H., Tong, X., TextFlow: Towards better understanding of evolving topics in text. IEEE TVCG 17:12 (2011), 2412–2421.
12. Cui, W., Liu, S., Wu, Z., Wei, H., How hierarchical topics evolve in large text corpora. IEEE TVCG 20:12 (2014), 2281–2290.
13. DAR, 2016. Explainable Artificial Intelligence (XAI). http://www.darpa.mil/program/explainable-artificial-intelligence.
14. Dou, W., Liu, S., Topic-and time-oriented visual text analysis. IEEE Comput. Graph. Appl. 36:4 (2016), 8–13.
15. Dou, W., Yu, L., Wang, X., Ma, Z., Ribarsky, W., HierarchicalTopics: Visually exploring large text collections using topic hierarchies. IEEE TVCG 19:12 (2013), 2002–2011.
16. Fekete, J.-D., Visual analytics infrastructures: From data management to exploration. Computer 46:7 (2013), 22–29.
17. Harley, A.W., An interactive node-link visualization of convolutional neural networks. International Symposium on Visual Computing, 2015, Springer, 867–877.
18. Heimerl, F., Koch, S., Bosch, H., Ertl, T., Visual classifier training for text document retrieval. IEEE TVCG 18:12 (2012), 2839–2848.
19. Krizhevsky, A., Sutskever, I., Hinton, G.E. 2012. Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105.
20. Lake, B.M., Salakhutdinov, R., Tenenbaum, J.B., Human-level concept learning through probabilistic program induction. Science 350:6266 (2015), 1332–1338.
21. LeCun, Y., Bengio, Y., Hinton, G., Deep learning. Nature 521:7553 (2015), 436–444.
22. Lee, D.D., Seung, H.S., Learning the parts of objects by non-negative matrix factorization. Nature 401:6755 (1999), 788–791.
23. Letham, B., Rudin, C., McCormick, T.H., Madigan, D., Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. Ann. Appl. Stat. 9:3 (2015), 1350–1371.
24. Liu, S., Cui, W., Wu, Y., Liu, M., A survey on information visualization: Recent advances and challenges. The Visual Computer 30:12 (2014), 1373–1393.
25. Liu, M., Liu, S., Zhu, X., Liao, Q., Wei, F., Pan, S., An uncertainty-aware approach for exploratory microblog retrieval. IEEE TVCG 22:1 (2016), 250–259, 10.1109/TVCG.2015.2467554.
26. Liu, M., Shi, J., Li, Z., Li, C., Zhu, J.J.H., Liu, S., Towards better analysis of deep convolutional neural networks. IEEE TVCG 23:1 (2017), 91–100, 10.1109/IEEE TVCG.2016.2598831.
27. Liu, S., Wang, X., Chen, J., Zhu, J., Guo, B. 2014b. TopicPanorama: A full picture of relevant topics. In: VAST, pp. 183–192. http://dx.doi.org/10.1109/VAST.2014.7042494.
28. Liu, S., Yin, J., Wang, X., Cui, W., Cao, K., Pei, J., Online visual analytics of text streams. IEEE TVCG 22:11 (2016), 2451–2466, 10.1109/IEEE TVCG.2015.2509990.
29. Liu, S., Zhou, M.X., Pan, S., Song, Y., Qian, W., Cai, W., Lian, X., TIARA: Interactive, topic-based visual text summarization and analysis. ACM TIST 3:2 (2012), 25:1–25:28.
30. Maaten, L.v.d., Hinton, G., Visualizing data using t-SNE. J. Mach. Learn. Res. 9:Nov (2008), 2579–2605.
31. McCallumzy, A.K., Nigamy, K. 1998. Employing EM and pool-based active learning for text classification. In: ICML, pp. 359–367.
32. Mühlbacher, T., Piringer, H., Gratzl, S., Sedlmair, M., Streit, M., Opening the black box: Strategies for increased user involvement in existing algorithm implementations. IEEE TVCG 20:12 (2014), 1643–1652, 10.1109/IEEE TVCG.2014.2346578.
33. Paiva, J.G., Florian, L., Pedrini, H., Telles, G., Minghim, R., Improved similarity trees and their application to visual data classification. IEEE TVCG 17:12 (2011), 2459–2468.
34. Paiva, J.G.S., Schwartz, W.R., Pedrini, H., Minghim, R., An approach to supporting incremental visual data classification. IEEE TVCG 21:1 (2015), 4–17, 10.1109/TVCG.2014.2331979.
35. Pezzotti, N., Lelieveldt, B., Maaten, L.V.D., Hollt, T., Eisemann, E., Vilanova, A., Approximated and user steerable tSNE for progressive visual analytics. IEEE TVCG PP:99 (2016), 1–14, 10.1109/TVCG.2016.2570755.
36. Rauber, P.E., Fadel, S., Falcao, A., Telea, A., Visualizing the hidden activity of artificial neural networks. IEEE TVCG 23:1 (2017), 101–110, 10.1109/TVCG.2016.2598838.
37. Ren, D., Amershi, S., Lee, B., Suh, J., Williams, J.D., Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE TVCG 23:1 (2017), 61–70, 10.1109/TVCG.2016.2598828.
38. Sacha, D., Senaratne, H., Kwon, B.C., Ellis, G., Keim, D.A., The role of uncertainty, awareness, and trust in visual analytics. IEEE TVCG 22:1 (2016), 240–249.
39. Settles, B., Active Learning Synthesis Lectures on Artificial Intelligence and Machine Learning, 2012, Morgan & Claypool.
40. Stolper, C.D., Perer, A., Gotz, D., Progressive visual analytics: User-driven visual exploration of in-progress analytics. IEEE TVCG 20:12 (2014), 1653–1662, 10.1109/TVCG.2014.2346574.
41. Streeter, M.J., Ward, M.O., Alvarez, S.A., 2001. Nvis: An interactive visualization tool for neural networks.
42. Tam, G.L., Kothari, V., Chen, M., An analysis of machine- and human-analytics in classification. IEEE TVCG 23:1 (2017), 71–80, 10.1109/TVCG.2016.2598829.
43. Turner, R. 2016. A model explanation system: Latest updates and extensions. In: ICML Workshop.
44. Tzeng, F.Y., Ma, K.L. 2004. Acluster-space visual interface for arbitrary dimensional classification of volume data. In: Sixth Joint Eurographics - IEEE TCVG Conference on Visualization, pp. 17–24. http://dx.doi.org/10.2312/VisSym/VisSym04/017-024.
45. Tzeng, F.Y., Ma, K.L. 2005. Opening the black box - data driven visualization of neural networks. In: IEEE Visualization, pp. 383–390. http://dx.doi.org/10.1109/VISUAL.2005.1532820.
46. Wang, X., Liu, S., Liu, J., Chen, J., Zhu, J., Guo, B., TopicPanorama: A full picture of relevant topics. IEEE TVCG 22:12 (2016), 2508–2521, 10.1109/TVCG.2016.2515592.
47. Wang, X., Liu, S., Song, Y., Guo, B. 2013. Mining evolutionary multi-branch trees from text streams. In: KDD, pp. 722–730.
48. Wei, F., Li, W., Lu, Q., He, Y. 2008. Query-sensitive mutual reinforcement chain and its application in query-oriented multi-document summarization. In: SIGIR, pp. 283–290.
49. Wold, S., Esbensen, K., Geladi, P., Principal component analysis. Chemometr. Intell. Lab. Syst. 2:1 (1987), 37–52.
50. Wu, Y., Wei, F., Liu, S., Au, N., Cui, W., Zhou, H., Qu, H., OpinionSeer: Interactive visualization of hotel customer feedback. IEEE TVCG 16:6 (2010), 1109–1118.
51. Wu, Y., Yuan, G.X., Ma, K.L., Visualizing flow of uncertainty through analytical processes. IEEE TVCG 18:12 (2012), 2526–2535.
52. Zahavy, T., Ben-Zrihem, N., Mannor, S. 2016. Graying the black box: Understanding dqns. In: ICML pp. 1899–1908.
53. Zhao, J., Cao, N., Wen, Z., Song, Y., Lin, Y.R., Collins, C., #FluxFlow: Visual analysis of anomalous information spreading on social media. IEEE TVCG 20:12 (2014), 1773–1782.