The State of the Art in Integrating Machine Learning into Visual Analytics

Computer Graphics Forum

Volumn 36, Issue 8, 2017, Pages 458-486

  Endert, A ^a   Ribarsky, W ^b   Turkay, C ^c   Wong, B L William ^d   Nabney, I ^e   Blanco, I Díaz ^f   Rossi, F ^g  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY OF NORTH CAROLINA AT CHARLOTTE   (United States)

^c CITY UNIVERSITY   (United Kingdom)

^d MIDDLESEX UNIVERSITY   (United Kingdom)

^e ASTON UNIVERSITY   (United Kingdom)

^f UNIVERSITY OF OVIEDO   (Spain)

^g UNIVERSITÉ PARIS 1 PANTHÉON SORBONNE   (France)

Author keywords

data mining; information visualization; visual analytics

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER SUPPORTED COOPERATIVE WORK; DATA MINING; DATA VISUALIZATION; INFORMATION SYSTEMS; VISUALIZATION;

ANALYTIC TECHNIQUE; ANALYTICAL REASONING; FUTURE RESEARCH DIRECTIONS; INFORMATION VISUALIZATION; INTEGRATING MACHINES; RESEARCH ADVANCES; VISUAL ANALYTICS; VISUAL ANALYTICS SYSTEMS;

LEARNING SYSTEMS;

EID: 85016273988     PISSN: 01677055     EISSN: 14678659     Source Type: Journal    
DOI: 10.1111/cgf.13092     Document Type: Article

References (186)

1. [20008] Alias-i. 2008. LingPipe 4.0.1, 2008. URL: http://alias-i.com/lingpipe.
2. [AAB*10] Andrienko G., Andrienko N., Bremm S., Schreck T., Von Landesberger T., Bak P., Keim D.: Space-in-time and time-in-space self-organizing maps for exploring spatiotemporal patterns. In Computer Graphics Forum (2010), vol. 29, Wiley Online Library, pp. 913–922.
3. [AAR*09] Andrienko G., Andrienko N., Rinzivillo S., Nanni M., Pedreschi D., Giannotti F.: Interactive visual clustering of large collections of trajectories. In Visual Analytics Science and Technology, 2009. VAST 2009. IEEE Symposium on (2009), IEEE, pp. 3–10.
4. [ABV14] Awasthi P., Balcan M., Voevodski K.: Local algorithms for interactive clustering. In Proceedings of The 31st International Conference on Machine Learning (2014), E. P. Xing and T. Jebara (Eds.), vol. 32 of JMLR Proceedings, JMLR.org, pp. 550–558.
5. [ACKK14] Amershi S., Cakmak M., Knox W. B., Kulesza T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35, 4 (2014), 105–120.
6. [ADT*13] Amir E.-a. D., Davis K. L., Tadmor M. D., Simonds E. F., Levine J. H., Bendall S. C., Shenfeld D. K., Krishnaswamy S., Nolan G. P., Pe'er D.: Visne enables visualization of high dimensional single-cell data and reveals phenotypic heterogeneity of leukemia. Nature Biotechnology 35, 6 (2013), 545–552.
7. [AEN10] Andrews C., Endert A., North C.: Space to think: large high-resolution displays for sensemaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2010), ACM, pp. 55–64.
8. [AES05] Amar R., Eagan J., Stasko J.: Low-level components of analytic activity in information visualization. In IEEE Symposium on Information Visualization, 2005. INFOVIS 2005 (2005), pp. 111–117. doi:10.1109/INFVIS.2005.1532136.
9. [Alp14] Alpaydin E.: Introduction to Machine Learning. MIT Press, Cambridge, MA, 2014.
10. [AN12] Andrews C., North C.: Analyst's Workspace: An embodied sensemaking environment for large, high-resolution displays. In 2012 IEEE Conference on Visual Analytics Science and Technology (VAST) (2012), pp. 123–131. doi:10.1109/VAST.2012.6400559.
11. [Anc12] Ancona D.: Framing and acting in the unknown. S. Snook, N. Nohria, & R. Khurana, The Handbook for Teaching Leadership (2012), 3–19.
12. [AW12] Ahmed Z., Weaver C.: An Adaptive Parameter Space-Filling Algorithm for Highly Interactive Cluster Exploration. In Procedings of IEEE Symposium on Visual Analytics Science and Technology (VAST) (2012).
13. [BBM04] Basu S., Banerjee A., Mooney R. J.: Active semi-supervision for pairwise constrained clustering. In Proceedings of the 2004 SIAM International Conference on Data Mining (2004), pp. 333–344. URL: http://epubs.siam.org/doi/abs/10.1137/1.9781611972740.31, arXiv:http://epubs.siam.org/doi/pdf/10.1137/1.9781611972740.31, doi:10.1137/1.9781611972740.31.
14. [BDW08] Basu S., Davidson I., Wagstaff K.: Constrained Clustering: Advances in Algorithms, Theory, and Applications. CRC Press, 2008.
15. [BEK14] Best D. M., Endert A., Kidwell D.: 7 Key Challenges for Visualization in Cyber Network Defense. In Proceedings of the Eleventh Workshop on Visualization for Cyber Security (New York, NY, USA, 2014), VizSec '14, ACM, pp. 33–40. doi:10.1145/2671491.2671497.
16. [BF11] Bennett K. B., Flach J. M.: Display and Interface Design: Subtle Science, Exact Art. CRC Press, 2011.
17. [BH12] Balcan M. F., Hanneke S.: Robust interactive learning. In Proceedings of the 25th Annual Conference on Learning Theory (COLT) (Edinburgh, Scotland, June 2012), vol. 23 of JMLR Workshop and Conference Proceedings.
18. [BKSS14] Behrisch M., Korkmaz F., Shao L., Schreck T.: Feedback-driven interactive exploration of large multidimensional data supported by visual classifier. In Visual Analytics Science and Technology (VAST), 2014 IEEE Conference on (2014), IEEE, pp. 43–52.
19. [BL09] Blei D., Lafferty J.: Text mining: Theory and applications, chapter topic models, 2009.
20. [BLBC12] Brown E. T., Liu J., Brodley C. E., Chang R.: Dis-function: Learning distance functions interactively. In Visual Analytics Science and Technology (VAST), 2012 IEEE Conference on (2012), IEEE, pp. 83–92.
21. [BM13] Bryan N. J., Mysore G. J.: An efficient posterior regularized latent variable model for interactive sound source separation. In Proceedings of The 30th International Conference on Machine Learning (ICML) (Atlanta, Georgia, USA, 2013), S. Dasgupta and D. McAllester (Eds.), vol. 28 of JMLR Workshop and Conference Proceedings.
22. [BNJ03] Blei D. M., Ng A. Y., Jordan M. I.: Latent dirichlet allocation. The Journal of Machine Learning Research 3 (2003), 993–1022.
23. [BOZ*14] Brown E. T., Ottley A., Zhao H., Lin Q., Souvenir R., Endert A., Chang R.: Finding Waldo: Learning about Users from their Interactions.
24. [BPFG11] Berger W., Piringer H., Filzmoser P., Gröller E.: Uncertainty-aware exploration of continuous parameter spaces using multivariate prediction. Computer Graphics Forum 30, 3 (2011), 911–920.
25. [CBN*12] Choo J., Bohn S., Nakamura G., White A. M., Park H.: Heterogeneous data fusion via space alignment using nonmetric multidimensional scaling. In SDM (2012), SIAM, pp. 177–188.
26. [CCM08] Cohn D., Caruana R., McCallum A.: Semi-supervised clustering with user feedback. In Constrained Clustering: Advances in Algorithms, Theory, and Applications, S. Basu, I. Davidson, K. Wagstaff (Eds.). CRC Press, 2008, ch. 2, pp. 17–32.
27. [CDS09] Crossno P. J., Dunlavy D. M., Shead T. M.: Lsaview: a tool for visual exploration of latent semantic modeling. In Visual Analytics Science and Technology, 2009. VAST 2009. IEEE Symposium on (2009), IEEE, pp. 83–90.
28. [CDW*16] Cho I., Dou W., Wang D. X., Sauda E., Ribarsky W.: Vairoma: A visual analytics system for making sense of places, times, and events in roman history. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2016), 210–219.
29. [Chi96] Chickering D. M.: Learning bayesian networks is np-complete. In Learning from Data. Springer, 1996, pp. 121–130.
30. [Chr06] Chris N.: Toward Measuring Visualization Insight. 6–9. doi:10.1109/MCG.2006.70.
31. [CLKP10] Choo J., Lee H., Kihm J., Park H.: ivisclassifier: An interactive visual analytics system for classification based on supervised dimension reduction. In Visual Analytics Science and Technology (VAST), 2010 IEEE Symposium on (2010), IEEE, pp. 27–34.
32. [CMS99] Card S. K., Mackinlay J. D., Shneiderman B.: Readings in Information Visualization: Using Vision to Think. Morgan Kaufmann Publishers Inc., 1999.
33. [CO10] Conti, S., OâHagan, A.: Bayesian emulation of complex multi-output and dynamic computer models. Journal of Statistical Planning and Inference 140, 3 (2010), 640–651.
34. [CP13] Choo J., Park H.: Customizing Computational Methods for Visual Analytics with Big Data. IEEE Computer Graphics and Applications 33, 4 (2013), 22–28. doi:10.1109/MCG.2013.39.
35. [DCE*15] Dou W., Cho I., ElTayeby O., Choo J., Wang X., Ribarsky W.: Demographicvis: Analyzing demographic information based on user generated content. In 2015 IEEE Conference on Visual Analytics Science and Technology (VAST), (2015), IEEE, pp. 57–64.
36. [DCP*14] Díaz I., Cuadrado A. A., Pérez D., García F. J., Verleysen M.: Interactive Dimensionality Reduction for Visual Analytics. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (Bruges, Belgium, 2014).
37. [DCV16] Díaz I., Cuadrado A. A., Verleysen M.: A state-space model on interactive dimensionality reduction. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (Bruges, Belgium, April 2016), M. Verleysen M. (Ed.), pp. 647–652.
38. [DFB11] Drucker S. M., Fisher D., Basu S.: Helping Users Sort Faster with Adaptive Machine Learning Recommendations. Springer-Verlag, pp. 187–203.
39. [DHF07] De Haan L., Ferreira A.: Extreme Value Theory: An Introduction. Springer Science & Business Media, 2007.
40. [DWCR11] Dou W., Wang X., Chang R., Ribarsky W.: Paralleltopics: A probabilistic approach to exploring document collections. In Visual Analytics Science and Technology (VAST), 2011 IEEE Conference on (2011), IEEE, pp. 231–240.
41. [DWS*12] Dou W., Wang X., Skau D., Ribarsky W., Zhou M. X.: Leadline: Interactive visual analysis of text data through event identification and exploration. In Visual Analytics Science and Technology (VAST), 2012 IEEE Conference on (2012), IEEE, pp. 93–102.
42. [EBN13] Endert A., Bradel L., North C.: Beyond Control Panels: Direct Manipulation for Visual Analytics. IEEE Computer Graphics and Applications 33, 4 (2013), 6–13. doi:10.1109/MCG.2013.53.
43. [ECNZ15] Endert A., Chang R., North C., Zhou M.: Semantic Interaction: Coupling Cognition and Computation through Usable Interactive Analytics. IEEE Computer Graphics and Applications 35, 4 (July 2015), 94–99. doi:10.1109/MCG.2015.91.
44. [EFN12a] Endert A., Fiaux P., North C.: Semantic Interaction for Sensemaking: Inferring Analytical Reasoning for Model Steering. Visualization and Computer Graphics, IEEE Transactions on 18 (2012), 2879–2888. 12. doi:10.1109/tvcg.2012.260.
45. [EFN12b] Endert A., Fiaux P., North C.: Semantic interaction for visual text analytics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2012), ACM, pp. 473–482.
46. [EHM*11] Endert A., Han C., Maiti D., House L., Leman S. C., North C.: Observation-level Interaction with Statistical Models for Visual Analytics. In IEEE VAST (2011), pp. 121–130.
47. [EHR*14] Endert A., Hossain M. S., Ramakrishnan N., North C., Fiaux P., Andrews C.: The human is the loop: new directions for visual analytics. Journal of Intelligent Information Systems (Jan. 2014), 1–25. doi:10.1007/s10844-014-0304-9.
48. [EKHW08] Eccles R., Kapler T., Harper R., Wright W.: Stories in GeoTime. Information Visualization 7 (2008), 3–17. 1. doi:10.1145/1391107.1391109.
49. [EMJ*11] Elmqvist N., Moere A. V., Jetter H.-C., Cernea D., Reiterer H., Jankun-Kelly T.: Fluid interaction for information visualization. Information Visualization 10 4 (2011), 327–340.
50. [EPT*05] Elm W., Potter S., Tittle J., Woods D., Grossman J., Patterson E.: Finding decision support requirements for effective intelligence analysis tools. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (2005), vol. 49, SAGE Publications, pp. 297–301.
51. [FHT01] Friedman J., Hastie T., Tibshirani R.: The Elements of Statistical Learning, vol. 1. Springer series in statistics Springer, Berlin, 2001.
52. [FJA*11] Fernstad S., Johansson J., Adams S., Shaw J., Taylor D.: Visual exploration of microbial populations. In Biological Data Visualization (BioVis), 2011 IEEE Symposium on (2011), pp. 127–134.
53. [FMK12] Fischer F., Mansmann F., Keim D. A.: Real-time visual analytics for event data streams. In Proceedings of the 27th Annual ACM Symposium on Applied Computing (New York, NY, USA, 2012), SAC '12, ACM, pp. 801–806. URL: http://doi.acm.org/10.1145/2245276.2245432, doi:10.1145/2245276.2245432.
54. [FOJ03] Fails J. A., Olsen Jr D. R.: Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent user Interfaces (2003), ACM, pp. 39–45.
55. [FP07] Fu W.-T., Pirolli P.: Snif-act: A cognitive model of user navigation on the world wide web. Human–Computer Interaction 22, 4 (2007), 355–412.
56. [FPDs12] Fisher D., Popov I., Drucker S., schraefel m.: Trust Me, I'M Partially Right: Incremental Visualization Lets Analysts Explore Large Datasets Faster. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2012), CHI '12, ACM, pp. 1673–1682. doi:10.1145/2207676.2208294.
57. [FSB*13] Fiaux P., Sun M., Bradel L., North C., Ramakrishnan N., Endert A.: Bixplorer: Visual Analytics with Biclusters. Computer 46, 8 (2013), 90–94.
58. [FWG09] Fuchs R., Waser J., Gröller M. E.: Visual human+machine learning. IEEE TVCG 15, 6 (2009), 1327–1334.
59. [GB11] Guillory A., Bilmes J.: Simultaneous learning and covering with adversarial noise. In Proceedings of the 28th International Conference on Machine Learning (ICML-11) (New York, NY, USA, June 2011), L. Getoor and T. Scheffer (Eds.), ICML '11, ACM, pp. 369–376.
60. [GNRM08] Garg S., Nam J. E., Ramakrishnan I., Mueller K.: Model-driven visual analytics. In Visual Analytics Science and Technology, 2008. VAST'08. IEEE Symposium on (2008), IEEE, pp. 19–26.
61. [GOB*10] Gehlenborg N., O'Donoghue S., Baliga N., Goesmann A., Hibbs M., Kitano H., Kohlbacher O., Neuweger H., Schneider R., Tenenbaum D., et al.: Visualization of omics data for systems biology. Nature Methods 7 (2010), S56–S68.
62. [GRF09] Green T. M., Ribarsky W., Fisher B.: Building and applying a human cognition model for visual analytics. Information Visualization 8, (2009), 1–13. 1. doi:10.1057/palgrave.ivs.2008.28.
63. [GRVE07] Grottel S., Reina G., Vrabec J., Ertl T.: Visual verification and analysis of cluster detection for molecular dynamics. IEEE Transactions on Visualization and Computer Graphics 13, 6 (2007), 1624–1631. doi:https://doi.org/10.1109/TVCG.2007.70614.
64. [GS04] Griffiths T. L., Steyvers M.: Finding scientific topics. Proceedings of the National Academy of Sciences 101, suppl 1 (2004), 5228–5235.
65. [GSFS06] Gray W. D., Sims C. R., Fu W.-T., Schoelles M. J.: The soft constraints hypothesis: A rational analysis approach to resource allocation for interactive behavior. Psychological Review 113, 3 (2006), 461.
66. [GWHR01] Gahegan M., Wachowicz M., Harrower M., Rhyne T.-M.: The integration of geographic visualization with knowledge discovery in databases and geocomputation. Cartography and Geographic Information Science 28, 1 (2001), 29–44.
67. [HB11] Hoober S., Berkman E.: Designing Mobile Interfaces. O'Reilly Media, Inc., 2011.
68. [HBGSS14] Hu Y., Boyd-Graber J., Satinoff B., Smith A.: Interactive topic modeling. Machine Learning 95, 3 (2014), 423–469. doi:10.1007/s10994-013-5413-0.
69. [HBM*13] Hu X., Bradel L., Maiti D., House L., North C., Leman S.: Semantics of directly manipulating spatializations. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2052–2059.
70. [Hee06] Heer J.: prefuse manual, 2006. URL: http://prefuse.org.
71. [HHE*13] Henry M. J., Hampton S., Endert A., Roberts I., Payne D.: MultiFacet: A Faceted Interface for Browsing Large Multimedia Collections. In 2013 IEEE International Symposium on Multimedia (ISM) (Dec. 2013), pp. 347–350. doi:10.1109/ISM.2013.66.
72. [HOG*12] Hossain M. S., Ojili P. K. R., Grimm C., Müller R., Watson L. T., Ramakrishnan N.: Scatter/gather clustering: Flexibly incorporating user feedback to steer clustering results. Visualization and Computer Graphics, IEEE Transactions on 18, 12 (2012), 2829–2838.
73. [Hor99] Horvitz E.: Principles of Mixed-initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA, 1999), CHI '99, ACM, pp. 159–166. doi:10.1145/302979.303030.
74. [HSCW13] Hadlak S., Schumann H., Cap C. H., Wollenberg T.: Supporting the visual analysis of dynamic networks by clustering associated temporal attributes. Visualization and Computer Graphics, IEEE Transactions on 19, 12 (2013), 2267–2276.
75. [i2]i2 Analyst's Notebook. URL: http://www.i2inc.com/products/analysts_notebook/.
76. [IHG13] Iwata T., Houlsby N., Ghahramani Z.: Active learning for interactive visualization. In Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2013 (Scottsdale, AZ, USA, April 2013), vol. 31 of JMLR Proceedings, JMLR.org, pp. 342–350. URL: http://dblp.uni-trier.de/db/conf/aistats/aistats2013.html#IwataHG13.
77. [IYU08] Iwata T., Yamada T., Ueda N.: Probabilistic latent semantic visualization: topic model for visualizing documents. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining (2008), ACM, pp. 363–371.
78. [JBS08] Jänicke H., Böttinger M., Scheuermann G.: Brushing of attribute clouds for the visualization of multivariate data. IEEE Transactions on Visualization and Computer Graphics (2008), 1459–1466.
79. [JJ09] Johansson S., Johansson J.: Interactive dimensionality reduction through user-defined combinations of quality metrics. Visualization and Computer Graphics, IEEE Transactions on 15, 6 (2009), 993–1000.
80. [JZF*09] Jeong D. H., Ziemkiewicz C., Fisher B., Ribarsky W., Chang R.: iPCA: An Interactive System for PCA-based Visual Analytics. Computer Graphics Forum 28, 3 (2009), 767–774. doi:10.1111/j.1467-8659.2009.01475.x.
81. [Kan12] Kandogan E.: Just-in-time annotation of clusters, outliers, and trends in point-based data visualizations. In Visual Analytics Science and Technology (VAST), 2012 IEEE Conference on (2012), IEEE, pp. 73–82.
82. [KAW*13] Kodagoda N., Attfield S., Wong B., Rooney C., Choudhury S.: Using interactive visual reasoning to support sense-making: Implications for design. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2217–2226.
83. [KBK11] Krstajić M., Bertini E., Keim D. A.: Cloudlines: Compact display of event episodes in multiple time-series. IEEE Transactions on Visualization and Computer Graphics 17, 12 (2011), 2432–2439.
84. [KCPE16] Kim H., Choo J., Park H., Endert A.: Interaxis: Steering scatterplot axes via observation-level interaction. IEEE Transactions on Visualization and Computer Graphics 22, 1 (Jan 2016), 131–140. doi:10.1109/TVCG.2015.2467615.
85. [Kei02] Keim D. A.: Information visualization and visual data mining. IEEE Transactions on Visualization and Computer Graphics, 8 (2002), 1–8. 1.
86. [KG90] Kendall M., Gibbons J. D.: Rank Correlation Methods. Oxford University Press, 1990.
87. [KGL*15] Klemm P., GlaÃer S., Lawonn K., Rak M., Völzke H., Hegenscheid K., Preim B.: Interactive visual analysis of lumbar back pain - what the lumbar spine tells about your life. In Proceedings of the 6th International Conference on Information Visualization Theory and Applications (VISIGRAPP 2015) (2015), pp. 85–92. doi:10.5220/0005235500850092.
88. [KK08] Kumar N., Kummamuru K.: Semisupervised clustering with metric learning using relative comparisons. IEEE Transactions on Knowledge and Data Engineering 20, 4 (April 2008), 496–503. doi:10.1109/TKDE.2007.190715.
89. [KKP05] Kumar N., Kummamuru K., Paranjpe D.: Semi-supervised clustering with metric learning using relative comparisons. In Fifth IEEE International Conference on Data Mining (ICDM'05) (Nov 2005). doi:10.1109/ICDM.2005.128.
90. [KKW*16] Kwon B. C., Kim H., Wall E., Choo J., Park H., Endert A.: Axisketcher: Interactive nonlinear axis mapping of visualizations through user drawings. IEEE Transactions on Visualization and Computer Graphics (2016).
91. [KLG*16] Klemm P., Lawonn K., Glaßer S., Niemann U., Hegenscheid K., VoÌlzke H., Preim B.: 3d regression heat map analysis of population study data. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2016), 81–90.
92. [KMH06a] Klein G., Moon B., Hoffman R.: Making Sense of Sensemaking 1: Alternative Perspectives. IEEE Intelligent Systems 21, 4 (2006), 70–73. doi:10.1109/MIS.2006.75.
93. [KMH06b] Klein G., Moon B., Hoffman R.: Making Sense of Sensemaking 2: A Macrocognitive Model. IEEE Intelligent Systems 21, 5 (2006), 88–92. doi:10.1109/MIS.2006.100.
94. [KMRV15] Keim D. A., Munzner T., Rossi F., Verleysen M. (Eds.):. Bridging Information Visualization with Machine Learning (Dagstuhl Seminar 15101) (Dagstuhl, Germany, 2015), vol. 5, Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik. URL: http://drops.dagstuhl.de/opus/volltexte/2015/5266, doi:https://doi.org/10.4230/DagRep.5.3.1.
95. [KMSZ06] Keim D. A., Mansmann F., Schneidewind J., Ziegler H.: Challenges in Visual Data Analysis. IEEE Computer Society. doi:10.1109/iv.2006.31.
96. [KP11] Kaski S., Peltonen J.: Dimensionality reduction for data visualization. IEEE Signal Processing Magazine 28, 2 (2011), 100–104. doi:10.1109/MSP.2010.940003.
97. [KPB14] Krause J., Perer A., Bertini E.: Infuse: interactive feature selection for predictive modeling of high dimensional data. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 1614–1623.
98. [KS11] Kang Y.-a., Stasko J.: Characterizing the intelligence analysis process: Informing visual analytics design through a longitudinal field study. In Visual Analytics Science and Technology (VAST), 2011 IEEE Conference on (2011), IEEE, pp. 21–30.
99. [KSF*08] Kerren A., Stasko J., Fekete J.-D., North C., Keim D., Andrienko G., Görg C., Kohlhammer J., Melançon G.: Visual Analytics: Definition, Process, and Challenges. In Information Visualization, vol. 4950 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg, 2008, pp. 154–175. doi:10.1007/978-3-540-70956-5_7.
100. [LD97] Landauer T. K., Dumais S. T.: A solution to plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review 104, 2 (1997), 211.
101. [LHE10] Lin C., He Y., Everson R.: A comparative study of bayesian models for unsupervised sentiment detection. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (2010), Association for computational linguistics, pp. 144–152.
102. [LHM*11] Leman S. C., House L., Maiti D., Endert A., North C.: A Bi-directional Visualization Pipeline that Enables Visual to Parametric Interation (V2PI). Tech. rep., 2011. FODAVA-10-41.
103. [LI57] Lonergan B. J., Insight A.: A study of human understanding. New York 298 (1957).
104. [LKT*14] Lu Y., Krüger R., Thom D., Wang F., Koch S., Ertl T., Maciejewski R.: Integrating predictive analytics and social media. In 2014 IEEE Conference on Visual Analytics Science and Technology (VAST) (Oct 2014), pp. 193–202. doi:10.1109/VAST.2014.7042495.
105. [LPP*06] Lee B., Plaisant C., Parr C. S., Fekete J.-D., Henry N.: Task taxonomy for graph visualization. In Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization (2006), ACM, pp. 1–5.
106. [LSP*10] Lex A., Streit M., Partl C., Kashofer K., Schmalstieg D.: Comparative analysis of multidimensional, quantitative data. IEEE Transactions on Visualization and Computer Graphics (Proceedings Visualization / Information Visualization 2010) 16 6 (2010), 1027–1035.
107. [LSS*12] Lex A., Streit M., Schulz H.-J., Partl C., Schmalstieg D., Park P. J., Gehlenborg N.: StratomeX: visual analysis of large-scale heterogeneous genomics data for cancer subtype characterization. Computer Graphics Forum (EuroVis '12) 31, 3 (2012), 1175?1184. doi:10.1111/j.1467-8659.2012.03110.x.
108. [LYK*12] Luo D., Yang J., Krstajic M., Ribarsky W., Keim D.: Eventriver: Visually exploring text collections with temporal references. Visualization and Computer Graphics, IEEE Transactions on 18, 1 (2012), 93–105.
109. [LZP*09] Liu S., Zhou M. X., Pan S., Qian W., Cai W., Lian X.: Interactive, topic-based visual text summarization and analysis. In Proceedings of the 18th ACM conference on Information and knowledge management (2009), ACM, pp. 543–552.
110. [MBD*11] May T., Bannach A., Davey J., Ruppert T., Kohlhammer J.: Guiding feature subset selection with an interactive visualization. In Visual Analytics Science and Technology (VAST), 2011 IEEE Conference on (2011), IEEE, pp. 111–120.
111. [MGJH08] Matković K., Gračanin D., Jelović M., Hauser H.: Interactive visual steering-rapid visual prototyping of a common rail injection system. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008), 1699–1706.
112. [MGS*14] Matkovic K., Gracanin D., Splechtna R., Jelovic M., Stehno B., Hauser H., Purgathofer W.: Visual analytics for complex engineering systems: Hybrid visual steering of simulation ensembles. Visualization and Computer Graphics, IEEE Transactions on 20, 12 (2014), 1803–1812.
113. [MJR*11] MacEachren A. M., Jaiswal A., Robinson A. C., Pezanowski S., Savelyev A., Mitra P., Zhang X., Blanford J.: Senseplace2: Geotwitter analytics support for situational awareness. In Visual Analytics Science and Technology (VAST), 2011 IEEE Conference on (2011), IEEE, pp. 181–190.
114. [MK08] May T., Kohlhammer J.: Towards closing the analysis gap: Visual generation of decision supporting schemes from raw data. In Computer Graphics Forum (2008), vol. 27, Wiley Online Library, pp. 911–918.
115. [MME*12] Malik A., Maciejewski R., Elmqvist N., Jang Y., Ebert D. S., Huang W.: A correlative analysis process in a visual analytics environment. In Visual Analytics Science and Technology (VAST), 2012 IEEE Conference on (2012), IEEE, pp. 33–42.
116. [MO04] Madeira S. C., Oliveira A. L.: Biclustering algorithms for biological data analysis: a survey. IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB) 1, 1 (2004), 24–45.
117. [MP13] Muhlbacher T., Piringer H.: A partition-based framework for building and validating regression models. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 1962–1971.
118. [Mur85] Murtagh F.: A survey of algorithms for contiguity-constrained clustering and related problems. The Computer Journal 28, 1 (1985), 82–88.
119. [NM13] Nam J., Mueller K.: Tripadvisorn-d: A tourism-inspired high-dimensional space exploration framework with overview and detail. Visualization and Computer Graphics, IEEE Transactions on 19, 2 (2013), 291–305.
120. [Nor86] Norman D. A.: Cognitive engineering. User centered system design: New perspectives on human-computer interaction 3161 (1986).
121. [Orm] Ormand C.: What constitutes a complex system? (http://serc.carleton.edu/nagtworkshops/complexsystems/introduction.html).
122. [OST*10] Oesterling P., Scheuermann G., Teresniak S., Heyer G., Koch S., Ertl T., Weber G. H.: Two-stage framework for a topology-based projection and visualization of classified document collections. In Visual Analytics Science and Technology (VAST), 2010 IEEE Symposium on (2010), IEEE, pp. 91–98.
123. [PBK10] Piringer H., Berger W., Krasser J.: Hypermoval: Interactive visual validation of regression models for real-time simulation. In Proceedings of the 12th Eurographics / IEEE - VGTC Conference on Visualization (Aire-la-Ville, Switzerland, Switzerland, 2010), EuroVis'10, Eurographics Association, pp. 983–992. URL: https://doi.org/10.1111/j.1467-8659.2009.01684.x, doi:10.1111/j.1467-8659.2009.01684.x.
124. [PC05] Pirolli P., Card S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of International Conference on Intelligence Analysis (2005), vol. 5, pp. 2–4.
125. [PES*06] Poulin B., Eisner R., Szafron D., Lu P., Greiner R., Wishart D. S., Fyshe A., Pearcy B., Macdonell C., Anvik J.: Visual explanation of evidence with additive classifiers. In Proceedings, The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, July 16-20, 2006, (Boston, Massachusetts, USA, July 2006), AAAI Press, pp. 1822–1829. URL: http://www.aaai.org/Library/AAAI/2006/aaai06-301.php.
126. [PSCO09] Pike W. A., Stasko J., Chang R., O'Connell T. A.: The science of interaction. Information Visualization 8, 4 (2009), 263–274.
127. [PSPM15] Paiva J. G. S., Schwartz W. R., Pedrini H., Minghim R.: An approach to supporting incremental visual data classification. IEEE Transactions on Visualization and Computer Graphics 21, 1 (2015), 4–17.
128. [PTH13] Porter R., Theiler J., Hush D.: Interactive machine learning in data exploitation. Computing in Science and Engineering 15, 5 (2013), 12–20. doi:http://doi.ieeecomputersociety.org/10.1109/MCSE.2013.74.
129. [PTRV13] Parulek J., Turkay C., Reuter N., Viola I.: Visual cavity analysis in molecular simulations. BMC Bioinformatics 14, 19 (2013), 1–15.
130. [PZS*15] Pérez, D., Zhang L., Schaefer M., Schreck T., Keim D., Díaz I.: Interactive feature space extension for multidimensional data projection. Neurocomputing 150, Part (2015), 611–626. URL: http://www.sciencedirect.com/science/article/pii/S0925231214012879, doi:https://doi.org/10.1016/j.neucom.2014.09.061.
131. [RAWC14] Rooney C., Attfield S., Wong B. W., Choudhury S.: Invisque as a tool for intelligence analysis: the construction of explanatory narratives. International Journal of Human-Computer Interaction 30, 9 (2014), 703–717.
132. [RECC10] Rose S., Engel D., Cramer N., Cowley W.: Automatic Keyword Extraction from Individual Documents. John Wiley & Sons, Ltd, pp. 1–20. doi:10.1002/9780470689646.ch1.
133. [RF16] Ribarsky W., Fisher B.: The human-computer system: Towards an operational model for problem-solving. In Hawaii International Conference on Systems Science (HICSS 2016) (2016).
134. [RK04] Rasmussen M., Karypis G.: gCLUTO–An Interactive Clustering, Visualization, and Analysis System., University of Minnesota, Department of Computer Science and Engineering, CSE. Tech. rep., UMN Technical Report: TR, 2004.
135. [RPN*08] Rinzivillo S., Pedreschi D., Nanni M., Giannotti F., Andrienko N., Andrienko G.: Visually driven analysis of movement data by progressive clustering. Information Visualization 7, 3 (2008), 225–239.
136. [RSE09] Rudolph S., Savikhin A., Ebert D. S.: Finvis: Applied visual analytics for personal financial planning. In Visual Analytics Science and Technology, 2009. VAST 2009. IEEE Symposium on (2009), IEEE, pp. 195–202.
137. [RSPC93] Russell D. M., Stefik M. J., Pirolli P., Card S. K.: The cost structure of sensemaking. In Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems (1993), ACM, pp. 269–276.
138. [SBTK08] Schreck T., Bernard J., Tekusova T., Kohlhammer J.: Visual cluster analysis of trajectory data with interactive Kohonen Maps. In IEEE Symposium on Visual Analytics Science and Technology, 2008. VAST'08 (2008), pp. 3–10.
139. [SDLC93] Spiegelhalter D. J., Dawid A. P., Lauritzen S. L., Cowell R. G.: Bayesian analysis in expert systems. Statistical science (1993), 219–247.
140. [SDMT16] Stahnke J., Dork M., MuÌller B., Thom A.: Probing projections: Interaction techniques for interpreting arrangements and errors of dimensionality reductions. Visualization and Computer Graphics, IEEE Transactions on 22, 1 (2016), 629–638.
141. [Set09] Settles B.: Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison, 2009.
142. [SGG*14] Streit M., Gratzl S., Gillhofer M., Mayr A., Mitterecker A., Hochreiter S.: Furby: fuzzy force-directed bicluster visualization. BMC bioinformatics 15, Suppl 6 (2014), S4.
143. [SGL08] Stasko J., Goerg C., Liu Z.: Jigsaw: supporting investigative analysis through interactive visualization. Information Visualization 7 (2008), 118–132. 2. doi:10.1145/1466620.1466622.
144. [SHA*01] Shipman F., Hsieh H., Airhart R., Maloor P., Moore J. M., Shah D.: Emergent Structure in Analytic Workspaces: Design and Use of the Visual Knowledge Builder. pp. 132–139.
145. [She00] Shearer C.: The crisp-dm model: the new blueprint for data mining. Journal of Data Warehousing 5, 4 (2000), 13–22.
146. [Shn83] Shneiderman B.: Direct Manipulation: A Step Beyond Programming Languages. Computer 16, 8 (1983), 57–69. doi:10.1109/MC.1983.1654471.
147. [SK10] Strumbelj E., Kononenko I.: An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research 11 (2010), 1–18. URL: http://doi.acm.org/10.1145/1756006.1756007, doi:10.1145/1756006.1756007.
148. [Sku02] Skupin A.: A Cartographic Approach to Visualizing Conference Abstracts. IEEE Computer Graphics and Applications 22 (2002), 50–58. doi:10.1109/38.974518.
149. [SM99] Shipman F., Marshall C.: Formality Considered Harmful: Experiences, Emerging Themes, and Directions on the Use of Formal Representations in Interactive Systems. Comput. Supported Coop. Work 8 (1999), 333–352. 4. doi:10.1023/A:1008716330212.
150. [SMNR16] Sun M., Mi P., North C., Ramakrishnan N.: Biset: Semantic edge bundling with biclusters for sensemaking. IEEE transactions on visualization and computer graphics 22, 1 (2016), 310–319.
151. [SNR14] Sun M., North C., Ramakrishnan N.: A five-level design framework for bicluster visualizations. IEEE transactions on visualization and computer graphics 20, 12 (2014), 1713–1722.
152. [SPG14] Stolper C. D., Perer A., Gotz D.: Progressive visual analytics: User-driven visual exploration of in-progress analytics. Visualization and Computer Graphics, IEEE Transactions on 20, 12 (2014), 1653–1662.
153. [SS02] Seo J., Shneiderman B.: Interactively exploring hierarchical clustering results. IEEE Computer 35, 7 (2002), 80–86.
154. [SS11] Satinover J., Sornette D.: Taming manias: On the origins, inevitability, prediction and regulation of bubbles and crashes, chapter of the book “governance and control of financial systems: A resilience engineering perspective,” published by Ashgate Publishing Group in their Resilience Engineering Perspectives series (2011).
155. [SSS*14] Sacha D., Stoffel A., Stoffel F., Kwon B. C., Ellis G., Keim D. A.: Knowledge generation model for visual analytics. Visualization and Computer Graphics, IEEE Transactions on 20, 12 (2014), 1604–1613.
156. [SWB97] Sarter N. B., Woods D. D., Billings C. E.: Automation surprises. Handbook of Human Factors and Ergonomics 2, (1997), 1926–1943.
157. [SZS*16] Sacha D., Zhang L., Sedlmair M., Lee J. A., Peltonen J., Weiskopf D., North S. C., Keim D. A.: Visual interaction with dimensionality reduction: A structured literature analysis. IEEE Transactions on Visualization and Computer Graphics (2016).
158. [TC05] Thomas J. J., Cook K. A.: Illuminating the Path: The Research and Development Agenda for Visual Analytics. IEEE Computer Society Press, 2005.
159. [TFH11] Turkay C., Filzmoser P., Hauser H.: Brushing dimensions-a dual visual analysis model for high-dimensional data. Visualization and Computer Graphics, IEEE Transactions on 17, 12 (2011), 2591–2599.
160. [TJHH14] Turkay C., Jeanquartier F., Holzinger A., Hauser H.: On computationally-enhanced visual analysis of heterogeneous data and its application in biomedical informatics. In Interactive Knowledge Discovery and Data Mining in Biomedical Informatics. Springer, 2014, pp. 117–140.
161. [TKBH17] Turkay C., Kaya E., Balcisoy S., Hauser H.: Designing progressive and interactive analytics processes for high-dimensional data analysis. IEEE Transactions on Visualization & Computer Graphics 23, 1 (2017), 131–140.
162. [TLLH12] Turkay C., Lundervold A., Lundervold A., Hauser H.: Representative factor generation for the interactive visual analysis of high-dimensional data. Visualization and Computer Graphics, IEEE Transactions on 18, 12 (2012), 2621–2630.
163. [TLS*14] Turkay C., Lex A., Streit M., Pfister H., Hauser H.: Characterizing cancer subtypes using dual analysis in caleydo stratomex. IEEE Computer Graphics and Applications 34, 2 (2014), 38–47. doi:http://doi.ieeecomputersociety.org/10.1109/MCG.2014.1.
164. [TPRH11a] Turkay C., Parulek J., Reuter N., Hauser H.: Integrating cluster formation and cluster evaluation in interactive visual analysis. In Proceedings of the 27th Spring Conference on Computer Graphics (2011), ACM, pp. 77–86.
165. [TPRH11b] Turkay C., Parulek J., Reuter N., Hauser H.: Interactive visual analysis of temporal cluster structures. Computer Graphics Forum 30, 3 (2011), 711–720. URL: https://doi.org/10.1111/j.1467-8659.2011.01920.x.
166. [Van05] Van Wijk, J. J.: The value of visualization. In 16th IEEE Visualization 2005 (VIS 2005) (2005), IEEE Computer Society, p. 11. doi:10.1109/VIS.2005.102.
167. [vdEvW11] van den Elzen S., van Wijk J. J.: Baobabview: Interactive construction and analysis of decision trees. In Visual Analytics Science and Technology (VAST), 2011 IEEE Conference on (2011), IEEE, pp. 151–160.
168. [vdMW12] van der Maaten, L., Weinberger K.: Stochastic triplet embedding. In 2012 IEEE International Workshop on Machine Learning for Signal Processing (Sept 2012), pp. 1–6. doi:10.1109/MLSP.2012.6349720.
169. [Ves99] Vesanto J.: SOM-based data visualization methods. Intelligent Data Analysis 3, 2 (1999), 111–126. doi:10.3233/IDA-1999-3203.
170. [VL13] Verleysen M., Lee J. A.: Nonlinear Dimensionality Reduction for Visualization. In Neural Information Processing (2013), Springer, pp. 617–622.
171. [Wei95] Weick K. E.: Sensemaking in organizations, vol. 3. Sage, 1995.
172. [WH00] Wirth R., Hipp J.: Crisp-dm: Towards a standard process model for data mining. In Proceedings of the 4th International Conference on the Practical Applications of Knowledge Discovery and Data Mining (2000), Citeseer, pp. 29–39.
173. [WKKB15] Wilber M. J., Kwak I. S., Kriegman D., Belongie S.: Learning concept embeddings with combined human-machine expertise. In 2015 IEEE International Conference on Computer Vision (ICCV) (Dec 2015), pp. 981–989. doi:10.1109/ICCV.2015.118.
174. [WLS*10] Wei F., Liu S., Song Y., Pan S., Zhou M. X., Qian W., Shi L., Tan L., Zhang Q.: Tiara: a visual exploratory text analytic system. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (2010), ACM, pp. 153–162.
175. [WM04] Williams M., Munzner T.: Steerable, progressive multidimensional scaling. In Proceedings of the IEEE Symposium on Information Visualization (Washington, DC, USA, 2004), IEEE Computer Society, pp. 57–64.
176. [Won14] Wong B.: How analysts think (?): Early observations. In Intelligence and Security Informatics Conference (JISIC), 2014 IEEE Joint (2014), IEEE, pp. 296–299.
177. [WSO05] Weick K. E., Sutcliffe K. M., Obstfeld D.: Organizing and the process of sensemaking. Organization Science 16, 4 (2005), 409–421.
178. [WSP*06] Wright W., Schroh D., Proulx P., Skaburskis A., Cort B.: The Sandbox for analysis: concepts and methods. ACM, pp. 801–810. doi:10.1145/1124772.1124890.
179. [WTP*99] Wise J. A., Thomas J. J., Pennock K., Lantrip D., Pottier M., Schur A., Crow V.: Visualizing the non-visual: spatial analysis and interaction with information for text documents. Morgan Kaufmann Publishers Inc., pp. 442–450.
180. [WV12] Wong B. W., Varga M.: Black holes, keyholes and brown worms: Challenges in sense making. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (2012), vol. 56, SAGE Publications, pp. 287–291.
181. [YaKSJ07] Yi J. S., ah Kang Y., Stasko J. T., Jacko J. A.: Toward a deeper understanding of the role of interaction in information visualization. Visualization and Computer Graphics, IEEE Transactions on 13, 6 (2007), 1224–1231.
182. [YKJ16] Yeon H., Kim S., Jang Y.: Predictive visual analytics of event evolution for user-created context. Journal of Visualization (2016), 1–16.
183. [YNM*13] Younesy H., Nielsen C. B., Möller T., Alder O., Cullum R., Lorincz M. C., Karimi M. M., Jones S. J.: An interactive analysis and exploration tool for epigenomic data. In Computer Graphics Forum (2013), vol. 32, Wiley Online Library, pp. 91–100.
184. [ZC07] Zhu W., Chen C.: Storylines: Visual exploration and analysis in latent semantic spaces. Computers & Graphics 31, 3 (2007), 338–349.
185. [ZJGK10] Ziegler H., Jenny M., Gruse T., Keim D. A.: Visual market sector analysis for financial time series data. In Visual Analytics Science and Technology (VAST), 2010 IEEE Symposium on (2010), IEEE, pp. 83–90.
186. [ZNK08] Ziegler H., Nietzschmann T., Keim D. A.: Visual analytics on the financial market: Pixel-based analysis and comparison of long-term investments. In Information Visualisation, 2008. IV'08. 12th International Conference (2008), IEEE, pp. 287–295.