User studies of knowledge acquisition tools: Methodology and lessons learned

Journal of Experimental and Theoretical Artificial Intelligence

Volumn 13, Issue 4, 2001, Pages 359-378

  Tallis, Marcelo ^a,b   Kim, Jihie ^b   Gil, Yolanda ^b  

^a Teknowledge Corporation   (United States)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0035563228     PISSN: 0952813X     EISSN: None     Source Type: Journal    
DOI: 10.1080/09528130110100261     Document Type: Article

References (44)

1. Basili, V., Selby, R. W., and Hutchens, D. H., 1986, Experimentation in software engineering. IEEE Transactions in Software Engineering, SE-12(7): 733-743.
2. Birmingham, W., and Klinker, G., 1993, Knowledge acquisition tools with explicit problem-solving methods. The Knowledge Engineering Review, 8(1): 5-25.
3. Blake, C., and Merz, C., 1998, UCI repository of machine learning databases. University of California, Irvine, Dept. of Information and Computer Sciences, http://www.ics.uci.edu/~mlearn/ MLRepository.html
4. Blythe, J., Kim, J., Ramachandran, S., and Gil, Y., 2001, An integrated environment for knowledge acquisition. In Proceedings of the Intelligent User Interface Conference (IUI-2001), Sante Fe, NM.
5. Calfee, R. C., 1975, Human Experimental Psychology (New York: Holt, Rinehart and Winston).
6. Chi, M., Glaser, R., and Farr, M., 1988, The Nature of Expertise (Lawrence Erlbaum).
7. Corbridge, C., Major, N. P., and Shadbolt, N. R., 1995, Models exposed: an empirical study. In Proceedings of the Ninth Knowledge-Acquisition for Knowledge-Based Systems Workshop (KAW-95), Banff, Alberta, pp. 13-1-13-21.
8. Davis, R., 1979, Interactive transfer of expertise: acquisition of new inference rules. Artificial Intelligence, 12: 121-157.
9. Eriksson, H., Shahar, Y., Tu, S. W., Puerta, A. R., and Musen, M., 1995, Task modeling with reusable problem-solving methods. Artificial Intelligence, 79: 293-326.
10. Estlin, T. A., 1998, Using Multi-Strategy Learning to Improve Planning Efficiency and Quality. PhD thesis, University of Texas at Austin, Computer Science Department, Austin, TX.
11. Fenton, N. E., and Pfleeger, S. L., 1997, Software Metrics: A Rigorous and Practical Approach (Boston, MA: International Thomson Computer Press).
12. Gil, Y., 1991, A specification of process planning for PRODIGY. Technical Report CMU-CS-91-179, Computer Science Department, Carnegie-Mellon University.
13. Gil, Y., 1992, Acquiring Domain Knowledge for Planning by Experimentation. PhD thesis, Carnegie Mellon University, School of Computer Science, Pittsburgh, PA.
14. Gil, Y., 1994, Knowledge refinement in a reflective architecture. In Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI-94), Seattle, WA, pp. 520-526.
15. Gil, Y., and Linster, M., 1995, Dimensions to analyze applications. In Proceedings of the Ninth Knowledge-Acquisition for Knowledge-Based Systems Workshop (KAW-95), Banff, Alberta, pp. 16-1-16-1 5.
16. Gil, Y., and Melz, E., 1996, Explicit representations of problem-solving strategies to support knowledge acquisition. In Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96), Portland, OR, pp. 469-476.
17. Gil, Y., and Tallis, M., 1997, A script-based approach to modifying knowledge-based systems. In Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97), Providence, RI, pp. 377-383.
18. Hayes-Roth, F., Waterman, D. A., and Lenat, D. B., 1983, Building Expert Systems (Reading, MA, Addison-Wesley Publishing Company).
19. Joseph, R. L., 1992, Knowledge Acquisition for Visually Oriented Planning. PhD thesis, Carnegie Mellon University, School of Computer Science, Pittsburgh, PA.
20. Kim, J., and Gil, Y., 1999, Deriving expectations to guide knowledge base creation. In Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI-99), Orlando, FL, pp. 235-241.
21. Kim, J., and Gil, Y., 2000a, Acquiring problem-solving knowledge from end users: putting interdependency models to the test. In Proceedings of the Seventeenth National Conference on Artificial Intelligence (AAAI-2000), Austin, TX, pp. 223-229.
22. Kim, J., and Gil, Y., 2000b, User studies of an interdependency-based interface for acquiring problem-solving knowledge. In Proceedings of the Intelligent User Interface Conference (IUI-2000), New Orleans, LA, pp. 165-168.
23. Kim, J., and Rosenbloom, P. S., 1996, Learning efficient rules by maintaining the explanation structure. In Proceedings of the Thirteenth National conference on Artificial Intelligence (AAAI-96), Portland, OR, pp. 763-770.
24. Linster, M., 1994, Sisyphus '91/92: models of problem solving. International Journal of Human-Computer Studies, 40(2): 189-192.
25. Marcus, S., and McDermott, J., 1989, SALT: a knowledge acquisition language for propose-and-revise systems. Artificial Intelligence, 39 (1): 1-37.
26. Maulsby, D., Greenberg, S., and Mander, R., 1993, Prototyping an intelligent agent through Wizard of Oz. In Proceedings of INTERCHI-93, pp. 277-285.
27. McDermott, D., 2000, The 1998 AI planning systems competition. AI Magazine 21(2): pp. 35-55.
28. Murray, K. S., 1995, Learning as Knowledge Integration. PhD thesis, University of Texas at Austin, Computer Science Department, Austin, TX.
29. Olson, G. M., and Moran, T. P., 1998, Special issue on experimental comparisons of usability evaluation methods. Human-Computer Interaction, 13(3) 199-201.
30. Pérez, A. M., and Carbonell, J. G., 1994, Control knowledge to improve plan quality. In Proceedings of the Second International Conference on AI Planning Systems (AIPS-94), Chicago, IL, pp. 323-328.
31. Quinlan, J. R., 1993, C4.5: Programs for machine learning (San Mateo, CA: Morgan Kaufmann).
32. Rombach, H. D., Basili, V. R., and Selby, R. W., (eds) 1992, Proceedings of the International Workshop on Experimental Software Engineering Issues: Critical Assessment and Future Directions., Lecture Notes in Computer Science series, 1993 (Dagstuhl Castle, Germany: Springer Verlag).
33. Runkel, J. T., and Birmingham, W. P., 1995, Knowledge acquisition in the small: building knowledge-acquisition tools from pieces. Knowledge Acquisition, 5(2): 221-243.
34. Schreiber, A. T., and Birmingham, W. P., 1996, The Sisyphus-VT initiative. International Journal of Human-Computer Studies, 44: 275-280.
35. Self, J., 1993, Special issue on evaluation. Journal of Artificial Intelligence in Education, 4(2/3): 129-413.
36. Shadbolt, N., O'Hara, K., and Crow, L., 1999, The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions. International Journal of Human-Computer Studies, 51(4): 729-755.
37. Swartout, W., and Gil, Y., 1995, EXPECT: explicit representations for flexible acquisition. In Proceedings of the Ninth Knowledge-Acquisition for Knowledge-Based Systems Workshop (KAW-95), Banff, Alberta, pp. 34-1-34-19.
38. Tallis, M., 2000, A Script-Based Approach to Modifying Knowledge-Based Systems. PhD thesis, University of Southern California, Computer Science Department, Los Angeles, CA.
39. Tallis, M., and Gil, Y., 1999, Designing scripts to guide users in modifying knowledge-based systems. In Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI-99), Orlando, FL, pp. 242-249.
40. Tambe, M., and Rosenbloom, P. S., 1990, A framework for investigating production system formulations with polynomially bounded match. In Proceedings of the Eighth National Conference on Artificial Intelligence (AAAI-90), Boston, MA, pp. 693-700.
41. Webb, G. L, Wells, J., and Zheng, Z., 1999, An experimental evaluation of integrating machine learning with knowledge acquisition. Machine Learning, 35: 5-23.
42. Yost, G. R., 1992, TAQL: A Problem Space Tool for Expert System Development. PhD thesis, Carnegie Mellon University, School of Computer Science, Pittsburgh, PA.
43. Yost, G. R., 1993, Knowledge acquisition in soar. IEEE Expert, 8(3): 26-34.
44. Zelkowitz, M. V., and Wallace, D., 1998, Experimental models for validating computer technology. IEEE Computer, 735-744.