My computer is an honor student - But how intelligent is it? Standardized tests as a measure of AI

AI Magazine

Volumn 37, Issue 1, 2016, Pages 5-12

  Clark, Peter ^a   Etzioni, Oren ^a  

^a Allen Institute for Artificial Intelligence   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

HIGH LEVEL LANGUAGES;

CHALLENGE PROBLEMS; LANGUAGE UNDERSTANDING; MACHINE PERFORMANCE; PRACTICAL TESTS; STANDARDIZED TESTS; STATE OF THE ART; TURING TESTS; WORLD MODEL;

MODELING LANGUAGES;

EID: 85011952375     PISSN: 07384602     EISSN: None     Source Type: Journal    
DOI: 10.1609/aimag.v37i1.2636     Document Type: Article

References (28)

1. Bayer, S.; Damianos, L.; Doran, C.; Ferro, L.; Fish, R.; Hirschman, L.; Mani, I.; Riek, L.; and Oshika, B. 2005. Selected Grand Challenges in Cognitive Science, MITRE Technical Report 05-1218. Bedford MA: The MITRE Corporation.
2. Bringsjord, S. 2011. Psychometric Artificial Intelligence. Journal of Experimental and Theoretical Artificial Intelligence (JETAI) 23(3): 271-277.
3. Bringsjord, S., and Schimanski, B. 2003. What Is Artificial Intelligence? Psychometric AI as an Answer. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, 887-893. San Francisco: Morgan Kaufmann Publishers. dx.doi.org/10.1080/0952813X.2010.502314
4. Clark, P.; Harrison, P.; and Balasubramanian, N. 2013. A Study of the Knowledge Base Requirements for Passing an Elementary Science Test. In AKBC'13: Proceedings of the 2013 Workshop on Automated Knowledge Base Construction. New York: Association for Computing Machinery. dx.doi.org/10.1145/2509558.2509565
5. Clark, P.; Etzioni, O.; Khashabi, D.; Khot, T.; Sabharwal, A.; Tafjord, O.; Turney, P. 2016. Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions. In Proceedings of the Thirtieth Conference of the Association for the Advancement of Artificial Intelligence. Menlo Park, CA: AAAI Press.
6. Davis, E. 2014. The Limitations of Standardized Science Tests as Benchmarks for AI Research. Technical Report, New York University. arXiv Preprint arXiv:1411.1629. Ithaca, NY: Cornell University Library
7. Davis, E., and Marcus, G. 2015. Common-sense Reasoning and Commonsense Knowledge in Artificial Intelligence. Communications of the ACM 58(9): 92-103. dx.doi.org/10.1145/2701413
8. Fujita, A.; Kameda, A.; Kawazoe, A.; and Miyao, Y. 2014. Overview of Todai Robot Project and Evaluation Framework of its NLP-based Problem Solving. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2014). Paris: European Language Resources Association.
9. Hayes, P., and Ford, K. 1995. Turing Test Considered Harmful. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann Publishers.
10. Hosseini, M.; Hajishirzi, H.; Etzioni, O.; and Kushman, N. 2014. Learning to Solve Arithmetic Word Problems with Verb Categorization. In EMNLP 2014: Proceedings of the Empirical Methods in Natural Language Processing. Stroudsburg, PA: Association for Computational Linguistics. dx.doi.org/10.3115/v1/D14-1058
11. Khot, T.; Balasubramanian, N.; Gribkoff, E.; Sabharwal, A.; Clark P.; and Etzioni, O. 2015. Exploring Markov Logic Networks for Question Answering. In EMNLP 2015: Proceedings of the Empirical Methods in Natural Language Processing. Stroudsburg, PA: Association for Computational Linguistics. dx.doi.org/10.18653/v1/D15-1080
12. Koncel-Kedziorski, R.; Hajishirzi, H.; Sabharwal, A.; Ang, S. D.; and Etzioni, O. 2015. Parsing Algebraic Word Problems into Equations. Transactions of the Association for Computational Linguistics, Volume 3 (2015).
13. Kushman, N.; Artzi, Y.; Zettlemoyer, L.; and Barzilay, R. 2014. Learning to Automatically Solve Algebra Word Problems. In EMNLP 2014: Proceedings of the Empirical Methods in Natural Language Processing Conference. Stroudsburg, PA: Association for Computational Linguistics. dx.doi.org/10.3115/v1/p14-1026
14. Landauer, T. K., and Dumais, S. T. 1997. A Solution to Plato's Problem: The Latent Semantic Analysis Theory of the Acquisition, Induction, and Representation of Knowledge. Psychological Review 104(2): 211-240. dx.doi.org/10.1037/0033-295X.104.2.211
15. Legg, S., and Hutter, M. A. 2007. Collection of Definitions of Intelligence. In Advances in Artificial General Intelligence: Concepts, Architectures, and Algorithms. Frontiers in Artificial Intelligence and Applications Volume 157. Amsterdam, The Netherlands: IOS Press.
16. Levesque, H.; Davis, E.; and Morgenstern, L. 2012. The Winograd Schema Challenge. In Principles of Knowledge Representation and Reasoning: Proceedings of the Thirteenth International Conference (KR2012), 552-561. Palo Alto: AAAI Press.
17. Li, Y., and Clark, P. 2015. Answering Elementary Science Questions via Coherent Scene Extraction from Knowledge Graphs. In EMNLP 2015: Proceedings of the Empirical Methods in Natural Language Processing Conference. Stroudsburg, PA: Association for Computational Linguistics. dx.doi.org/10.18653/v1/D15-1236
18. Mohammad, S. M.; Dorr, B. J.; Hirst, G.; and Turney, P. D. 2013. Computing Lexical Contrast. Computational Linguistics 39(3): 555-590. dx.doi.org/10.1162/COLI-a-00143
19. New York State Education Department. 2014. The Grade 4 Elementary-Level Science Test. Albany, NY: University of the State of New York.
20. Reddy, R. 1996. To Dream the Possible Dream. Communications of the ACM 39(5): 105-112. dx.doi.org/10.1145/229459.233436
21. Richardson, M.; Burges, C.; and Renshaw, E. 2013. MCTest: A Challenge Dataset for the Machine Comprehension of Text. In EMNLP 2013: Proceedings of the Empirical Methods in Natural Language Processing Conference. Stroudsburg, PA: Association for Computational Linguistics. dx.doi.org/10.18653/v1/D15-1171
22. Seo, M.; Hajishirzi, H.; Farhadi, A.; and Etzioni, O. 2014. Diagram Understanding in Geometry Questions. In Proceedings of the Twenty-Eighth Conference on Artificial Intelligence AAAI 2014. Palo Alto, CA: AAAI Press.
23. Seo, M.; Hajishirzi, H.; Farhadi, A.; Etzioni, O.; and Malcolm, C. 2015. Solving Geometry Problems: Combining Text and Diagram Interpretation. In EMNLP 2015: Proceedings of the Empirical Methods in Natural Language Processing Conference. Stroudsburg, PA: Association for Computational Linguistics.
24. Strickland, E. 2013. Can an AI Get into the University of Tokyo? IEEE Spectrum 21 August. dx.doi.org/10.1109/mspec.2013.6587172
25. Texas Education Agency. 2014. Texas Assessment of Knowledge and Skills. Austin, TX: State of Texas Education Agency.
26. Turing, A. 1950. Computing Machinery and Intelligence. Mind 59(236): 433-460. dx.doi.org/10.1093/mind/LIX.236.433
27. Turney P. D. 2006. Similarity of Semantic Relations. Computational Linguistics 32(3): 379-416. dx.doi.org/10.1162/coli.2006.32.3.379
28. Weston, J.; Bordes, A.; Chopra, S.; Mikolov, T; and Rush, A. 2015. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. arXiv Preprint arXiv:1502. 05698v6. Ithaca, NY: Cornell University Library.