Colorless green recurrent networks dream hierarchically

NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference

Volumn 1, Issue , 2018, Pages 1195-1205

  Gulordava, Kristina ^a   Bojanowski, Piotr ^b   Grave, Edouard ^b   Linzen, Tal ^c   Baroni, Marco ^b  

^a UNIVERSITY OF GENEVA   (Switzerland)

^b FACEBOOK AI RESEARCH   (United States)

^c JOHNS HOPKINS UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; MODELING LANGUAGES; SEMANTICS; SYNTACTICS;

HUMAN PERFORMANCE; LANGUAGE MODEL; LINGUISTIC PROCESSING; MODEL PERFORMANCE; NON-TRIVIAL; RECURRENT NETWORKS; RECURRENT NEURAL NETWORK (RNNS); SYNTACTIC STRUCTURE;

RECURRENT NEURAL NETWORKS;

EID: 85060078907     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (41)

1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR Conference Track. Toulon, France. Published online: https://openreview.net/group?id=ICLR.cc/2017/conference.
2. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of ACL. Vancouver, Canada, pages 861-872.
3. Jean-Philippe Bernardy and Shalom Lappin. 2017. Using deep neural networks to learn syntactic agreement. Linguistic Issues in Language Technology 15(2):1-15.
4. Barry Blake. 2001. Case. MIT Press, Cambridge, MA.
5. Kathryn Bock and Carol Miller. 1991. Broken agreement. Cognitive Psychology 23(1):45-93.
6. Bo Cartling. 2008. On the implicit acquisition of a context-free grammar by a simple recurrent neural network. Neurocomputing 71:1527-1537.
7. Ciprian Chelba, Mohammad Norouzi, and Samy Bengio. 2017. N-gram language modeling using recurrent neural network estimation. arXiv preprint arXiv:1703.10724 .
8. Noam Chomsky. 1957. Syntactic Structures. Mouton, Berlin, Germany.
9. Morten Christiansen and Nick Chater. 1999. Toward a connectionist model of recursion in human linguistic performance. Cognitive Science 23(2):157-205.
10. James Cross and Liang Huang. 2016. Incremental parsing with minimal features using bi-directional LSTM. In Proceedings of ACL (Short Papers). Berlin, Germany, pages 32-37.
11. Peter Culicover and Paul Postal, editors. 2001. Parasitic gaps. MIT Press, Cambridge, MA.
12. Jeffrey Elman. 1990. Finding structure in time. Cognitive Science 14:179-211.
13. Jeffrey Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning 7:195-225.
14. Jeffrey Elman. 1993. Learning and development in neural networks: The importance of starting small. Cognition 48:71-99.
15. Emile Enguehard Yoav Goldberg Tal Linzen 2017 Exploring the syntactic abilities of RNNs with multi-Task learning Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017 3-14
16. Martin Everaert, Marinus Huybregts, Noam Chomsky, Robert Berwick, and Johan Bolhuis. 2015. Structures, not strings: Linguistics as part of the cognitive sciences. Trends in Cognitive Sciences 19(12):729-743.
17. Marcello Federico, Nicola Bertoldi, and Mauro Cettolo. 2008. Irstlm: An open source toolkit for handling large scale language models. In Ninth Annual Conference of the International Speech Communication Association. pages 1618-1621.
18. Angela D. Friederici, Martin Meyer, and D. Yves Von Cramon. 2000. Auditory language comprehension: An event-related fMRI study on the processing of syntactic and lexical information. Brain and Language 74(2):289-300.
19. Felix Gers and Jürgen Schmidhuber. 2001. LSTM recurrent networks learn simple context-free and context-sensitive languages. IEEE Transactions on Neural Networks 12(6):1333-1340.
20. Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks. Springer, Berlin.
21. Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, pages 187-197.
22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-Term memory. Neural Computation 9(8):1735-178-.
23. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2017. Visualisation and diagnostic classifiers reveal how recurrent and recursive neural networks process hierarchical structure. http://arxiv.org/abs/1711.10203.
24. Matt A. Johnson and Adele E. Goldberg. 2013. Evidence for automatic accessing of constructional meaning: Jabberwocky sentences prime associated verbs. Language and Cognitive Processes 28(10):1439-1452.
25. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics 4:313-327.
26. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Acoustics, Speech, and Signal Processing, 1995. ICASSP-95., 1995 International Conference on. IEEE, volume 1, pages 181-184.
27. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics 4:521-535.
28. Heidi Lorimor, Kathryn Bock, Ekaterina Zalkind, Alina Sheyman, and Robert Beard. 2008. Agreement and attraction in Russian. Language and Cognitive Processes 23(6):769-799.
29. Tomas Mikolov. 2012. Statistical language models based on neural networks. Dissertation, Brno University of Technology.
30. Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of INTERSPEECH. Makuhari, Japan, pages 1045-1048.
31. Tomas Mikolov, Wen-Tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of NAACL. Atlanta, Georgia, pages 746-751.
32. Andrea Moro, Marco Tettamanti, Daniela Perani, Caterina Donati, Stefano F Cappa, and Ferruccio Fazio. 2001. Syntax and the brain: disentangling grammar by selective anomalies. Neuroimage 13(1):110-118.
33. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association (ELRA), Paris, France.
34. Fernando Pereira. 2000. Formal grammar and information theory: Together again? Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 358(1769):1239-1253.
35. Douglas Rohde and David Plaut. 1997. Simple recurrent networks and natural language: How important is starting small? In Proceedings of CogSci. Stanford, CA, pages 656-661.
36. Helmut Schmid. 1995. Improvements in part-ofspeech tagging with an application to German. In Proceedings of the EACL-SIGDAT Workshop. Dublin, Ireland.
37. Rico Sennrich. 2017. How grammatical is characterlevel neural machine translation? assessing MT quality with contrastive translation pairs. In Proceedings of EACL (Short Papers). Valencia, Spain, pages 376-382.
38. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of EMNLP. Austin, Texas, pages 1526-1534.
39. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. http://arxiv.org/abs/1609.08144.
40. Dani Yogatama, Yishu Miao, Gabor Melis, Wang Ling, Adhiguna Kuncoro, Chris Dyer, and Phil Blunsom. 2018. Memory architectures in recurrent neural network language models. In International Conference on Learning Representations. https://openreview.net/forum?id=SkFqf0lAZ.
41. Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. 2017. Dependency parsing as head selection. In Proceedings of EACL. Valencia, Spain, pages 665-676.