Deep neural language models like BERT have recently demonstrated a fascinating level of understanding of human language. Multilingual versions of these models, like Multilingual BERT (mBERT), are able to understand a large number of languages simultaneously. To what extent do these models share what they’ve learned between languages? Focusing on the syntax, or grammatical structure, of these languages, we show that Multilingual BERT is able to learn a general syntactic structure applicable to a variety of natural languages. Additionally, we find evidence that mBERT learns cross-lingual syntactic categories like “subject” and “adverb”—categories that largely agree with traditional linguistic concepts of syntax! Our results imply that simply by reading a large amount of text, mBERT is able to represent syntax—something fundamental to understanding language—in a way that seems to apply across many of the languages it comprehends. More specifically, we present the following: We apply the structural probe method of Hewitt and Manning (2019) to 10 languages, finding syntactic subspaces in a multilingual setting. Through zero-shot transfer experiments, we demonstrate that mBERT represents some syntactic features in syntactic subspaces that overlap between languages. Through an unsupervised method, we find that mBERT natively represents dependency clusters that largely overlap with the UD standard. Our results are presented in the forthcoming ACL 2020 paper, Finding Universal Grammatical Relations in Multilingual BERT . This post draws from the paper, which is joint work with John Hewitt and Chris Manning. You can also find the code here . If you’d like to skip the background and jump to the discussion of our methods, click here . Otherwise, read on! 链接地址： http://ai.stanford.edu/blog/finding-crosslingual-syntax/?u=1402400261&m=4511636251815795&cu=1968044071