It’s hard to keep up with the rapid progress of natural language processing (NLP). To organize my thoughts better, I took some time to review my notes, compare the various papers, and sort them chronologically. This helped in my understanding of how NLP (and its building blocks) has evolved over time. To reinforce my learning, I’m writing this summary of the broad strokes, including brief explanations of how models work and some details (e.g., corpora, ablation studies). Here, we’ll see how NLP has progressed from 1985 till now: Sequential models: RNN (1985), LSTM (1997), GRU (2014) Word embeddings: Word2vec (2013), GloVe (2014), FastText (2016) Word embeddings with context: ELMo (2018) Attention: Transformer (2017) Pre-training: ULMFiT (2017), GPT (2017) Combining the above: BERT (2018) Improving BERT: DistilBERT, ALBERT, RoBERTa, XLNet (2019); Big Bird, Multilingual embeddings (2020) Everything is text-to-text: T5 (2019) (Did I miss anything important or oversimplify? Any errors? Please reach out with suggestions and I’ll update. Thank you!) 链接地址：https://eugeneyan.com/writing/nlp-supervised-learning-survey/