While Computer Vision is making amazing progress on self-supervised learning only in the last few years, self-supervised learning has been a first-class citizen in NLP research for quite a while. Language Models have existed since the 90’s even before the phrase “self-supervised learning” was termed. The Word2Vec paper from 2013 popularized this paradigm and the field has rapidly progressed applying these self-supervised methods across many problems.
At the core of these self-supervised methods lies a framing called “pretext task” that allows us to use the data itself to generate labels and use supervised methods to solve unsupervised problems. These are also referred to as “auxiliary task” or “pre-training task“. The representations learned by performing this task can be used as a starting point for our downstream supervised tasks.
1. Center Word Prediction
2. Neighbor Word Prediction
3. Neighbor Sentence Prediction
4. Auto-regressive Language Modeling
5. Masked Language Modeling
6. Next Sentence Prediction
7. Sentence Order Prediction
8. Sentence Permutation
9. Document Rotation
10. Emoji Prediction
References
Ryan Kiros, et al. “Skip-Thought Vectors”
Tomas Mikolov, et al. “Efficient Estimation of Word Representations in Vector Space”
Alec Radford, et al. “Improving Language Understanding by Generative Pre-Training”
Jacob Devlin, et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”
Yinhan Liu, et al. “RoBERTa: A Robustly Optimized BERT Pretraining Approach”
Zhenzhong Lan, et al. “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations”
Mike Lewis, et al. “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension”
Bjarke Felbo, et al. “Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm”
链接地址：https://amitness.com/2020/05/self-supervised-learning-nlp/