As the size of NLP models increases into the hundreds of billions of parameters, so does the importance of being able to create more compact representations of these models. Knowledge distillation has successfully enabled this wherein one instance 96% of the teacher’s performance was retained in a 7x smaller model . However, knowledge distillation is still considered an afterthought when designing the teacher models which could reduce the effectiveness, leaving potential performance improvements for the student on the table. Furthermore, the difficulties in fine-tuning small student models after the initial distillation, without degrading their performance, requires us to both pre-train and fine-tune the teachers on the tasks we want the student to be able to perform. Training a student model through knowledge distillation will, therefore, require more training compared to only training the teacher, which limits the benefits of a student model to inference-time. What would be possible if, instead, knowledge distillation was put front and center during the design and training of the teacher model? Could we design and successfully train a model that is supposed to be distilled, and could the distilled version successfully be fine-tuned on any downstream task? These are some of the questions addressed in MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices which we summarize in this article. 链接地址： https://medium.com/dair-ai/mobilebert-a-task-agnostic-bert-for-resource-limited-devices-%EF%B8%8F-c4fe3ee3815d