从 GRU 到 Transformer

Attention-based networks have been shown to outperform recurrent neural networks and its variants for various deep learning tasks including Machine Translation, Speech, and even Visio-Linguistic tasks. The Transformer [Vaswani et. al., 2017] is a model, at the fore-front of using only self-attention in its architecture, avoiding recurrence and enabling parallel computations.

To understand how the self-attention mechanism is applied in Transformers, it might be intuitive from a mathematical perspective to build-up step-by-step from what is known, i.e. Recurrent Neural Networks such as LSTMs or GRUs to a self-attention network such as Transformers. Blog posts such as Jalammar, The Annotated Transformer, Vandergoten have attacked the explanation of Transformers from different perspectives but I believe this article will give another perspective and help engineers and researchers understand Self-Attention better, as I did.

For a beautiful explanation of everything Attention, check out Lilianweng post on Attention

Here’s what we will cover:

1. Introduction

2. Gated Recurrent Neural Networks

3. What are these shortcut connections

1. Gated Recurrent Units to Causal Attention

2. Let’s free the dependent weights

3. Let’s free up candidate vectors

4. Let’s separate Keys and Values

5. Let’s have multiple attention heads

1. Gated Recurrent Units to Non-Causal Attention

2. Let’s look at the entire input sequence

3. Let’s give a sense of position to the attention mechanism

4. Let’s use Non-Linear Attention

1. Full Self-Attention Layer

2. Conclusion

This article is based on a lecture given by Kyunghyun Cho at AMMI, 2020. Special thanks to him and his team for a beautiful NLP course.

链接地址：https://ogunlao.github.io/blog/2020/06/12/from_gru_to_transformer.html