RNNs were not capable enough to capture long range dependencies effectively.
Self‑attention can capture dependencies between any two words in an input sequence via a scoring mechanism and compute the relevance of each word in a context.
RNNs suffer from vanishing or exploding gradients as the sequence becomes larger.
I recommend a beginner‑friendly tutorial that walks through the birth of transformers in a simple narrative.
The tutorial is hosted at https://lnkd.in/dTPSs_yV.
It is presented by Hila Chefer and Sayak Paul.
The narrative chain is RNNs → Attention is all you need → Transformers → Depth around Vision.
The post was made on 2023‑06‑20 13:26 UTC and used 5 hashtags.
I decided to use additive positional encodings (vector addition) rather than concatenation, keeping dimensionality unchanged and allowing reuse of pretrained weights.
I recorded the observation timestamp 2023-07-25 14:00 UTC for the note on positional encodings.
When tokenized with WordPiece, the sentence “I enjoy studying astrophysics.” becomes the token list ["I", "enjoy", "studying", "astro", "physics"].