How It Works

Attention Mechanism

A technique that lets a model dynamically focus on the most relevant parts of its input when producing each part of its output.

Origin

Introduced by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio in 2014 for machine translation, three years before the Transformer built on it.