Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
In brief
Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.