Add FAVOR+ / Performer attention

# 🌟 FAVOR+ / Performer attention addition Are there any plans to add this new attention approximation block to Transformers library? ## Model description The new attention mechanism with linear time and space complexity was introduced in _Rethinking Attention with Performers_ [[https://arxiv.org/abs/2009.14794](https://arxiv.org/abs/2009.14794)]. Authors of the paper claim that the new attention mechanism is backward-compatbile with already existing models > Backwards compatibility with pretrained models is available as a benefit from softmax approximation, via small finetuning (required due to error propagation) <!-- Important information --> ## Open source status * [x] the model implementation is available: it's an original Trax implementation from Google: https://github.com/google-research/google-research/tree/master/performer/fast_self_attention * [ ] the model weights are available: probably not required as it's a building block for models rather than fully new architecture * [x] who are the authors: Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller

Posted by marrrcin on Oct 9, 2020, 11:51 AM

Tags

• github (Software)

Related issues

• [Meta] Feedback on granular control over Copilot changes
• unexpected status 401 Unauthorized issue
• [MODEL] Measurable quality regression in Claude generation 5 (Fable 5 / Opus 5 / Sonnet 5): worse nonsense detection, ~2x verbosity, under-disclosed model fallb
• [CRITICAL] Subagent spawning and subagent pattern bugs trigger infinite recursion, infinite token usage, grossly inefficient token usage, and lost accumulated s
• Video stops playing after some minutes
• [MODEL] Claude verbose code comments by default — ignores instructions to stop.