Flash attention(Fast and Memory-Efficient Exact Attention with IO-Awareness): A deep dive | Towards Data Science

Flash attention is power optimization transformer attention mechanism that provides 15% efficiency