Understanding FlashAttention Pt 1: Personal Notes

The article is the first part of a series called FlashAttention, written by Chizkidd, which aims to explain the concept of attention in deep learning. The author notes that attention has become a ubiquitous component in many deep learning architectures, but its underlying principles are not well understood by many practitioners. Chizkidd argues that this lack of understanding is due to the fact that attention is often explained in an overly simplified way, which can lead to misuse and misapplication of the concept. The author aims to provide a more in-depth explanation of attention, starting with its mathematical formulation and moving on to its implementation and applications.