attention-kernel-optimization

FlashAttention Explained: IO-Aware Attention, Tiling and Online Softmax A transformer's attention layer looks like three matrix multiplications and a softmax. On paper it is compute-bound math that a modern GPU…