As they wrote most of the wins are because of fusion and TimyGrad started to have fusion optimizations in the last few weeks.

GeoHot didn't want to make it only FlashAttention specific, he worked on FlashAttenrion being automatically generated by the optimizer. It's going well