amusingly ive been working on ultra sparse llm inference/ training/ model design because nature loaths a dense graph/matrix and cause i think it shoukd be possible. i actually stood up a 20-25 percent faster than sota causal fast attention kernel yesterday, will be standing up cuda/metal/armv8 kernels too and thats gonna be fun.

i genuinely think these models should be like 0.1 percent sparse for same capabilities we associate with them today, but theres no sane way to do that with extent tools. i built the right core tech for that in 2014 when there wasnt a market, but now there is and the experimentation velocity is wild.

amusingly llms really have a hard time using my simple apis because its not in distribution array programs. but i literally stood up cpu custom memory format and micro kernel for dense causal attention in less than 24-36 hours and outperforms the equivalent fused ggml/llama cpp fast oath by like 20-25 percent

Look forwarding your future releases