Is the inference engine open-source?

It will be when done. I have a lot of stuff in the works and switch between them based upon my current interest. I like plotting and writing stories and Gemma is one of the nicer writers.

The goal is to get the 31B working on a 16GB card. Which means weights have to be < 4 bits. The 9070 XT has wmma support for int4. Which doubles compute performance on pp compared to i8/fp8. tg is limited by bandwidth.