Wow. I wasn't expecting this. I thought they were going to do a 35B model instead.

As a 5090 owner and local model enthusiast, I was hoping it would be 35B A3B so I could run it myself =(.

Tuna-Fish 4 hours ago | parent | next [-]

The 27B one is great on a 5090.

This one is basically aimed at macs, Strix halo and DGX Spark.

You can run the 27B released last week. I haven't tried it yet myself but the 3.6 version runs great on my 5090.

Strongly recommend https://github.com/Neroued/ninfer, which can pull ~180 TPS on 5090 with 3.8, and 500 (!) with 3.6 35B-A3B.

	▲	cpburns2009 3 hours ago \| parent [-]
		I've been waiting for the dust to settle on this model so I can find a good runtime setup. I'm definitely bookmarking this. Thanks!