| ▲ | Philpax 4 hours ago | |
Strongly recommend https://github.com/Neroued/ninfer, which can pull ~180 TPS on 5090 with 3.8, and 500 (!) with 3.6 35B-A3B. | ||
| ▲ | cpburns2009 3 hours ago | parent [-] | |
I've been waiting for the dust to settle on this model so I can find a good runtime setup. I'm definitely bookmarking this. Thanks! | ||