| ▲ | faangguyindia 7 hours ago | |
On Gemma 4 12B, I am getting 220 ms per move or QS. I used it to play the Snake game locally: prompt_eval=244 ms wall=245 ms schema_cache=hit generated=0 Move limit reached after 200 moves: score=16, length=19. So, if a 12B dense model can offer this latency on a local old PC, then definitely you can scale it up with more powerful machines and get even lower latency. | ||