| ▲ | yakupov_bulat an hour ago | |
Wow, amazing! What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine. | ||
| ▲ | gitpusher42 an hour ago | parent | next [-] | |
It depends on the use case. I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s. The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit. If you need maximum speed and flexibility just use MLX | ||
| ▲ | 0gs an hour ago | parent | prev [-] | |
you could use mine ... github.com/0gsd/enough (it has other stuff too) | ||