| ▲ | agile-gift0262 3 days ago | |
In their page they say it will need at least 112GB[0], so including context, that would be a tight fit. I'm also hoping I can make a q4 fit on my 128GB strix halo [0]: https://unsloth.ai/docs/models/qwen3.8-next#qwen3.8-flash-ne... | ||
| ▲ | walrus01 2 days ago | parent [-] | |
in llama-server PR 27742 it fits fine in 128GB RAM on a CPU only system , this is with --load-mode mlock to stuff the whole thing persistently into memory at llama-server launch time, no mmap 0.01.033.250 I common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted | 0.01.033.261 I common_memory_breakdown_print: | - Host | 118186 = 106166 + 8898 + 3122 | 0.01.092.684 I common_params_fit_impl: projected to use 118186 MiB of host memory vs. 128855 MiB of total host memory | ||