> "dual xeon"

Does inference make full use of the memory bandwidth in a NUMA system?

Memory throughput performance is better than many equivalently priced (talking about the cheaper end of the used/refurb market here) single socket systems because each CPU is talking to its own banks of RAM.

Using an example of llama-server on a 256GB (each CPU is attached to 128GB) system running qwen 3.8-flash-next Q8, which takes up about 188GB with the model and context, it's not that the inference engine 'knows' about NUMA, but if you use --load-mode mlock to load the whole thing into memory at once at the time llama-server launches, the kernel will distribute it relatively evenly across the memory connected to both CPUs.