| ▲ | xlayn an hour ago | |
I do use 2 amd gpus and I get high 40 for generation, 500 for pp and low 20/100 by the end of the context of 256k. llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8_u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1 if you have an igpu and want to exclude or just use some gpus you can use --device Vulkan3,Vulkan2,Vulkan1 in my case vulkan because of amd, you can see your devices with llama-server2 --list-devices Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free) | ||