Thanks for sharing! I would love for some go to source for this type of insider info from the people who actually have tried. (The best I've found so far are some guys on Twitter/X like Alex Fateev (https://x.com/superalesha) and AJ (https://x.com/ItsmeAjayKV))

I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.

At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.

Try to use your IGPU as well if you have enough RAM, almost everyone has dual GPUs in their computers.

Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.

Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.

I might write more about this setup soon.