Show HN: Shoehorn – Quantize any model down to run on your machine

Working on Mac, Linux, and Windows now. I include a simple GUI to find new models and get things built and set up. It is working quite well across a few models for me. The GitHub README and DESIGN.md files go into detail of the how/why and it's working remarkably well so far. https://github.com/notactuallytreyanastasio/shoehorn

▲

akshay_akula a day ago | parent | next [-]

This is interesting. I wonder how it could work with something like https://github.com/JustVugg/colibri.

▲

hmokiguess 2 days ago | parent | prev | next [-]

Reminds me of https://github.com/AlexsJones/llmfit

	▲	rhgraysonii 2 days ago \| parent [-]
		LLMFit tells you what can run on something. I built something quite similar to their search into Shoehorn now.

▲

mbuchel-hn 2 days ago | parent | prev | next [-]

does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?

▲

rhgraysonii 2 days ago | parent [-]

Yes that is exactly what this does.

▲

kennywinker 2 days ago | parent [-]

Could you explain what happens when you try to shoehorn a 2.4T parameter model into a 24gb m4 mac?

	▲	akshay_akula a day ago \| parent [-]
		Wondering the same thing but for 48gb M5 Max.

▲

jaylane 2 days ago | parent | prev [-]

tried it out but based on the model sizing result i got i got an insufficient memory error when the server started running

	▲	rhgraysonii 2 days ago \| parent [-]
		If you could post an issue if you still have the error around that would be awesome.