I use ChatGPT and Claude daily, but I can't see a use case for why would I use LLM outside of these services.

What do you use Llama.cpp for?

I get you can ask it a question in natural language and it will spit out sort of an answer, but what would you do with it, what do you ask it?

▲

anon373839 10 months ago | parent | next [-]

You can run a model with substantially similar capabilities to Claude or ChatGPT locally, with absolute data privacy guaranteed. Whereas with Claude or ChatGPT, all you can do is trust and hope they won’t use your data against you at some point in the future. If you’re more technically minded, you can hack on the model itself, the sampling method, etc., and have a level of fine-grained control over the technology that isn’t possible with a cloud model.

▲

ukuina 10 months ago | parent [-]

> You can run a model with substantially similar capabilities to Claude or ChatGPT locally

I am all for local models, but this is massively overselling what they are capable of on common consumer hardware (32GB RAM).

If you are interested in what your hardware can pull off, find the top-ranking ~30b models on lmarena.ai and initiate a direct chat with them on the same site. Pose your common questions and see if they are answered to your satisfaction.

	▲	anon373839 10 months ago \| parent [-]
		Two points: 1) I actually think that smaller models are substantially similar to frontier models. Of course the latter are more capable, but they’re more similar than different (which I think the ELO scores on lmarena.ai suggests). 2) You can run much larger models on Apple Silicon with surprisingly decent speed.

▲

SteelPh0enix 10 months ago | parent | prev [-]

I use llama.cpp mostly for working with code that i can't share with any online provider. Simple NDA stuff. Some refactors are easier to do via LLM than manually. It's a decent debugging duck too.

	▲	varispeed 10 months ago \| parent [-]
		Do you know any tutorial that could help me set something like this up?