Can someone tell me what the advantages are of doing this over using, e.g., the ChatGPT web interface? Is it just a privacy thing?

▲

0000000000100 10 months ago | parent | next [-]

Privacy is a big one, but avoiding censorship and reducing costs are the other ones I’ve seen.

Not so sure about the reducing costs argument anymore though, you'd have to use LLMs a ton to make buying brand new GPUs worth it (models are pretty reasonably priced these days).

▲

stuckkeys 10 months ago | parent [-]

I never understand these guardrails. The whole point of llms (imo) is for quick access to knowledge. If I want to better understand reverse shell or kernel hooking, why not tell me? But instead, “sorry, I ain’t telling you because you will do harm” lol

▲

TeMPOraL 10 months ago | parent [-]

Key insight: the guardrails aren't there to protect you from harmful knowledge; they're there to protect the company from all those wackos on the Internet who love to feign offense at anything that can get them a retweet, and journalists who amplify their outrage into storms big enough to depress company stock - or, in worst cases, attract attention of politicians.

	▲	mistermann 10 months ago \| parent [-]
		There are also plausibly some guardrails resulting from oversight by three letter agencies. I don't take everything Marc Andreessen said in his recent interview with Joe Rogan at face value, but I don't dismiss any of it either.

▲

explorigin 10 months ago | parent | prev | next [-]

Privacy, available offline, software that lasts as long as the hardware can run it.

▲

JKCalhoun 10 months ago | parent [-]

Yeah, that's me. nCapture a snapshot of it, from time to time — so if it ever goes offline (or off the rails: requires a subscription, begins to serve up ads), you have the last "good" one locally.

I have a snapshot of Wikipedia as well (well, not the whole of Wikipedia, but 90GB worth).

▲

3eb7988a1663 10 months ago | parent [-]

Which Wikipedia snapshot do you grab? I keep meaning to do this, but whenever I skim the Wikipedia downloads pages, they offer hundreds of different flavors without any immediate documentation as to what differentiates the products.

▲

severine 10 months ago | parent | next [-]

You can use Kiwix: https://kiwix.org/en/

▲

JKCalhoun 10 months ago | parent | prev [-]

wikipedia_en_all_maxi

I guess that means English ... and maxi? As I say, was something around 90GB or so.

	▲	3eb7988a1663 10 months ago \| parent [-]
		Was hoping you had more insight than "maxi sounds good" which is the also the best I have.

▲

elpocko 10 months ago | parent | prev | next [-]

Privacy, freedom, huge selection of models, no censorship, higher flexibility, and it's free as in beer.

	▲	fzzzy 10 months ago \| parent [-]
		Ability to have a stable model version with stable weights until the end of time

▲

cess11 10 months ago | parent | prev | next [-]

For work I routinely need to do translations of confidential documents. Sending those to some web service in a state that doesn't even have basic data protection guarantees is not an option.

Putting them into a local LLM is rather efficient, however.

▲

zarekr 10 months ago | parent | prev | next [-]

This is a way to run open source models locally. You need the right hardware but it is a very efficient way to experiment with the newest models, fine tuning etc. ChatGPT uses massive model which are not practical to run on your own hardware. Privacy is also an issue for many people, particularly enterprise.

	▲	LoganDark 9 months ago \| parent [-]
		That said though, there's currently no practical way to reach ChatGPT's performance locally, but I have hope that will change eventually. Of course, even today you can reach quite good performance locally, and for a lot of people it's more than good enough for their needs.

▲

JKCalhoun 10 months ago | parent | prev | next [-]

I did a blog post about my preference for offline [1]. LLM's would fall under the same criteria for me. Maybe not so much the distraction-free aspect of being offline, but as a guard against the ephemeral aspect of online.

I'm less concerned about privacy for whatever reason.

[1] https://engineersneedart.com/blog/offlineadvocate/offlineadv...

▲

pletnes 10 months ago | parent | prev | next [-]

You can chug through a big text corpus at little cost.

▲

priprimer 10 months ago | parent | prev | next [-]

you get to find out all the steps!

meaning you learn more

	▲	loudmax 10 months ago \| parent [-]
		Yeah, agreed. If you think artificial intelligence is going to be an important technology in the coming years, and you want to get a better understanding of how it works, it's useful to be able to run something that you have full control over. Especially since you become very aware what the shortcomings are, and you appreciate the efforts that go into running the big online models.

▲

throwawaymaths 10 months ago | parent | prev [-]

Yeah but I think of you've got a GPU you should probably think about using vllm. Last I tried using llama.cpp (which granted was several months ago) the ux was atrocious -- vllm basically gives you an openai api with no fuss. That's saying something as generally speaking I loathe Python.