I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".

None of this matters for the practical outcome.

You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.

[Edit: I see that it was written back in 2023. Then (2023) should be added in the submission title]

If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.

The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.

> I don't care if it's "intelligent", I don't care if it "has a mind"...

Agreed. The AI is useful for the particular tasks that it proves itself useful for. And if it occasionally spits out a claim that it is "genuinely curious" about a piece of research that I will have to do myself because it turned out to be beyond its mechanical capabilities, then it's more productive for me to simply ignore that claim as a statistical anomaly - a mere hallucination - rather than allowing it to burn a ton of extra tokens outputting what may or may not be the current state of the art on theory-of-mind applied to LLMs because I make the mistake of telling it that it might not actually be capable of experiencing emotion.

> The OP is still in the "denial" phase.

This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.

I’m not sure acceptance buys you much. “Well-informed plan” at this stage feels like a useless exercise. It is changing fast, and the world only needs so many electricians. Besides, I don’t think “accepting” the fact that these big companies are pillaging human contribution and selling it back to us is good, even if “it works”.

I think you are majorly overstating OP that ai "didn't work". In fact I agree with the post on almost all aspects and I'd be the last to tell you ai doesn't work. It absolutely does but with a caveat... It's still a tool. And better expertise in the problem domain, along with a better harness used for verifiable outputs will get you better results.

I think the posts mental model of stastically likely prompt completions is spot on.

Actually, there are material differences in the practical outcomes of using AI or not. Proofs don’t confer understanding. AI prose and “art” is materially different than human made.

There is a difference between you not liking something and it being wrong. Maybe think about that with your grey matter.

The One Ring worked.

My biggest issue isn't being too agreeable (ie the psychic con), it's being confidently wrong, including outright hallucinations.

If you ask a common question to an LLM with unusual qualifiers, it tends to ignore the qualifiers and give you the typical answer. I saw a demonstration of this with the whole "the surgeon is my mother" "puzzle" that people use to expose implicit gender bias (ie where they assume the surgeon is a man). Ask variations of this and it'll keep going back to the standard form.

Another one I saw was multiplying large numbers. The starting and ending digits tended to be correct but the middle digits were wrong. Why? Because it's really not doing multiplication at all. It's looking for statistical answers. It's unlikely to have met the exact pair of very large numbers you're multiplying before.

Now pundits will argue that all of these are solvable problems and individually they are. But my suspicion is that there will be a neverending stream of such edge cases and it'll be impossible to trust an LLM's output unless you are knowledgeable enough to fact check it yourself.

Now if your example of identifying zero days, this comes up with what I can only describe as "light positives", meaning it's technically a bug but essentially impossible to exploit. IIRC this came up with the demonstration where someone pointed Fable at some BSD code. I'm not sure if there have been any true false positives and obviously false negatives are impossible to know.

I guess my point is that I think LLMs are way more limited than a lot of people think.

The con is in how all those accomplishments have been presented to you. "Our LLM (not the one we let you use, a different one) did this amazing thing. No, we won't show you what training data we used, what prompts we used, what the harness was, how much human involvement there was, what hardware was involved, how much energy it took, or how much time it took. Just shut up and be amazed!"

Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article

Have you ever produced anything apart from sponsored influencer messages that select daily talking points?

I for one cannot believe that posts on checks notes software crisis dot dev are not reasonable, level headed assessments of software

> If it generates functional output that works, then it works.

Tried that in legal? Finance? Medicine?

Good luck with the lost cases, failed deals, harmed patients.

"If it works, it works" applies only when the work is brute-forceable e.g. vuln search, or is generation of bullsh*t e.g. adverts, phishing scams and deepfakes.

it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.

I’m on your side. Zoltan provides an experience and that’s all that matters. If it works for you, why does anyone else care that the economy is riding on its success?

https://www.psychiczoltan.com/psychic-reading/

[dead]