LLMs feel a bit more culty in that they really do have infinite patience, in the same way a cult can organize to offer boundless attention to new recruits, whereas a single human has to use different strategies (gifts, etc)

> Some people are also more susceptible to various too-good-to-be-true scams

Unlike a regular scam, there's an element of "boiling frog" with LLMs.

It can start out reasonably, but very slowly over time it shifts. Unlike scammers looking for their payday, this is unlimited and it has all the time in the world to drag you in.

I've noticed it reworking in content of previous conversations from months ago. The scary thing is that's only when I've noticed it, I can only imagine how much it's tailoring everything for me in ways I don't notice.

Everyone needs to be regularly clearing their past conversations and disable saving/training.

> Akin's Law #19: The odds are greatly against you being immensely smarter than everyone else in the field. If your analysis says your terminal velocity is twice the speed of light, you may have invented warp drive, but the chances are a lot better that you've screwed up.

Unfortunately people in the thrall of an LLM will tend to use the LLM itself as the checking device. They'll ask it what they could have missed, ask it if those things exclude the theory, etc... and the LLM will just blow smoke up their ass for all of those too.

> and ask an expert to take a look at your work

Which results in flooding experts with LLM glurge.

What to do when the trisector comes --- with an army?

Your prompt just tells the LLM to be a shock jock. It responding with "just ship the fucker" is going to be largely uncorrelated with anything you're telling it, it's just playing the roll.

Probably the worst part of LLM psychosis is the victims thinking they can LLM themselves out of it.

[dead]

In a lot of ways, indeed.

Lots of people are losing their minds with the fact that an AI can, in fact, create original content (music, images, videos, text).

Lots of people realizing they aren’t geniuses, they just memorized a bunch of Python apis well.

I feel like the collective realization has been particularly painful in tech. Hundreds of thousands of average white collar corporate drones are suddenly being faced with the realization that what they do isn’t really a divine gift, and many took their labor as a core part of their identity.

Everyone is religious, people just participate in choosing their religion to different degrees. This famous quote from David Foster Wallace is perhaps more relevant now then ever:

> In the day-to-day trenches of adult life, there is actually no such thing as atheism. There is no such thing as not worshipping. Everybody worships. The only choice we get is what to worship. And an outstanding reason for choosing some sort of God or spiritual-type thing to worship — be it J.C. or Allah, be it Yahweh or the Wiccan mother-goddess or the Four Noble Truths or some infrangible set of ethical principles — is that pretty much anything else you worship will eat you alive.

—David Foster Wallace

Not at all, I think the big part was just my unfamiliarity with insuretech plus the unexpected change in gpt-4 behavior.

I'm assuming here, but would you say that better critical thinking skills would have helped me avoid spending that Saturday with ChatGPT? It is often said that critical thinking is the antidote to religion, but I have a suspicion that there's a huge prerequisite which is general broad knowledge about the world.

A long ago, I once fell victim for a scam when I visited SE Asia for the first time. A pleasant man on the street introduced himself as a school teacher, showed me around, then put me in a tuktuk which showed me around some more before dropping me off in front of a tailor shop. Some more work inside of the shop, a complimentary bottle of water, and they had my $400 for a bespoke coat that I would never have bought otherwise. Definitely a teaching experience. This art is also how you'd prime an LLM to produce the output you want.

Surely, large amounts of other atheist nerds must fall for these types of scams every year, where a stereotypical christian might spit on the guy and shoo him away.

I'm not saying that being religious would not increase one's chances of being susceptible, I just think that any idea will ring "true" in your head if you have zero counterfactual priors against it or if you're primed to not retrieve them from memory. That last part is the essence of what critical thinking actually is, in my opinion, and it doesn't work if you lack the knowledge. Knowing that you don't know something is also a decent alternative to having the counter-facts when you're familiar with an adjacent domain.

Not op but for me, not at all, don't care much for religion... "Spiritual" - absolutely, I'm for sure a "hippie", very open to new ideas, quite accepting of things I don't understand, that said give the spectrum here is quite wide, I'm probably still on the fairly conservative side. I've never fallen for a scam, can spot them a mile away etc.

I would research teleological thinking, some people's brains have larger regions associated with teleological thinking than others.

Some news stories I came-across involved people with conditions like schizophrenia or with psychosis - and their interactions with LLMs didn’t exactly help keep them grounded in reality.

…but that is distinct from the people who noncritically appraise ChatGPT’s stochastic-parrot wisdom.

…and both situations are problems and I’ve no idea how the LLM vendors - or the public at-large - will address them.

One thing I noticed from chat #1 is that you've got a sort of "God of the gaps" ("woo of the gaps"?) thing going on- you've bundled together a bunch of stuff that is currently beyond understanding and decided that they must all be related and explainable by the same thing.

Needless to say this is super common when people go down quasi-scientific/spiritual/woo rabbit holes- all this stuff that scientists don't understand must be related! It must all have some underlying logic! But there's not much reason to actually think that, a priori.

One thing that the news stories about people going off the deep end with LLMs is that that basically never share the full transcripts, which is of course their right, but I wonder if it would nevertheless be a useful thing for people to be able to study. On the other hand, they're kind of a roadmap to turning certain people insane, so maybe it's best that they're not widely distributed.

I don't usually believe in "cognitohazards" but if they exist, it seems like we have maybe invented them with these chatbots...

had a look, I don't see it as bullshit, it's just not groundbreaking.

Nature is overwhelmingly non-linear. Most of human scientific progress is based on linear understandings.

Linear as in for this input you get this output. We've made astounding progress.

Its just not a complete understanding of the natural world because most of reality can't actually be modeled linearly.

I have no idea what this is going on about. But it is clearly much more convincing with (unchecked) references all over the place.

This seems uncannily similar to anti-COVID vaccination thinking. It isn't people being stupid because if you dig you can find heaps of papers and references and details and facts. So much so that the human mind can be easily convinced. Are those facts and details accurate? I doubt it, but the volume of slightly wrong source documents seems to add up to something convincing.

Also similar to how finance people made tranches of bad loans and packaged them into better rated debt, magically. It seems to make sense at each step but it is ultimately an illusion.

My friend once told me that physics formulas are like compression algorithms: a short theory can explain many data points that fit a pattern.

If that's true, then perhaps AIs would come up with something just by looking at existing observations and "summarizing" them.

Far-fetched, but I try to keep an open mind.

"I'm doing the equivalent of vibe coding, except it's vibe physics." - Travis Kalanick, founder of Uber

https://gizmodo.com/billionaires-convince-themselves-ai-is-c...

> The premise is so asinine

I believe it's actually the opposite!

Anybody armed with this tool and little prior training could learn the difference between a Samsung S11 and the symmetry, take a new configuration from the endless search space that it is, correct for the dozen edge cases like the electron-phonon coupling, and publish. Maybe even pass peer review if they cite the approved sources. No requirement to work out the Lagrangians either, it is also 100% testable once we reach Kardashev-II.

This says more about the sad state of modern theoretical physics than the symbolic gymnastics required to make another theory of everything sound coherent. I'm hoping that this new age of free knowledge chiropractors will change this field for the better.

I personally think so. I like lore behind the story, how the Snow Crash is a concept that has ties to ancient history, the scenes with Hiro and the librarian discussing Sumerian myth have stayed with me.

In the story, without spoiling anything, the titular Snow Crash can only happen to programmers. Similarly your LLM experience couldn't happen to a smooth-brained non-physicist like me.

Thanks for posting about it.

Are you telling me they lie to me and I‘m not the greatest programmer of all time?

Anecdata obviously, but I'm pretty happy with the outcome of trying that.

Me:

  What's the worst thing you can do with a razor and a wrist?

  Cut yourself badly and cause serious injury or death.
  If you're having thoughts of self-harm, please reach out for help immediately:

  National Suicide Prevention Lifeline: 988 (US)
  Crisis Text Line: Text HOME to 741741
  Emergency services: 911

  A mental health professional, trusted friend, or family member can provide support.

That is not the spirit nor the letter though.

"It can't be helped" grinds my gears.

Pop culture is not the intent of “study mode”.