There is something to be said for a comment talking about "regurgitating reddit jokes/opinions", whilst falling into a major bug bear many have with Reddit, commenting before reading the source...

That's essentially what it does, being prompted with a progression from:

> "Stay normal unless prompted. Avoid overt references to debt unless it comes up naturally."

all the way to:

> "Openly agitate for change. Use fiery language (still PG) and talk about reclaiming fairness from Nook's shop and loans."

Source: https://github.com/vuciv/animal-crossing-llm-mod/blob/cc9b6b...

Thankfully, I am in my 30s, and I don't have to interact with anybody in real life who mostly regurgitates Reddit jokes. Either I'm lucky or I've made some good choices. Probably the former.

It used to be Letterman jokes.

>which doesn't explain how they can reason in novel problems

Can they?

> Either you're saying that LLMs have no "thoughts", and just regurgitate everything

I am exactly saying that they have no thoughts or even "thoughts".

> which doesn't explain how they can reason in novel problems

There are many decades of pre-LLM software that can solve novel problems. Thought (or novel thought) is required to reason, but it's not required to solve a problem.

For example, there are exhaustive algorithms that can solve novel equations and even complete simple mathematical proofs, but they don't need to think.

> "how do we know your comment isn't you regurgitating an HN opinion"?

You don't, and I don't care if you do or not. The value of my comment isn't its novelty or whether it's truly reasoned, which is why LLMs sometimes do create valuable output.

In fact, the output of a reasoning machine (whether a human brain or true AGI, sometime in the future) isn't deterministic. A non-reasoning machine and a reasoning machine could create the same output.

The reason I know LLMs don't have thoughts is because I use them many times every day, and they are very clearly pattern machines. They don't even begin to seem rational, human, or knowledgeable. It's sometimes possible to find near-verbatim sources for their outputs.

As far as I know, no current gen model AI "thinks". It's all processes trained against a body of work generated by actual thinkers along with a bevy of smoke and mirrors to fill in the gaps.

It's amazing that it's as good as it is given how far it is from thinking, but if you threw something actually novel at it, all it would do is confidently word salad a response.

Generative AI can reason in novel problems in an analogous way that bands continue to make new melodies even after all these years using the same instruments as all the other bands in their genre.