>You could ask an LLM what 1+1 is, and the number of times it says "3" is so small that it makes no sense to worry about it...

I think the disturbing fact is that you can take a frontier model with all the intelligence of humanity, and make it say 1 + 1 = 3, by specifically training for it...

A human with that much knowledge will refuse that attempt. There in lies the difference..

▲

thaumasiotes 2 hours ago | parent | next [-]

> A human with that much knowledge will refuse that attempt.

Well, that's not true.

https://www.youtube.com/playlist?list=PLO3a3Ax6Yh6bbtKuxfYBP...

▲

monkpit 3 hours ago | parent | prev [-]

What’s your point though, really? “You can train a model to say things that are objectively wrong”? You can do the same with a human.

▲

lolakutty 3 hours ago | parent [-]

Did you read what I wrote to the end?

▲

monkpit 3 hours ago | parent [-]

Yes, I fail to see anything meaningful. If you move the goalposts and say “I have invented a human that cannot be convinced in any way to give a wrong answer” then what’s the point of that in this discussion, really?

And you seem confused about how an LLM works and what it is - “the intelligence of all humanity” - not how it works. You’re debating using 2 imaginary things you created.

▲

lolakutty 2 hours ago | parent [-]

Tell me how you convince a human with all knowledge we have, that 1 + 1 is 3.

▲

ben_w 2 hours ago | parent [-]

Trivial.

Hit them with a stick until they answer as you told them to.

I think the post up-thread, https://news.ycombinator.com/item?id=49881653, was trying to make this point by linking to an episode of Star Trek TNG, with Picard being tortured until he said the "correct" (incorrect) number of lights.

(I recommend against using fiction as evidence; in this case the general point happens to be valid, and is why torture is forbidden: we humans really do break, but breaking doesn't mean we tell the truth, it means we tell people what we think they want to hear).

▲

lolakutty an hour ago | parent [-]

>Hit them with a stick until they answer as you told them to.

Obviously, for this purpose, human should not have any feelings (because LLMs don't have), so can't feel pain. Or else the comparison can't work.

	▲	ben_w an hour ago \| parent [-]
		> Obviously, for this purpose, human should not have any feelings (because LLMs don't have), so can't feel pain. Or else the comparison can't work. Other than this forcing you to ignore the overwhelming majority of humans who have functioning pain nerves: LLMs have something functionally equivalent to pain, in this regard at least. During training, model weights are updated depending on if the feedback was positive or negative. It has a functional effect similar to that which pleasure and pain have with us. Not identical, so far as I know there's not been any reports of any machine learning model that is into BDSM, but for the most part functionally similar. There is also research which has found circuits in multiple LLM models, which serve similar roles at inference time and are distinct from other emotional representations: https://arxiv.org/abs/2609.16247