This implies that weights are a type of language. Neural networks are good at language because they are good at patterns. They can reason about it because their "reasoning" isn’t how we think. The same goes for how they encode images, their vision is entirely different from ours. That's why adversarial attacks work on them and not on us. We don't see anything change in the image, while the neural network now sees that the dog is a cat.

▲

jgilias an hour ago | parent | next [-]

Ok? But does that mean that the “weights language” is universal? E.g. same weight patterns mean the same thing regardless of if it’s Kimi, Grok, or Astra?

	▲	HAL3000 an hour ago \| parent [-]
		There could be different patterns that solve the same problem. Just as you can have different algorithms, there could be different weights, but the abstracted result or logic from those weights would be the same. Take these words I'm writing now, they will be transformed into numbers, then sent to your screen, and then your brain will transform them into analog signals. I bet the neural networks in our brains have "different numbers", but the algorithms that understand language and other concepts are similar and could be universal. It's like a sorting algorithm, which I can implement in many different ways in just a few dozen lines. If you look at the binary code (so like raw weights) of the compiled program, you will see just a bunch of a different numbers, but if you read the code, in most cases you will understand that it's a sorting algorithm. So it seems they just operate on different levels of abstraction that are not available to our brains.

▲

cout an hour ago | parent | prev [-]

How do we (or can we) know adversarial attacks don't work on us?

	▲	HAL3000 21 minutes ago \| parent \| next [-]
		That's not the point I'm making. So in context of vision we pattern match too, and you can see it in the hundreds of optical illusions online. Our visual system relies on built in shortcuts, and they break in different places than a neural network's do. Image models get fooled by things we would never notice, and we get fooled by things they handle fine. IMO that's good evidence that we and these models use different algorithms to solve similar problems. I think the same is probably true of LLM "reasoning", there's some overlap with how we think, but a lot of it likely works very differently under the hood. Good example of this is anthropic's interpretability research where they found claude doing addition in a way that's nothing like the carry the one method it describes when you ask how it did it.
	▲	naniwaduni an hour ago \| parent \| prev [-]
		We're pretty confident that adversarial attacks do work on us, actually—we have entire industries built around it. Some of them even start with "adver-"... There are classes of adversarial attacks targeting them that don't work as well on us, and vice versa.