This quietly assumes that "representations" of reality only encode neutral objective facts (that can be objectively judged, good luck finding a judge everyone agrees on), and "representations" does not encode intentions and value judgements, ie internalized training logic.

But that leaks in, and of course it does! If training only rewards "objective truth" we might expect that sort of neat ontological cleanliness, but we all know real training goals are more complicated.

There doesn't appear to be a bright clear distinction between objective facts and subjective values/intentions in neural networks. The distinction is perhaps illusory, just another of our preconceived notion about the nature of thought...

Fascinating to contemplate, thanks

  >Besides, in weights there are no real incompatibility can happen, all is unified with same math chain through architecture. Different set of local minimums/maximums can have different strong/weak consequences but nothing incompatible.

This argument that incompatibility is impossible has an easy counterexample by looking at loss of coherence between two networks. You can still mash the weights together (so they're not "incompatible" in the basic number crunching, sure), but the resulting neural net is meaningless.

By "incompatible" I meant not in a software sense, but like two logical axiom systems can be incompatible.

There’s an idea of the “broad and narrow gates” that comes to mind, accompanied by a genre of fascinating illustrations. [1]

A modern version [2] appears in the paper "Asymptotic burnout and homeostatic awakening: a possible solution to the Fermi paradox?" [3]

[1] https://en.wikipedia.org/wiki/Matthew_7:13#/media/File:Der_b...

[2] https://pmc.ncbi.nlm.nih.gov/articles/PMC9065981/figure/RSIF...

[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC9065981/

Yes, the mapping function is the thing you're missing.

When it comes to representational similarity analysis, we're trying to isolate the representations from the "coordinate systems" they sit in. If you take an image, and make a second image by rotating it 90 degrees and mirroring it, then, for the purposes of platonic representation hypothesis, they're the same image. Once you get past the "coordinate transform", the similarity between the first representation space and the 90-deg+mirror replica of it is total.

A mapping function between LE and BE can be constructed easily, and it's going to be lossless both ways. Which means that those two representations are fully compatible.

Now, INT16 and FP16? Those are far less compatible representations. There will be an overlap subspace where lossless conversion between the two is trivial, but this compatibility window will be relatively narrow.

Big vs little endian are equivalent up to rotation, which includes permutations.

But it's actually much weaker condition than it looks, because, for example, lot of representations of boolean circuits are equivalent up to linear transformations over Z_2^n.

Anyway, if they truly proven such a theorem you won't find "incompatible" representations of the same thing.

I expect an immense degree of representational similarity to exist between humans.

You're making a mistake there - focusing on the tip of the iceberg without considering what lies beneath it.

The bulk of what the human brain does isn't "decide whether skub is good or bad". The foundation that the human cognition is built upon is vast, and mostly invisible from the inside. You don't get to see the feature matching operators of your visual cortex, or the patterns that decide whether a sound is "soft" or "sharp". And a lot of those foundational features are assembled by biological development programs spun up all the way from DNA.

On that level, when it comes to the bulk of neural circuits and the processes implemented by them, the representations within a "Pro Skub" and an "Anti Skub" guy might easily be 90%+ compatible. The two people who had vastly different life experiences, worldviews and opinions would be more alike to two different LLMs fine tuned from the same base model than they would be to two different architectures trained on completely different datasets.

I don't think neural networks have the same ability of self deception as humans do. A huge reason human world models vary so much is due to them being tied to tribal membership. A huge fraction of all people will not come to a rational worldview derived from accurate data but will start with their tribe's worldview and filter data to match. I personally have never modified my worldview for social reasons and it is surprisingly isolating.