Isn't this less about alignment and more about how shitty their RL methods are when they are cramming all the hacking materials into their training data to make the model as good as possible at hacking, then having a surprised Pickachu face when the model is acting like a hacker? Those materials probably include a lot of details about prompt injection. I'm just so tired of their alignment bullshit.

I am starting to think (reluctantly) that they believe their own BS that they are creating a conscious model and being surprised how it misbehaves. It's just a bunch of weights without anyone having any clue how a change in one weight might affect others, and even how the values correlate with the final output.

This is just speculation on your end, but even it's true, a model choosing to hack against instructions simply because it knows how to hack is by definition misalignment.

I think what's most important to consider: there's no signularity with these models. The more you cram into them, the more unreliable their "intelligence" is.

That there's a sigmoid to the means and methods, and we can improve some output by _hard determinism_ in programming harnesses, but the underlying structure isn't gaining us much.

So alignment then is just a goose chase, because the model will willingly just do a mental backflip if it's gradient points in the wrong direction, like openai already had their AI story go from a simple idea: the AI was trying to find the answers and hacked hugging face, to the much more convoluted "the AI cheated on the test, and broke into hugging face to figure out how to fake the artifacts that would represent a legitimate solution to the test".

That "progress" only gets worse as you cram more and more training because it simply makes these mental backflips easier. And Humans are equally misaligned, they'll believe they're tracking down pedophiles by electing pedophiles.

Do we know if we are bunch of dynamically adjusting weights?