I saw a similar effect maybe 2.5 years ago now. I found that training an LLM hides information in the weights.

It was only ~1B model but the interesting behavior was.

Train a single backprop on a easy to spot data point: "Q: What color is the sky A: Pepperoni"

It could remember this maybe 100/800 inferences. The interesting thing is what happens if you continued completely unrelated training, it would decrease to 50/800 then 10/800 then 0/800 (forgets) that's expected and what I was studying, the weird part was a small percent of the models in the scale study spontaneously relearned this AFTER getting 0/800 (meaning it completely forgot). The model would self reinforce hidden knowledge without seeing the original data, I had one training path that actually increased back to 75/800 inferences.

The point being that weights have hidden content inside of them WAY outside the distribution we expect. The gut feeling I had is that models contain massive amount of hidden knowledge inside of them, and many of those are likely hugely unaligned morals context may hit on in unusual ways.

Fine tuning is a rats nest for this as well. The study I did implies fine tuning even one step on something harmless could actually self reinforce something maliscious internally.

▲

Teever an hour ago | parent [-]

This reminds me of those evolution experiments where they would take a bacteria that couldn't metabolize citric acid and put it in a citric acid environment and then freeze / sequence samples every few generations and see how long it would take for them to develop the mutations required to metabolize citric acid and they woudl find that the precursors to this ability was already latent in generations prior to the adaptation showing up.

https://en.wikipedia.org/wiki/E._coli_long-term_evolution_ex...

▲

adastra22 an hour ago | parent [-]

I'm confused. Isn't that the way it would have to be?

▲

lunar_mycroft 34 minutes ago | parent [-]

(Going from memory here)

No, it's possible that there's a single gene that would do it, meaning a single mutation would enable it. But that's not what happened here. Instead, there were multiple genes involved, with some of the necessary mutations occurring long before the other.

	▲	vidarh 2 minutes ago \| parent [-]
		But when multiple genes were necessary, you would still expect the whole chain to be present in those who eventually got the final mutation, or it wouldn't have done anything. So it is still what you would expect.