The problem with language models is their bias. Train a model on pure math and THEN when it starts talking by itself about consciousness (*), then we can say: that's odd.

(*) of course in its own language

Well we haven’t quite gotten there. But models trained purely on DNA sequences have demonstrated in-context learning on various simple reasoning tasks.