You are completely right it's a what is the next token guessing machine so without corpus it's guesses are worse as expected.
I’m glad to see someone else in these parts understands.