If I had a pseudo random number generator that somehow kept outputting winning lottery numbers a reasonable percentage of the time, I would use it to enter the lottery. Even if it seemed completely impossible and despite studying the algorithm, I did not understand how it worked—I would still use it.

That's basically how I feel about LLMs right now. I have no idea how any of this is possible but somehow it seems to be working.