Ever heard of the infinite monkey theorem?

This is basically what LLMs do on really hard tasks. Prompt it a million times on a really hard problem and it might output the correct answer once.

The infinite monkey theorem assumes random distribution of symbols*.

Given the tokenizers have a vocabulary in the 10k-100k range, "a million attempts" will generally still only get the first token of the answer correct.

Even really rubbish models, e.g. talkie, the "what if we only use pre-1930s data to train a model?"** model, had to be almost all the way to the right answer to reach the really low HumanEval pass@100 score of ~0.04 (I'm only eyeballing the relevant chart).

* Actual monkeys not being like this is, while amusing, irrelevant

** https://talkie-lm.com/introducing-talkie

>Ever heard of the infinite monkey theorem?

Even if every atom in the universe were a supercomputer generating a trillion trillion random characters every second since the Big Bang, the chance of producing Hamlet would still be essentially zero.