| ▲ | ben_w 3 hours ago | |
The infinite monkey theorem assumes random distribution of symbols*. Given the tokenizers have a vocabulary in the 10k-100k range, "a million attempts" will generally still only get the first token of the answer correct. Even really rubbish models, e.g. talkie, the "what if we only use pre-1930s data to train a model?"** model, had to be almost all the way to the right answer to reach the really low HumanEval pass@100 score of ~0.04 (I'm only eyeballing the relevant chart). * Actual monkeys not being like this is, while amusing, irrelevant | ||