One thing that I really want to know - the better models from today vs a year ago - what has changed. They have already pre-trained on all available public data. Scooping up the last percentage of archaic texts which were never digitized is not going to move the needle.

Is it just that the providers are generating tons of synthetic datasets on coding tasks so that the models get more exposure to the right thing to do? Every time someone points out an LLM stupidity they add some training data to patch over the weakness (trivial to generate "there are two 'l's in llama")?