> Some models are genuinely trained to be better at some domain, such as 2D layouts or vector graphics in this case. It all depends on particular recipes and datasets.

I'd argue this is inherently true for every single model today, none of them have completely generalized to be able to solve any task, so whenever people come up with new evaluations and benchmarks, all the models score relatively poorly initially, until researchers start to tune the models to do well in the domains that the evaluations and benchmarks tests, and then we see strong improvements in that domain, which then tapers out to incremental improvements, and some other domain is chosen to be the new focus.

Models aren't better agents today merely by chance, but because it's explicitly part of the training data. They do well with software because we've talked so much about software on the internet until this point and that's part of the training data, but pit them against problems people don't talk so much about, and if the labs creating and training these models didn't consider those problems, then the model will pretty much suck at it.

I guess eventually they will literally cover every single task the model could ever come across, at least some variant/permutation of it, but until then every benchmark/evaluation will just uncover "did the labs consider this and who considered it most important before/during training?" basically.