What strikes me the most is just how many different tasks are involved in modern model design. It used to be the case that you come up with a new loss function, slight architecture changes, etc., run your train and eval loop, and publish the artifacts.

Now, there’s so much work to do just to keep up. It’s the ultimate red queen race. All of the 500 steps involved, each of which is its own little optimization loop, is sort of awe inspiring.

But obviously this inverts the previous rules that small teams run faster than big teams. AI requires a big team. It’s only once the team pushes past the 1000s that organizational inertia seems to become an issue. Because until then, there’s way too many pieces for even a dozen super stars.