> “ The agent acted like a hyperparameter optimization algorithm with some basic reasoning baked in.”

Good lens.

The crux of the auto research repo is basically one file - program.md which is a system prompt that can be summarized as “do this in a loop: improve train.py, run the training, run evals, record result. Favor simplicity”. The other files are an arbitrary ML model that is being trained.

This is something I could almost never be bothered to do before, but I can now very lazily set up large parameter sweeps and visualization scripts to really probe things. There's a danger of "analysis paralysis" but I've still found it quite useful. Although I'm not sure it saves me time as much as sanity.