Is the publishing cycle this long? Those models were relevant 18 months ago

Working out the kinks in your methodology obviously takes some time. But is there any structural reason not to rerun the benchmarks on current models in the final stages before submission? Assuming the results generalize to those newer models

It's challenging to keep running experiments. In a perfect world, we would run our experiments again and update the results. But some experiments are expensive, and some experiments are inherently difficult to reproduce. I have a paper nearing completion that involves interaction between trained (human) professionals and a model. It was a logistical challenge to bring those people together (also, we had to pay them), and the roleplaying that we asked LLMs to do can't be done anymore--they refuse for safety reasons. Stuff like this is pretty common.

I am hopeful that we can eventually move to open weights models, but at the time we did the experiments, the open weights models did not have the capabilities we needed. The ever-shifting sands of frontier models makes reproducibility difficult, and not just for scientists.

Unfortunately it is this long. I know people in academia studying human-AI interfaces and up until recently they were still using gpt-3.5.