Simon, could you try reproducing Experiment 5 from the blog post, if you have a chance? Apparently this person tried [1] with API models and only the latest models can do the impressive 10-20x vs. PPO baseline.

[1] - https://nitter.netbub.com/AlexKrentsel/status/21173766837246...