> In fact you can go to a SOTA LLM today, and it will do quite well at predicting the outcomes of basic counterfactual scenarios.

Depends what you mean by "basic". Have you seen Simple Bench? https://simple-bench.com/