Yeah would also be interested to see the responses without RLHF. Not quite the same, but have you interacted with AI base models at all? They're pretty fascinating. You can talk to one on openrouter: https://openrouter.ai/meta-llama/llama-3.1-405b and we're publishing a demo with it soon.

Agreed on RLHF dominating the results here, which I'd argue is a good thing, compared to the alternative of them mimicking training data on these questions. But obviously not perfect, as the demo tries to show.