I've found it to be useful for prototyping. You can throw a game mechanic up, have it build some AI players, tell you any balance issues it runs into, play against the AI yourself to see if the mechanic feels fun. Then iterate.

A few times I've made a tweak to a mechanic and the LLM balance tests immediately found a major issue. Sometimes the AI players it creates just suck at the game. You still have to think, but just having balance tests and then having them cheap to update when I experiment, it extremely valuable. I tested this on a game I had mocked up with cardboard and forced my friends to play. It found all sorts of stuff I missed in those gameplay tests. It didn't replace those sorts of tests with humans, but it let find issues faster so I got more out of human interaction tests.