Is this science? Perhaps I should submit some of the random roleplay scenarios that I've run with LLMs to New Scientist.

Yes, if you do a bunch of simulations and write-up a technical report, that is science.

https://arxiv.org/abs/2602.14740v1