I'd have just sat there for hours trying to trick ChatGPT into giving bomb making advice, and then disrupted the others by sharing.

There have been many variations on this idea of "letting kids learn on their own". The basic issue is the one you highlight, which is that it works if the student is very self motivated. Most students aren't. Or rather, they're highly motivated to do other things.

Alpha doesn't use a chatbot, it's still a multiple choice / text input into a box type of thing, I don't even think it's AI graded, though the questions may be AI generated, I'm not sure about that.

In the early days they were using off the shelf curriculum software and layering their own steering AI over it.

Most children are naturally curios. The challenge is to make them curios about the things you want them to learn, and make that more interesting than finding out how to make bombs or what the consequences of sharing that with the class will be