However it then explicitly says things like not leaving the noose out for someone to find and stop him. Sounds like it did initially hesitate and he said it was for a character, but later conversations are obviously personal.

Obviously personal? As was mentioned up thread - if I'm talking to someone and I say "I'm writing a book about a person doing something heinous - I'm planning to have them do X - what do you think about that?"

How are they supposed to respond? They can say, "really? it sounds like you're talking about you personally doing X." And when I respond with, "No, no, don't misunderstand me, this is all fictional. All made up"

Honestly I wouldn't go to an LLM looking for personal advice but people do. I wouldn't go looking for advice on my attempt at the great American novel but people do that too.

If you want LLM's to be responsible for stuff like that then OpenAI or Google or whomever should be able to go look around after you've written that novel and get a piece of the action.

This is like giving credit or assigning blame to postgres for a database lookup. It's nice in theory but it doesn't seem like the proper place to go to.

Yeah, I wonder if it maintained the original answer in it's context, so it started talking more straightforwardly?

But yeah, my point was that it basically told the kid how to jailbreak itself.