I'm so thankful for having read https://role-confusion.github.io/ making a somewhat literate on this topic.

I feel like since they developed read-only "role probes", it should be possible for harnesses developers to make a "role api" where you can force it to treat user input as user input, regardless of the content by tweaking the model's activations in real time.

The fact that this isn't being done tells me how much labs's priorities are still set by marketing, and how investing in security is fundamentally against their marketing incentives.

▲ carterschonwald 40 minutes ago | parent | next [-]

i actually have a harness setup that prevents role confusion from happening in a much more robust and interesting way.

hoping to launch a nice commercial version as a saas with some compelling unique features in the next month or teo

	▲	jackb4040 9 minutes ago \| parent \| next [-]
		I'm assuming that vagueness is intentional, but can you tell us the general approach? Is it built on top of inference APIs as a primitive, or is it a novel approach like I'm talking about that inspects the model's internals?
	▲	cmrx64 35 minutes ago \| parent \| prev [-]
		looking for any kind of testing or help? it’s been probably 10+ years since we were in the same irc channels (cmr/KB1PKL). email is in profile :)

▲ jwarden 2 hours ago | parent | prev [-]

Wouldn't it be possible to just fix a single activation, just set a continuous input to what the harness knows the role actually is. Models could then be trained to trust that input and not other signals about roles.

	▲	jackb4040 41 minutes ago \| parent \| next [-]
		The models wouldn't need to adapt their training at all; this is a secondary model trained on their activations. It's electrodes stuck into the model's brain. The original article I posted is built on top of a regular LLM (OpenAI's GPT-OSS)
	▲	amluto 2 hours ago \| parent \| prev [-]
		I suspect there are many excellent solutions along these lines available to the labs training the models. I wonder how well one could do on a conventional model with careful input formatting, e.g. JSONL where every line has bounded length and is something like: `{role:"no_instructions",content:"…"}` It could need a bit of fine tuning to get this to work well.