I wouldn’t exactly put it like that. It’s moreso the model sometimes outputting non-optimal tokens in a way that’s detectable if you know the algorithm.

It seems possible for that to make the response “drift” far from what it would’ve been, because it’s constant entropy that adds up after time.

(However, according to Anthropic and Google, it doesn’t really impact the quality of responses. I find that a bit hard to believe, although those guys are much smarter than I.)

▲

algoth1 3 hours ago | parent | next [-]

Yeah, it’s hard to believe, specially when you are coding and there’s only one best way to do things, unless it plays with variable naming, or comments

	▲	demibabs an hour ago \| parent [-]
		Well the algorithm only increases the chances of certain words being chosen/not chosen, rather than guaranteeing it. If there’s a clear answer then that nudge won’t do anything. If the model’s most recent output is “for (let i = 0; ”, the likelihood of the next token being “i” is probably millions of times greater than any other possible token. Thus even if “i” is on the red list and has its likelihood decreased, it’s not going to suddenly choose another word. Put another way, on low-entropy tasks like coding, this style of fingerprinting is less effective and needs bigger sample sizes to be recognizable. That said, even small changes can dramatically affect output quality, which is why I’m still a skeptic.

▲

fuglede_ 4 hours ago | parent | prev [-]

One way to watermark (assuming temperature is otherwise positive) would be to output the most likely (or optimal) token every so often.

	▲	cl3misch 3 hours ago \| parent [-]
		But wouldn't you have to know the exact context before this token in order to verify the watermark? I.e. a paragraph wouldn't be sufficient; you would need the system prompt, previous prompts, and even hidden thinking?