I think, there's a systematic flaw in RLHF: constructs reinforced as effective or sophisticated style will always suffer overuse and significant overexposure. Which will be also very obvious when used out of the intended context, it's copied from. There will be always a "smell", something that causes us to turn our backs to these texts, and possible the supposed authors, as well.

There's also the problem that certain types of phrasing are perceived as effective, because they mark a pivotal point in the progress of the text and are, as such, used sparingly, but are now becoming everyday templates that incorporate whatever is available in the context. There is no way this passes the Turing test of a competent reader.

And there's yet another issue: in social research, there has been the concept of semantic position, indicated by deviation from the mean (or median). If you don't deviate from the mean, your semantic position is zero. There's simply no expression. In this sense, next token prediction really amounts to a desemantificiation of the context. There's really no sense in uttering any of these productions, no plausible motivation, other than for the purpose of raising you hand to be seen.

> There is no way this passes the Turing test of a competent reader.

Every day I get more depressed thinking of what's happened to the supply of competent readers.

[dead]