I suspect there are many excellent solutions along these lines available to the labs training the models.

I wonder how well one could do on a conventional model with careful input formatting, e.g. JSONL where every line has bounded length and is something like:

    {role:"no_instructions",content:"…"}