Reminds me of the recent paper about delegating document editing tasks to LLMs across different disciplines [1]. That paper found that programming was the only discipline most LLMs can perform long horizon tasks on without accumulating errors & corrupting the document.

I've only read the abstract of this one so far but it seems like this paper has zoomed in on programming with greater fidelity and shown a similar phenomenon. But not about long horizon tasks, more like "long style horizons" of larger sets of structural constraints.

[1] https://arxiv.org/abs/2604.15597

Discussion: https://news.ycombinator.com/item?id=48073246

▲

emp17344 5 hours ago | parent [-]

If it’s not easily verifiable, LLMs aren’t good at it.

▲

jeremyjh 5 hours ago | parent [-]

I think that’s mostly because they get so much more of that reinforcement learning - since it is so economical. I dont know if there is any evidence of a fundamental reason they can’t be just as good at other tasks, but it might be economically infeasible for awhile yet.

▲

mjburgess 4 hours ago | parent | next [-]

No one is curating vast amounts of data for them in other domains. Programmers send programs with fixes

	▲	knollimar 3 hours ago \| parent [-]
		There's no diff of my excel lambdas being fixed? :(

▲

emp17344 3 hours ago | parent | prev [-]

RLVR doesn’t work for unverifiable tasks, so they won’t be able to effectively use tools to boost reliability for those tasks.