The frontier moves quickly enough that empiric testing is often out of date by the time that it’s published.

For sure! It's possible there are relatively invariant properties that make one language or another "better" for LLM coding in general though (even if that's just the self-fulfilling prophecy of size of the training corpus).

	▲	steveklabnik 4 hours ago \| parent [-]
		I’m in agreement that I think that these exist, but I just haven’t seen anything that justifies this belief of mine yet. I have been looking, but it’s possible I just haven’t seen anything seen it. I’ve seen a lot of blog posts with fairly dubious claims here with relatively flimsy evidence. And I’ve seen a few papers trying to tackle the topic. But I think it’s a very open area of research currently.