I tried to have multiple models convert a simple textmate grammar to a vim one, and none of them could do it. They couldn't even use the right names between the regex matches and the color definitions. I tried for about 30 minutes. It took me about 5

I tried having them work on a LSP. The fact I got a one shot half working autocomplete based on my existing work was cool, but again, they flailed on incredibly simple things like file path normalization / converting from a URI and I had to rewrite a decent amount of code. I don't think I saved any time

People keep throwing this out there but I keep wondering where are the receipts? I am seeing less interesting software released, anecdotally I know, since AI has taken hold, than before.

Did you try providing it documentation for the respective formats (via browsing/tool use or input to the prompt)? And were you using a modern thinking model from Anthropic or OpenAI?

The crucial breakdown here sounds like either lack of proper context/harness or insufficiently capable model (there's a huge gulf between GPT-5.5/Opus 4.8/Fable class models and anything not from the big three) or both.