Part of the annoying thing is that if you're working on a product which uses LLMs, at some level you run out of levers to pull in terms of being able to fix things. At best you're stacking hacks on top of hacks to prevent unwanted output, but at the end of the day if the LLM really decides it simply doesn't want to follow your instructions, you can't do much other than resign to adding *IMPORTANT* and hoping the next model fixes it.

The experience is much closer to working with an external API that you don't have control over and which simply doesn't do what the documentation says. Those have always been the most frustrating parts of programming, but at least previously you could reverse engineer the actual implementation to work around bugs. You can't even do that now because the "boundary" randomly change every day.