I understand your use case and the benefit experienced, but quite frankly, I don't think that's easy to generalize or extrapolate to common problems justifying the expenses of this technology.
Having a compiler or any kind of easy, fast formalized "sanity" check is a huge privilege of coding work when it comes to AI usage, something missing in almost every other industry. But even in tech the full extent of such capabilities is rather limited to a few programming languages etc.. Outside of those, confidence in understanding, vouching for the output is limited without actually RTFM. I mean, move fast and break things, but I don't think the quality of knowledge gain is comparable to doing it the hard way.
Side note: I also think, prospectively, it's really bad, if pressure for efficiency on the tech stack used is reduced by making any mess seemingly manageable by AI interfacing (which is insanely inefficient and wasteful in someone else's backyard). Dev "pain" and productivity pressure are needed to improve ergonomics and performance, driving innovation. Why would nix improve, if it can be managed through a chat interface? Similarly, the whole human communication quirk of writing prose with each other becomes utterly meaningless, if expansion and compression of information is expected to happen through AI interface intermediaries anyway. A prose letter is formality and subtext of respectful human interaction, which becomes worthless/offensive, if done by a mindless machine. And if the need for prose vanishes, the AI overhead is fantastical, ridiculous compared to effectively sending the prompt information bits directly to the recipient (without expansion and compression in-between). Nothing makes me wanna scream more than LLMs talking with each other, no matter the language. That's just insanely stupid. If there is not value in formality and indirect communication, we can just cut that and use a trivially simple and amazingly efficient protocol for information exchange.
> _I_ feel like this goes into the overthinking territory. I think software and systems will still die by their merits. Same applies to training data. If bugs regularly make it to end users and a competing solution has less defects, I don't think the buggy solution will stay any more afloat thanks to AI.
But where is the training data coming from? I also think this is fallaciously extrapolating from prior tech innovation and questionable market narratives (considering the tech oligopoly). These models are not like lego, can't be tuned or adjusted like that, there is nothing linear about them. And if you spend all that investment money on manually fine-tuning answers, the tech itself does not warrant the cash, to begin with. That's not AI, that's just a lot of effort. The pyramids are also evidence of laborious human hubris and great expense, a sight to behold, but hardly a technological revolution with great ROI.
I don't think refeeding model degradation is comparable to bad human input as with autopilot (besides, as if that's actually working/solved :D). Thing is, frequency of faulty human information is probably rather constant, while AI slop is exploding, drowning available total human crafted content (again, the scenario is AI wide adoption). And that's not even considering unique feedback mechanism, not fully understood. Who is gonna put out handcrafted, thoroughly thought out training content anymore, when the skill of learning itself atrophied widely? And who is gonna do it for free? Or who is gonna pay for it, when you also have to pay for the absurd energy and resource expenses at some point? Keep in mind, AI, in contrast to human intelligence, does not gain functional understanding and relies statistical tricks from crunching stupid amounts of data. One thought-out piece of code sufficient for you to get cooking, means nothing to the machine. To train these things, you need massive input. Again, where is all that clean data coming from? How much are you willing to pay for an AI service to help you a little with nix?
If there was no refeeding degradation, we would have escape velocity and AGI. In that case, all bets are off and money becomes meaningless anyway. The expenses, the investments don't make sense. Nothing of this shit makes sense :D