Maybe it sees a bunch of segfaults in the Zig codebase and decides a memory safe language is better, just as the human did in Bun's rewrite. I'm not sure why you think it wouldn't have been able to conceive a rewrite.

Such an AI would be unusable in production because there is no bound on the work that can be created and external side effects that could happen. All projects have real world constraints and side effects, budget, customer requirements, etc. Imagine a non-technical person prompting the AI "make the app faster and fix all the bugs" and the AI autonomously decides to rewrite the whole production app and all its dependencies in Rust that is live and serving thousands of users. Doing a full rewrite would take days to execute, and have all sorts of side effects on the actual users during deployment (even if it was done 100% correctly, work has to be paused, affairs need to be coordinated with real customers, etc.). Even if the AI has that capability (and I do believe it is possible, even with today's models), that's not what a business owner wants. You don't want to use an AI which, if you gave it a 3 word prompt, it could suddenly decide on its own to rewrite your whole entire business. An AI that had that much autonomy could just decide on its own to pivot your whole startup and sell something else. A business owner wouldn't even want to use such an AI which could have such large unbounded side effects.