But people are expected to understand the part of the system they are responsible for at the level of abstraction they are being paid to operate.

This new arrangement would be perfectly fine if they aren't responsible when/if it breaks.

I don't think there is anything new here and the metaphor holds up perfectly. There have always been bugs we don't understand in compilers or libraries or implementations beyond that, that make the path we chose unavailable to us at a certain level. The responsibility is to create a working solution, sure, but there is nothing that would prevent us from getting there by typing "Hey LLM, this is not working, let's try a different approach", even though it might not feel great.