> I do like integration tests, but I often tell people the art of modern software is to make reliable systems on top of unreliable components.

There is a dramatic difference between unreliable in the sense of S3 or other services and unreliable as in "we get different sets of logical outputs when we provide the same input to a LLM". In the first, you can prepare for what are logical outcomes -- network failures, durability loss, etc. In the latter, unless you know the total space of outputs for a LLM you cannot prepare. In the operational sense, LLMs are not a system component, they are a system builder. And a rather poor one, at that.

> And the integration tests should 100% include times when the network flakes out and drops 1/2 of replies and corrupts msgs and the like.

Yeah, it's not that hard to include that in modern testing.