So you stand upon a big pile of black boxes.

Black boxes aren't inherently bad. But if they don't have well defined mappings of inputs to outputs, they aren't good black boxes. That's the problem with Claude Code imo.