Isn’t this the ultimate black box? If a forecasting system is a black box, then you have no chance of understanding why its performance might deteriorate. Once that happens it essentially becomes a digital paper-weight.

That's not a good argument because it's like saying that an LLM is a black box, yet we use them all day every day. The two share the same engineering and operating principles.