> no way of knowing if the AI is bullshitting them

Back testing, hold-out sets, cross validation.

This has all happened before.