Don't buy the argument, this stuff is simple enough that we can formally verify and audit it very carefully.

Additionally, NIST had the foresight to derandomize all the algorithms, so we can now check e.g. what a correct implementation will produce on particular seeds. This is a big deal because a bunch of e.g. ECDSA bugs, such as biased nonces or reused nonces, are trivally caught by such tests/derandomization.

TLDR: the bugs are going to be elsewhere.