If the current state is anything to go by, an automated test would not only flag your out of distribution results but try to gaslight everyone reading its output with additional false indicators to map you into an area that's in distribution. Statistical models cannot accept the existence of extremely rare edge cases.

▲

ACCount37 2 hours ago | parent [-]

Modern LLMs routinely beat human doctors at diagnosing "extremely rare edge cases".

They have unmatched breadth of knowledge by default, and can maintain attention across entire medical histories.

	▲	multjoy 2 hours ago \| parent \| next [-]
		Citation very much needed.
	▲	fn-mote 2 hours ago \| parent \| prev [-]
		> Modern LLMs routinely beat human doctors at diagnosing "extremely rare edge cases". There is a selection bias here. Not saying it wouldn’t work, but right now you hear about exceptional cases, not when the LLM wants to amputate for a wart. We all work with LLMs, right? It hasn’t been long at all since an LLM gaslit me while attempting to recover an unbootable laptop. I should have been recommended a few simple steps to try; instead, it was unable to ignore the irrelevant details and led me on an hours-long chase. To me that means the LLM will also struggle to ignore irrelevant medical information.