I asked Claude once if there was a way to open a tun/tap on Windows without a driver. It hallucinated an entire supposedly undocumented NT kernel API that as far as I can tell has never existed, complete with parts of this hallucinated API being depreciated in favor of newer non-existent parts.

It was so detailed it makes me wonder if maybe it was in the training data somewhere. Maybe it ingested an internal MS doc for a proposed API or something. The case of the missing ARM instructions makes me wonder the same. Maybe someone on a forum proposed them and they were ingested.

I did actually verify on the OS that the calls do not exist in the kernel or any driver or DLL on the system.

LLM's generating text referencing nonexistent up API's or machine instructions seems to me to be exactly like LLMs generating text referencing nonexistent legal cases or generating fictional text referencing nonexistent people.

They've been fed enough real text in each of these various styles that they can parrot the shallow surface style without reference to the internals.