The official compatibility layer of Microsoft works the same way as 86box: https://betawiki.net/wiki/NTVDM

There is no magic

That link says NTVDM uses the CPUs virtual-8086 mode, rather than emulating a CPU; NTVDM's approach can't be done on a CPU that's in 64bit mode (per Intel's docs), the CPU would have to be stepped back into protected mode first (which isn't the same as when you run a 32bit application in a 64bit OS, that's a separate compatibility state within 64bit mode).