> The CPU also doesn't care if you use prefixes that aren't valid for a specific instruction, for example a REP on a non-repeatable instruction.

This is one of the reasons why the x86 could be extended so much. PAUSE is just REP NOP, for example. Segment prefixes in front of conditional branches were used as static branch prediction hints (which I believe have returned in some newer Intel CPUs). Useful if you want to make a hint on newer CPUs that is harmless on older CPUs.

Some prefixes have become part of the encoding for certain SIMD instructions, but that is a different case because those prefixes aren't hints.

Not at all.

The correct behavior for allowing future extensions has already been introduced by Intel with 80186, in 1982, which has introduced an invalid instruction exception, to be used for all undefined instruction opcodes.

This behavior was unlike 8086/8088, which happily executed any undefined instructions, most of them being aliases to defined instructions.

For any opcode where current CPUs generate invalid instruction exceptions, it is very easy to define them in future CPUs to encode useful instructions. Had REP NOP generated exceptions in old CPUs, it would have been still fine for it to become PAUSE in current CPUs. Unfortunately, the designers of Intel CPUs have not always followed their own documentation, so not all invalid opcodes generate the exception, as they should. The non-enforcing of this condition has led to the existence of even commercial programs that are invalid or of compilers that generate officially invalid instructions.

It is true that there are a few cases when Intel has exploited the fact that some encodings were equivalent with a NOP on old CPUs, by reusing them for some instruction on new CPUs, where this allowed the execution of a program compiled for new CPUs on old CPUs. However this has been possible only for very few instructions, e.g. for branch direction hints, when not executing them on old CPUs does not change the result of a program.

In general the reuse of an opcode for a new instruction, when that opcode does not generate exceptions on old CPUs, is very dangerous, because the execution on old CPUs of a program compiled for new CPUs will have unpredictable consequences, like destroying some property of the user.

Your example with PAUSE is also one of the very few examples, besides branch hints, where the execution of a new program on old computers is not dangerous, despite the reassignment of the opcode.

Some time ago there was a discussion about a bug in some CPU, but I do not remember in which one, where the bug was triggered when the order of the REP prefix and of the 64-bit REX prefix was invalid, but the invalid order was ignored by the older CPUs instead of generating the appropriate exception, which allowed the execution of invalid programs, which did not have any bad effects on old CPUs, but they triggered the bug on that specific new CPU.

The new CPU should have been bug-free, but also the programs that triggered the bug should not have existed, as they should have crashed immediately on any older CPU.