There is utility for having a reserved set of opcode space for "NOP if you don't know what the semantics are, but later ISAs may attach semantics for it," because this allows you to add various instructions that merely do nothing on processors that don't support them. The ENDBR32/ENDBR64 instructions for CET, XACQUIRE/XRELEASE hints for LOCK, the MPX instructions, the PREFETCH instructions all use reserved NOP space (0F0D and 0F18-0F1F opcode space).

This is true, but the encoding space reserved for future extensions that is interpreted as NOP should be many times smaller than the space for encodings that generate the invalid instruction exception.

The reason is that the number of useful instructions that are only performance hints or checks for some exceptional conditions, so that if they are ignored that does not have bad consequences, is very limited.

For the vast majority of instruction set extensions, not executing the new instructions completely changes the behavior of the program, which is not acceptable, so the execution of such programs must be prevented on older CPUs.

Regarding the order of prefixes, Intel has made mistakes in not specifying it initially in 8086 and in allowing redundant prefixes. The latter has been partially corrected in later CPUs by imposing a limit for the instruction length.

Because of this lack of specification, the various compilers and assemblers have generated any instruction formats that were accepted by an 8088, so it became impossible to tighten the specification.

However, what is really weird is why Intel and AMD have continued to accept incrorrect instruction encodings even after later extensions of the ISA that clearly specified only a certain encoding to be valid, but in reality the CPUs also accept other encodings and now there are programs that use those alternative encodings that were supposed to be invalid.