RSTS/E for the PDP-11 had an overlay facility too.

That machine had a 64k user space but could support more physical memory, a big PDP-11 installation might support 20 terminals which each offer a BASIC experience just a little bigger than the Apple ][ a few years earlier, particularly after they introduced separate address spaces for code and data.

In 1980 as a kid I was reading books about computers and would get my fingers on a TRS-80 or a PDP-8 and thinking the PDP-11 is what "the next thing" up from the "8-bit micros" but really the PDP-11 was long in the tooth in 1980 because it did not exceed 64k barrier of those chips.

The 8086 had a pragmatic approach to addressing more memory, you have to remember that when the IBM PC came out hardly anyone could afford to load the machine up with 640k or more memory, instead you would have 64k or 128k, so in the early days COM files did not seem limited but rather a way to migrate from the CP/M ecosystem in terms of compilers and tooling.

Personally I loved assembly language programming for the real-mode 80286 and did not feel the segmentation model to be much of a burden. I remember great portability of Unix games like Nethack to DOS. My 80286 beat the pants off any PDP-11 and my 80486 trounced the VAX-11/780... DEC's minicomputers were dead but the VAX was a pioneer machine that popularized the virtual memory model of modern 32-bit and 64-bit computers.