You've described the exact problem. Sub pixel rendering exploits the fact that most display pixels aren't discrete logical squares.

https://en.wikipedia.org/wiki/Subpixel_rendering

Right, but there's still an addressable set of display units that convert an RGB signal to actual glowing lights, right?

It seems like defining px to be perceptual is one of those compatibility hacks that probably helped a ton back in the day, but we have perceptual units and now we're stuck with that baggage.

I don't have much of an opinion on this (I use em) but it has taken me by surprise a few times.