Sounds like a further disadvantage of going that route, like maybe you get a swappable GPU and RAM, but your motherboard and power supply are custom to the case it came in or something.

Yep, this is what I've noticed. As long as you can stick with the same motherboard and PSU, you can kinda-sorta upgrade them. But after that you hit a wall.