I think that the main problem is that android has a lot of weird modifications that are not consistent with the rest of linux distros. The user data is suddenly in /data instead of /home, theres no package manager, no systemd (for better or worse), and there's hella lotta security gotchas, for example call recording is impossible without root as far as I know. I'm not saying that Android is not hackable, but it's a different type of hackability than desktop linux, you have to learn it all over again and in my opinion it's much harder to master than desktop linux.

I've been on ubports for 3 years and while it also has some weird caveats like read only rootfs, no working package manager (due to read-only fs. however ubports has pretty cool support for lxc containers where you can use apt). Due to chronic lack of time I haven't been able to sit down on my phone to play with it a bit (for example id like to install waydroid), but it seems a lot easier than android. For example, while there isn't an app for call recording, some guy worked around it by writing a systemd user service as a workaround[1]. This is exactly the type of thing I'm thinking about when talking "linux phone".

For me as a linux user, the difference if ubports was a human, I'd think that perhaps they were sick, whereas if android was a human, i'd shoot them in the face :)

[1] https://forums.ubports.com/post/75157