Some of it's size, some of it the fact that the camera is a second device, and some of it's workflow.
I tried a Sony RX100 (1" sensor) when they first came out, optimistic about the possibility of using it for 'general purpose' photography. After all, it's small enough.
The problem was, it's a second device to carry around and keep charged. Then once you capture the image, it's largely stuck on the device until you find a way to offload your images. I briefly experimented with cables that would let me do things like transfer images from the RX100 to my (Android at the time) mobile phone, for archiving and sending to family and friends. That turned the whole thing into the sort of science fair project that I didn't have time for as the parent of a very young child. (Although in fairness, I can't think of a single time in my life when I'd have had the patience, kids or not.)
This is why, for all the arguments you can make against them as cameras, I've come to be very thankful for the amount of effort that Apple and others have made to get appealing images out of devices I always carry around anyway. I can take a set of pictures, edit them, have them automatically archived to cloud storage, and send them to whoever I want.. all with a single device I was carrying around anyway.
This leaves open the fact that the 'real' camera workflow is still an option when there's the need for higher image quality and the time (or money to hire a photographer) to take advantage of what a DSLR or the like can do.
(When I compare what I can do with my iPhone to what my parents had available to them (a 110 format camera and 35mm Nikons), I like the tradeoffs a lot better. the image quality available now is definitely better than the 110. Some of those 35mm exposures are probably better quality than what I can get out of an iPhone, but they're all stuck in albums and slides, and nobody ever looks at them. )
