Most people don't run around holding out their smartphone directly in front of them. It has to be pointed at the subject, and tends to be obvious.

Smart glasses, however, are always aimed at whatever the wearer is looking at. They may or may not be recording (note the reports of people hiding the LED indicators), and at a fair distance could easily be mistaken for a normal pair.

The general populace is much more likely to notice the former recording rather than the latter.

I've seen people keep their phone in their shirt pocket. The only reason it tends to be obvious is that most people aren't trying to be covert. Those aren't the ones you should be worried about.