The problem is that sufficiently poor security is indistinguishable from authorized public access. And unfortunately a lot of real world "digital security" is in fact that bad.

A lot of these "hacks" are the equivalent of asking "hey, can I come in?" and the guard assuming that anyone who would ask is authorized, and thus saying "yes". But if the guard said "yes" then it seems a bit absurd to call it trespassing.