> So, a bad actor can now know when I'm at my house or not.

A bad actor who wants to act on that information can just wait near your house on a workday morning and watch you leave, likely not to return for the better part of the day. No techno-voodoo required.