You are right. What most people don’t understand however is that “the border” extends for 100 miles

https://www.aclu.org/know-your-rights/border-zone

This effectively means border patrol can do whatever they want in areas that do not involve a border crossing and involve citizens just living their normal lives

That concerns more than 220 million people in the USA, who can be arbitrarily stopped in the street and searched.

Be afraid...