Agreed, that seems like exactly the thing that would get you pulled aside at the border, and/or give ICE a reason to not trust your American passport.