IMO it sort of depends.

For a serious crime like murder you want to imprison them for their sentence and then deport them. You obviously don't want the most serious consequence for a foreigner coming to the US to murder someone to be deportation.

But for a minor crime like shoplifting maybe just tossing them out is fine. Is it worth locking them up for a couple months when you can just wash your hands of them?