Given all of that, the unemployment rate is still only 4.5%.

Sounds like the metric is unreliable and cannot be trusted as input for policy, based on the evidence and ground truth.

U-6 (the most inclusive unemployment rate) is 8.1 as of this comment, the highest it’s been in the last five years: https://fred.stlouisfed.org/series/U6RATE

So, start cutting labor visas until the unemployment rate improves. The domestic labor clearly exists.