Sure, but a big part of the reason for that is that we produce a huge surplus of food, so food prices are extremely low compared to how wealthy the US is. That means wages for farm workers are too low for typical Americans to want to do the job.

If our food production goes down significantly, that will raise prices which will let wages for farm workers rise to the point where more people will be willing to do the job. Will it be unpleasant? Sure, but not to the point of famine, we'll just go back to spending a larger portion of our household budgets on food like we used to fifty years ago.