Not to mention “labor they don’t understand” is only possible with specialization and modernity.

Back when everyone was a farmer, yeah sure everyone understood the job

I think that even back in early farming days, there were usually some guys who were good at using spears and didn't know much about farming who would go around taking part of the farmers' output. If the farmers were relatively lucky, the guys with the spears would stick around and defend the farmers from other guys with spears in a long-term arrangement of trading protection for material goods instead of just roaming around robbing farmers without giving anything back.