Not to even forget how unstable that sort of living is. A few bad seasons from various causes could really affect population. Just look at history of famines. It kinda works when you have industrialised agriculture in other places to fallback on, but without that it is very risky in long term.