Depends where you live. In my state you pretty much cannot build any kind of residence on land that is zoned for agriculture.

Generally you are allowed on resident per 40 acres or something similar - farms are getting larger and that leaves plenty of land that doesn't have a house that could.