The modern construcion site is a factory. They move the entire factor to the job, but that doesn't mean it isn't a factory.

It's sort of a blend. For many houses, the walls and roof trusses are delivered pre-built. Possibly even pre-wired, insulated, and drywalled. They are then stood up on site, and a moderate amount of connecting framing and finish work done by hand.

Labor and materials isn't the problem with housing costs. It's onerous permitting and zoning and code requirements.

The nice thing about a factory built house compared to an independent contractor is you don't have to wait weeks between job types, your plumbers are on hand as soon as it's ready for them to get started.

With that said, the advantage disappears compared to national builders- the guys who buy up big farm fields and build entire subdivisions all in one go before they even have buyers. They can keep crews rotated between jobs in a fairly predictable schedule, so the only thing holding anyone back compared to the factory is bad weather preventing digging out room for foundations.