Would this not require some control over the hardware? Which would be difficult for the FOSS community?

maybe not, heck people reverse engineered apple hardware and implemented it in various FOSS driver stacks

But yeah, vendors maintaining their drivers upstream in FOSS projects would obviously make it easer