If you're going this route, I have found netBSD a better option for this sort of thing.

It has a rump kernel architecture which makes reusing the drivers almost trivial compared to reusing linus drivers with a new kernel.