This is not completely true. You can use a generic kernel with a custom device tree.

The only problem is that distributions currently tend to package them together, but that shouldn't be obligatory.

You can't if the firmware provided DTB doesn't follow any upstream Linux approved bindings and instead uses some vendor kernel specific bindings.