You can, in fact, do that. It's called (aptly enough) the empirical Bayes method. [1]

[1] https://en.wikipedia.org/wiki/Empirical_Bayes_method

Empirical Bayes is exactly what I was getting at. It's a pragmatic modelling choice, but it loses the theoretical guarantees about uncertainty quantification that pure Bayesianism gives us.

(Though if you have a reference for why empirical Bayes does give theoretical guarantees, I'll be happy to change my mind!)