I think that's the route I'm gonna go, I want my configuration to be transparent and built so that I can understand it top-to-bottom. I've tried Doom, but it felt too magical.

As a Doom user, the benefit is largely in discovering the absolutely vast plethora of features that Emacs and Emacs packages lend to you in a nice set of defaults and aesthetics that shouldn't feel too alien as a Neovim user. I encourage you to give it a week in parallel with your actual config.