Honest question, what would be different from doing the same with Debian packages? Building packages from source is trivial in Debian. It is literally one command, not much different from installing binary packages. Hosting internal repositories is completely expected and well tested.

Personally I have used Gentoo for many years, though never professionally. Its secure extensions were very well tested and documented back in the day, before SELinux pretty much took over. The kernel build system was terrific.

But never once did I consider setting compiler flags an important feature. The person who knows what flags are best used with an individual package, that -Os is more important than -O3 for this particular loop, is either the packager (who knows the system architecture) or the original developer (who wrote the code). Never the end user.

Debian is comparably a more mature system. The downside is perhaps that its packages are a bit more complex than they should be. Functionality-wise it can do anything you could wish for. But if you are a big institution with a lot of competent people in house, go right ahead. You can't possibly go wrong with a well tested system and the knowledge to improve the bits you need.